2020-03-12T03:06:27.564 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-12T03:06:27.565 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.14z0Ko.tmp -- --verbose --lock --description rados:standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_20:05:04-rados:standalone-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_20:05:04-rados:standalone-wip-zafman-octo-diag-distro-basic-smithi/4848789 2020-03-12T03:06:27.584 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-12T03:06:27.631 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_20:05:04-rados:standalone-wip-zafman-octo-diag-distro-basic-smithi/4848789 branch: wip-zafman-octo-diag description: rados:standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848789' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_20:05:04-rados:standalone-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados:standalone suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27955 2020-03-12T03:06:27.631 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-12T03:06:27.631 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-12T03:06:27.632 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-12T03:06:27.633 INFO:teuthology.task.internal:Checking packages... 2020-03-12T03:06:27.648 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-12T03:06:27.648 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-12T03:06:27.648 INFO:teuthology.packaging:ref: None 2020-03-12T03:06:27.649 INFO:teuthology.packaging:tag: None 2020-03-12T03:06:27.649 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-12T03:06:27.649 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-12T03:06:27.649 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-12T03:06:27.859 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-12T03:06:27.859 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-12T03:06:27.868 INFO:teuthology.task.internal:no buildpackages task found 2020-03-12T03:06:27.868 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-12T03:06:27.899 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-12T03:06:27.900 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-12T03:06:27.944 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-12T03:06:37.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-12T03:06:47.981 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-12T03:06:58.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-12T03:07:08.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-12T03:07:18.048 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_20:05:04-rados:standalone-wip-zafman-octo-diag-distro-basic-smithi/4848789', 'machine_type': 'smithi'} 2020-03-12T03:07:18.065 DEBUG:teuthology.lock.ops:locked ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:07:18.107 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-12T03:07:18.295 INFO:teuthology.provision.fog.smithi180:Scheduling deploy of centos 8.1 2020-03-12T03:07:18.646 INFO:teuthology.orchestra.console:Power off smithi180 2020-03-12T03:07:18.646 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-12T03:07:18.667 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-12T03:07:22.698 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-12T03:07:26.810 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-12T03:07:27.021 INFO:teuthology.orchestra.console:Power off for smithi180 completed 2020-03-12T03:07:27.122 INFO:teuthology.orchestra.console:Power on smithi180 2020-03-12T03:07:27.122 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-12T03:07:27.142 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-12T03:07:31.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-12T03:07:35.285 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-12T03:07:35.524 INFO:teuthology.orchestra.console:Power on for smithi180 completed 2020-03-12T03:07:35.625 INFO:teuthology.provision.fog.smithi180:Waiting for deploy to finish 2020-03-12T03:10:21.848 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:10:30.910 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:10:39.994 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:10:49.051 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:10:58.111 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:11:07.194 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:11:16.251 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:11:25.307 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:12:31.310 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:12:31.630 INFO:teuthology.orchestra.run.smithi180:> hostname 2020-03-12T03:12:31.910 INFO:teuthology.orchestra.run.smithi180.stdout:smithi180.front.sepia.ceph.com 2020-03-12T03:12:31.911 INFO:teuthology.orchestra.run.smithi180:> grep smithi180.front.sepia.ceph.com /etc/hosts 2020-03-12T03:12:31.926 INFO:teuthology.orchestra.run.smithi180.stdout:172.21.15.180 smithi180.front.sepia.ceph.com smithi180 2020-03-12T03:12:31.926 INFO:teuthology.orchestra.run.smithi180:> sudo hostname smithi180 2020-03-12T03:12:32.046 INFO:teuthology.orchestra.run.smithi180:> sudo sed -i -e 's/smithi180.front.sepia.ceph.com/smithi180/g' /etc/hosts 2020-03-12T03:12:32.075 INFO:teuthology.orchestra.run.smithi180:> sudo sed -i -e 's/172.21.15.180/172.21.15.180/g' /etc/hosts 2020-03-12T03:12:32.142 INFO:teuthology.orchestra.run.smithi180:> sudo sed -i -e 's/smithi180.front.sepia.ceph.com/smithi180/g' /etc/hostname 2020-03-12T03:12:32.209 INFO:teuthology.orchestra.run.smithi180:> sudo hostname smithi180 2020-03-12T03:12:32.275 INFO:teuthology.provision.fog.smithi180:Deploy complete! 2020-03-12T03:12:32.277 DEBUG:teuthology.parallel:result is None 2020-03-12T03:12:32.370 INFO:teuthology.lock.ops:Checking smithi180.front.sepia.ceph.com 2020-03-12T03:12:32.371 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-12T03:12:32.390 INFO:teuthology.lock.ops:Updating [smithi180.front.sepia.ceph.com]: set os type and version on server 2020-03-12T03:12:32.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:12:32.391 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:12:32.464 INFO:teuthology.orchestra.run.smithi180:> true 2020-03-12T03:12:32.619 INFO:teuthology.orchestra.run.smithi180:> uname -m 2020-03-12T03:12:32.674 INFO:teuthology.orchestra.run.smithi180.stdout:x86_64 2020-03-12T03:12:32.674 INFO:teuthology.orchestra.run.smithi180:> cat /etc/os-release 2020-03-12T03:12:32.728 INFO:teuthology.orchestra.run.smithi180.stdout:NAME="CentOS Linux" 2020-03-12T03:12:32.729 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION="8 (Core)" 2020-03-12T03:12:32.729 INFO:teuthology.orchestra.run.smithi180.stdout:ID="centos" 2020-03-12T03:12:32.729 INFO:teuthology.orchestra.run.smithi180.stdout:ID_LIKE="rhel fedora" 2020-03-12T03:12:32.729 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION_ID="8" 2020-03-12T03:12:32.729 INFO:teuthology.orchestra.run.smithi180.stdout:PLATFORM_ID="platform:el8" 2020-03-12T03:12:32.730 INFO:teuthology.orchestra.run.smithi180.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-12T03:12:32.730 INFO:teuthology.orchestra.run.smithi180.stdout:ANSI_COLOR="0;31" 2020-03-12T03:12:32.730 INFO:teuthology.orchestra.run.smithi180.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-12T03:12:32.730 INFO:teuthology.orchestra.run.smithi180.stdout:HOME_URL="https://www.centos.org/" 2020-03-12T03:12:32.730 INFO:teuthology.orchestra.run.smithi180.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-12T03:12:32.731 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:32.731 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-12T03:12:32.731 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-12T03:12:32.731 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-12T03:12:32.732 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-12T03:12:32.732 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:32.733 INFO:teuthology.lock.ops:Updating smithi180.front.sepia.ceph.com on lock server 2020-03-12T03:12:32.749 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-12T03:12:32.764 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi180.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkaIWO/+9PtF2X+PUgrlbBwRjPBNNh2sVnNaxdzaJ/u83J0+aZ9tFTiU0mvT4pT5bJjQwB8CMaexJ+KfgAINA7jZu3YgeRyCtUxmJT0WoL9u7MPzFU/iG9fHrAPwGpi2maZHYY1S6JBeODQFX79lAYi5ZHgv8ycxtmrt2AmbSOl4+mfH48EZYW4AdXglqptNGSdktIPiXiYuELL+bGVBfC6LhQoVvIfQHY9bfGzqFci80iBpBo6Lau7n1SdLiBzbADJQUhaDAXvMSAzKRQrtH5zSVhR2go6efkP3R48KBVn/iTcD6w9zQKTLEWB6U813zfG5aWlq31g/hoMd5VfJsLVvxdEnd67CmxQk24u/o/qI/gacUUuBqAyVco1x7Zt3TChumvQdvRFM2AOtY0vDRsFUKMNP2WIP+DUc7rRoM3rPjWIdwIHvvbnxzdkBvx51QO81tm+JQfHpCGlT6u93wqSXGqKEFNf000fYBRgKppeHm1l0tzI7xyWrUk6kc9lMk= 2020-03-12T03:12:32.765 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-12T03:12:32.804 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-12T03:12:32.814 INFO:teuthology.task.internal:Saving configuration 2020-03-12T03:12:32.824 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-12T03:12:32.834 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-12T03:12:32.849 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-12 03:07:18.057992', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi180.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBESO7JHqAlrq0wbg/RcqlRoKJITrzFbrnR++6a6SIF9iPTwUo8XVbb/EBU8yhCMcDzkRJ6hrAwsFicOnK9Ot1tY=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_20:05:04-rados:standalone-wip-zafman-octo-diag-distro-basic-smithi/4848789'} 2020-03-12T03:12:32.849 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-12T03:12:32.859 INFO:teuthology.task.internal:roles: ubuntu@smithi180.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-12T03:12:32.859 INFO:teuthology.run_tasks:Running task console_log... 2020-03-12T03:12:32.904 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-12T03:12:32.904 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-12T03:12:32.914 INFO:teuthology.task.internal:Opening connections... 2020-03-12T03:12:32.914 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:12:32.916 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:12:32.984 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-12T03:12:32.994 INFO:teuthology.orchestra.run.smithi180:> uname -m 2020-03-12T03:12:33.144 INFO:teuthology.orchestra.run.smithi180.stdout:x86_64 2020-03-12T03:12:33.144 INFO:teuthology.orchestra.run.smithi180:> cat /etc/os-release 2020-03-12T03:12:33.197 INFO:teuthology.orchestra.run.smithi180.stdout:NAME="CentOS Linux" 2020-03-12T03:12:33.197 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION="8 (Core)" 2020-03-12T03:12:33.197 INFO:teuthology.orchestra.run.smithi180.stdout:ID="centos" 2020-03-12T03:12:33.197 INFO:teuthology.orchestra.run.smithi180.stdout:ID_LIKE="rhel fedora" 2020-03-12T03:12:33.197 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION_ID="8" 2020-03-12T03:12:33.198 INFO:teuthology.orchestra.run.smithi180.stdout:PLATFORM_ID="platform:el8" 2020-03-12T03:12:33.198 INFO:teuthology.orchestra.run.smithi180.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-12T03:12:33.198 INFO:teuthology.orchestra.run.smithi180.stdout:ANSI_COLOR="0;31" 2020-03-12T03:12:33.198 INFO:teuthology.orchestra.run.smithi180.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-12T03:12:33.198 INFO:teuthology.orchestra.run.smithi180.stdout:HOME_URL="https://www.centos.org/" 2020-03-12T03:12:33.198 INFO:teuthology.orchestra.run.smithi180.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-12T03:12:33.198 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:33.199 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-12T03:12:33.199 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-12T03:12:33.199 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-12T03:12:33.199 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-12T03:12:33.199 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:33.199 INFO:teuthology.lock.ops:Updating smithi180.front.sepia.ceph.com on lock server 2020-03-12T03:12:33.212 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-12T03:12:33.231 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-12T03:12:33.240 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-12T03:12:33.240 INFO:teuthology.orchestra.run.smithi180:> test '!' -e /home/ubuntu/cephtest 2020-03-12T03:12:33.253 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-12T03:12:33.263 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-12T03:12:33.263 INFO:teuthology.orchestra.run.smithi180:> test -z $(ls -A /var/lib/ceph) 2020-03-12T03:12:33.314 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-12T03:12:33.340 INFO:teuthology.run_tasks:Running task kernel... 2020-03-12T03:12:33.361 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-12T03:12:33.362 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-12T03:12:33.362 INFO:teuthology.orchestra.run.smithi180:> uname -r 2020-03-12T03:12:33.375 INFO:teuthology.orchestra.run.smithi180.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-12T03:12:33.375 INFO:teuthology.task.kernel:Running kernel on smithi180: 4.18.0-147.5.1.el8_1.x86_64 2020-03-12T03:12:33.376 INFO:teuthology.orchestra.run.smithi180:> sudo yum install -y kernel 2020-03-12T03:12:35.957 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 1 day, 11:46:42 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-12T03:12:36.341 INFO:teuthology.orchestra.run.smithi180.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-12T03:12:36.342 INFO:teuthology.orchestra.run.smithi180.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-12T03:12:36.502 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:12:36.503 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:12:36.503 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:12:36.633 INFO:teuthology.orchestra.run.smithi180:> echo no | sudo yum reinstall kernel || true 2020-03-12T03:12:37.022 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 1 day, 11:46:44 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-12T03:12:37.388 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:12:37.388 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:12:37.389 INFO:teuthology.orchestra.run.smithi180.stdout: Package Architecture Version Repository Size 2020-03-12T03:12:37.389 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:12:37.389 INFO:teuthology.orchestra.run.smithi180.stdout:Reinstalling: 2020-03-12T03:12:37.389 INFO:teuthology.orchestra.run.smithi180.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-12T03:12:37.389 INFO:teuthology.orchestra.run.smithi180.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-12T03:12:37.389 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:37.389 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:12:37.390 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:12:37.390 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:37.390 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 3.1 M 2020-03-12T03:12:37.391 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 0 2020-03-12T03:12:37.391 INFO:teuthology.orchestra.run.smithi180.stderr:Operation aborted. 2020-03-12T03:12:37.482 INFO:teuthology.orchestra.run.smithi180.stdout:Is this ok [y/N]: 2020-03-12T03:12:37.482 INFO:teuthology.orchestra.run.smithi180:> sudo yum reinstall -y kernel || true 2020-03-12T03:12:37.868 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 1 day, 11:46:44 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-12T03:12:38.228 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:12:38.228 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:12:38.229 INFO:teuthology.orchestra.run.smithi180.stdout: Package Architecture Version Repository Size 2020-03-12T03:12:38.229 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:12:38.229 INFO:teuthology.orchestra.run.smithi180.stdout:Reinstalling: 2020-03-12T03:12:38.229 INFO:teuthology.orchestra.run.smithi180.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-12T03:12:38.229 INFO:teuthology.orchestra.run.smithi180.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-12T03:12:38.230 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:38.230 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:12:38.230 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:12:38.230 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:38.231 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 3.1 M 2020-03-12T03:12:38.231 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 0 2020-03-12T03:12:38.231 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:12:38.324 INFO:teuthology.orchestra.run.smithi180.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 18 MB/s | 1.5 MB 00:00 2020-03-12T03:12:38.328 INFO:teuthology.orchestra.run.smithi180.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 17 MB/s | 1.5 MB 00:00 2020-03-12T03:12:38.329 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:12:38.330 INFO:teuthology.orchestra.run.smithi180.stdout:Total 30 MB/s | 3.1 MB 00:00 2020-03-12T03:12:38.355 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:12:38.534 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:12:38.534 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:12:38.601 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:12:38.601 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:12:38.731 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:12:38.892 INFO:teuthology.orchestra.run.smithi180.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-12T03:12:39.001 INFO:teuthology.orchestra.run.smithi180.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-12T03:12:39.002 INFO:teuthology.orchestra.run.smithi180.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-12T03:12:39.136 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-12T03:12:39.136 INFO:teuthology.orchestra.run.smithi180.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-12T03:12:39.352 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-12T03:12:39.352 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-12T03:12:39.352 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-12T03:12:39.352 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-12T03:12:39.584 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-12T03:12:39.585 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:39.585 INFO:teuthology.orchestra.run.smithi180.stdout:Reinstalled: 2020-03-12T03:12:39.585 INFO:teuthology.orchestra.run.smithi180.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-12T03:12:39.585 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:39.586 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:12:39.710 INFO:teuthology.orchestra.run.smithi180:> rpm -q kernel --last | head -n 1 2020-03-12T03:12:39.772 INFO:teuthology.orchestra.run.smithi180.stdout:kernel-4.18.0-147.el8.x86_64 Thu 12 Mar 2020 03:12:38 AM UTC 2020-03-12T03:12:39.774 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-12T03:12:39.774 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-12T03:12:39.774 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-12T03:12:39.774 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-12T03:12:39.774 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-12T03:12:39.774 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi180.front.sepia.ceph.com, path=None, version=distro) 2020-03-12T03:12:39.774 INFO:teuthology.orchestra.run.smithi180:> rpm -q kernel --last | head -n 1 2020-03-12T03:12:39.816 INFO:teuthology.orchestra.run.smithi180.stdout:kernel-4.18.0-147.el8.x86_64 Thu 12 Mar 2020 03:12:38 AM UTC 2020-03-12T03:12:39.817 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-12T03:12:39.817 INFO:teuthology.orchestra.run.smithi180:> sudo rpm -qi grub2-tools 2020-03-12T03:12:39.889 INFO:teuthology.orchestra.run.smithi180.stdout:Name : grub2-tools 2020-03-12T03:12:39.889 INFO:teuthology.orchestra.run.smithi180.stdout:Epoch : 1 2020-03-12T03:12:39.890 INFO:teuthology.orchestra.run.smithi180.stdout:Version : 2.02 2020-03-12T03:12:39.890 INFO:teuthology.orchestra.run.smithi180.stdout:Release : 78.el8 2020-03-12T03:12:39.890 INFO:teuthology.orchestra.run.smithi180.stdout:Architecture: x86_64 2020-03-12T03:12:39.890 INFO:teuthology.orchestra.run.smithi180.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-12T03:12:39.890 INFO:teuthology.orchestra.run.smithi180.stdout:Group : System Environment/Base 2020-03-12T03:12:39.890 INFO:teuthology.orchestra.run.smithi180.stdout:Size : 9192691 2020-03-12T03:12:39.890 INFO:teuthology.orchestra.run.smithi180.stdout:License : GPLv3+ 2020-03-12T03:12:39.891 INFO:teuthology.orchestra.run.smithi180.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-12T03:12:39.891 INFO:teuthology.orchestra.run.smithi180.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-12T03:12:39.891 INFO:teuthology.orchestra.run.smithi180.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-12T03:12:39.891 INFO:teuthology.orchestra.run.smithi180.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-12T03:12:39.891 INFO:teuthology.orchestra.run.smithi180.stdout:Relocations : (not relocatable) 2020-03-12T03:12:39.891 INFO:teuthology.orchestra.run.smithi180.stdout:Packager : CentOS BuildSystem 2020-03-12T03:12:39.891 INFO:teuthology.orchestra.run.smithi180.stdout:Vendor : CentOS 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout:Summary : Support tools for GRUB. 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout:Description : 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-12T03:12:39.892 INFO:teuthology.orchestra.run.smithi180.stdout:hardware devices. 2020-03-12T03:12:39.893 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:12:39.893 INFO:teuthology.orchestra.run.smithi180.stdout:This subpackage provides tools for support of all platforms. 2020-03-12T03:12:39.894 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-12T03:12:39.894 INFO:teuthology.task.kernel:Updating grub on smithi180 to boot 4.18.0-147.el8.x86_64 2020-03-12T03:12:39.894 INFO:teuthology.orchestra.run.smithi180:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-12T03:12:40.783 INFO:teuthology.orchestra.run.smithi180.stderr:Generating grub configuration file ... 2020-03-12T03:12:42.631 INFO:teuthology.orchestra.run.smithi180.stderr:done 2020-03-12T03:12:42.633 INFO:teuthology.orchestra.run.smithi180:> mktemp 2020-03-12T03:12:42.665 INFO:teuthology.orchestra.run.smithi180.stdout:/tmp/tmp.F85zCE1WfX 2020-03-12T03:12:42.666 INFO:teuthology.orchestra.run.smithi180:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.F85zCE1WfX 2020-03-12T03:12:42.745 INFO:teuthology.orchestra.run.smithi180:> sudo chmod 0666 /tmp/tmp.F85zCE1WfX 2020-03-12T03:12:42.906 DEBUG:teuthology.orchestra.remote:smithi180:/tmp/tmp.F85zCE1WfX is 4KB 2020-03-12T03:12:42.954 INFO:teuthology.orchestra.run.smithi180:> rm -fr /tmp/tmp.F85zCE1WfX 2020-03-12T03:12:42.967 INFO:teuthology.orchestra.run.smithi180:> sudo /bin/ls /boot/loader/entries || true 2020-03-12T03:12:43.033 INFO:teuthology.orchestra.run.smithi180.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-12T03:12:43.033 INFO:teuthology.orchestra.run.smithi180.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-12T03:12:43.033 INFO:teuthology.orchestra.run.smithi180.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-12T03:12:43.034 INFO:teuthology.orchestra.run.smithi180:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-12T03:12:43.216 INFO:teuthology.orchestra.run.smithi180:> sudo shutdown -r now 2020-03-12T03:13:13.245 INFO:teuthology.misc:Re-opening connections... 2020-03-12T03:13:13.246 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:13.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:13.246 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:31.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:31.622 DEBUG:teuthology.misc:waited 18.3762378693 2020-03-12T03:13:32.622 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:32.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:32.623 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:35.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:35.686 DEBUG:teuthology.misc:waited 22.4402248859 2020-03-12T03:13:36.687 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:36.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:36.687 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:38.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:38.758 DEBUG:teuthology.misc:waited 25.5123510361 2020-03-12T03:13:39.758 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:39.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:39.759 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:41.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:41.830 DEBUG:teuthology.misc:waited 28.5841858387 2020-03-12T03:13:42.830 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:42.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:42.831 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:44.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:44.902 DEBUG:teuthology.misc:waited 31.6562440395 2020-03-12T03:13:45.903 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:45.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:45.903 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:47.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:47.974 DEBUG:teuthology.misc:waited 34.7283070087 2020-03-12T03:13:48.975 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:48.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:48.975 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:51.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:51.046 DEBUG:teuthology.misc:waited 37.800235033 2020-03-12T03:13:52.046 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:52.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:52.047 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:54.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:54.118 DEBUG:teuthology.misc:waited 40.8723089695 2020-03-12T03:13:55.119 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:55.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:55.119 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:13:57.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.180 2020-03-12T03:13:57.190 DEBUG:teuthology.misc:waited 43.944234848 2020-03-12T03:13:58.190 INFO:teuthology.misc:trying to connect to ubuntu@smithi180.front.sepia.ceph.com 2020-03-12T03:13:58.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:13:58.191 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:14:30.085 INFO:teuthology.orchestra.run.smithi180:> true 2020-03-12T03:14:30.418 DEBUG:teuthology.misc:waited 77.1722888947 2020-03-12T03:14:31.418 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-12T03:14:31.419 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-12T03:14:31.419 INFO:teuthology.orchestra.run.smithi180:> uname -r 2020-03-12T03:14:31.435 INFO:teuthology.orchestra.run.smithi180.stdout:4.18.0-147.el8.x86_64 2020-03-12T03:14:31.435 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-12T03:14:31.435 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-12T03:14:32.436 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-12T03:14:32.437 INFO:teuthology.orchestra.run.smithi180:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-12T03:14:32.534 INFO:teuthology.orchestra.run.smithi180.stdout:ttyS1 2020-03-12T03:14:32.543 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-12T03:14:32.552 INFO:teuthology.task.internal:Creating test directory... 2020-03-12T03:14:32.552 INFO:teuthology.orchestra.run.smithi180:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-12T03:14:32.568 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-12T03:14:32.915 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-12T03:14:32.949 INFO:teuthology.task.internal:Creating archive directory... 2020-03-12T03:14:32.950 INFO:teuthology.orchestra.run.smithi180:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-12T03:14:32.979 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-12T03:14:32.990 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-12T03:14:32.990 INFO:teuthology.orchestra.run.smithi180:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-12T03:14:33.068 INFO:teuthology.orchestra.run.smithi180.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:14:33.083 INFO:teuthology.orchestra.run.smithi180.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:14:33.085 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-12T03:14:33.095 INFO:teuthology.task.internal:Configuring sudo... 2020-03-12T03:14:33.095 INFO:teuthology.orchestra.run.smithi180:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-12T03:14:33.128 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-12T03:14:33.140 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-12T03:14:33.140 INFO:teuthology.orchestra.run.smithi180:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-12T03:14:33.183 INFO:teuthology.orchestra.run.smithi180:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-12T03:14:33.259 INFO:teuthology.orchestra.run.smithi180:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-12T03:14:33.352 INFO:teuthology.orchestra.run.smithi180:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-12T03:14:33.390 INFO:teuthology.orchestra.run.smithi180:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-12T03:14:33.458 INFO:teuthology.orchestra.run.smithi180:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-12T03:14:33.543 INFO:teuthology.orchestra.run.smithi180:> sudo service rsyslog restart 2020-03-12T03:14:33.670 INFO:teuthology.orchestra.run.smithi180.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-12T03:14:33.993 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-12T03:14:34.007 INFO:teuthology.task.internal:Starting timer... 2020-03-12T03:14:34.007 INFO:teuthology.run_tasks:Running task pcp... 2020-03-12T03:14:34.505 INFO:teuthology.run_tasks:Running task selinux... 2020-03-12T03:14:34.515 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-12T03:14:34.515 INFO:teuthology.orchestra.run.smithi180:> sudo service auditd rotate 2020-03-12T03:14:34.596 INFO:teuthology.orchestra.run.smithi180.stdout:Rotating logs: [ OK ] 2020-03-12T03:14:34.597 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-12T03:14:34.597 INFO:teuthology.orchestra.run.smithi180:> /usr/sbin/getenforce 2020-03-12T03:14:34.614 INFO:teuthology.orchestra.run.smithi180.stdout:Permissive 2020-03-12T03:14:34.614 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi180.front.sepia.ceph.com': 'permissive'} 2020-03-12T03:14:34.615 INFO:teuthology.orchestra.run.smithi180:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-12T03:14:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-12T03:14:34.682 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-12T03:14:34.682 INFO:teuthology.orchestra.run.smithi180:> sudo /usr/sbin/setenforce permissive 2020-03-12T03:14:34.751 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-12T03:14:34.760 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-12T03:14:34.761 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-12T03:14:34.778 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-12T03:14:34.781 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi180.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-12T03:14:35.151 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-12T03:14:36.343 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-12T03:14:39.433 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-12T03:14:39.463 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:39 +0000 (0:00:00.069) 0:00:00.069 ******** 2020-03-12T03:14:43.542 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi180.front.sepia.ceph.com] 2020-03-12T03:14:43.621 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:43 +0000 (0:00:04.158) 0:00:04.228 ******** 2020-03-12T03:14:43.968 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:43.969 INFO:teuthology.task.ansible.out:TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-12T03:14:44.047 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:44 +0000 (0:00:00.426) 0:00:04.654 ******** 2020-03-12T03:14:44.622 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-12T03:14:44.624 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-12T03:14:44.664 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:44 +0000 (0:00:00.617) 0:00:05.271 ******** 2020-03-12T03:14:44.689 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-12T03:14:44.690 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:44.728 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:44 +0000 (0:00:00.064) 0:00:05.335 ******** 2020-03-12T03:14:45.085 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-12T03:14:45.086 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-12T03:14:45.124 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:45 +0000 (0:00:00.395) 0:00:05.731 ******** 2020-03-12T03:14:46.223 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-12T03:14:46.223 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583982885.14-183237278275005/source", "state": "file", "uid": 0} 2020-03-12T03:14:46.261 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:46 +0000 (0:00:01.137) 0:00:06.868 ******** 2020-03-12T03:14:46.698 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-12T03:14:46.699 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-12T03:14:46.700 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-12T03:14:46.817 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2020-03-12T03:14:46.819 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-12T03:14:46.823 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-12T03:14:47.003 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:47 +0000 (0:00:00.741) 0:00:07.610 ******** 2020-03-12T03:14:47.668 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi180.front.sepia.ceph.com] 2020-03-12T03:14:47.753 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-12T03:14:47.755 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:47 +0000 (0:00:00.750) 0:00:08.361 ******** 2020-03-12T03:14:48.281 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-12T03:14:48.282 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.248916", "end": "2020-03-12 03:14:48.254486", "rc": 0, "start": "2020-03-12 03:14:48.005570", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-12T03:14:48.319 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:48 +0000 (0:00:00.565) 0:00:08.927 ******** 2020-03-12T03:14:48.534 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-12T03:14:48.535 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013525", "end": "2020-03-12 03:14:48.508895", "failed_when_result": false, "rc": 0, "start": "2020-03-12 03:14:48.495370", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-12T03:14:48.584 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:48 +0000 (0:00:00.264) 0:00:09.191 ******** 2020-03-12T03:14:48.623 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-12T03:14:48.623 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-12T03:14:48.688 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:48 +0000 (0:00:00.103) 0:00:09.295 ******** 2020-03-12T03:14:48.689 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:48.714 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-12T03:14:48.715 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:48.756 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:48 +0000 (0:00:00.068) 0:00:09.363 ******** 2020-03-12T03:14:48.782 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-12T03:14:48.783 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:48.824 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:48 +0000 (0:00:00.067) 0:00:09.431 ******** 2020-03-12T03:14:48.850 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-12T03:14:48.851 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:48.895 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:48 +0000 (0:00:00.071) 0:00:09.502 ******** 2020-03-12T03:14:48.922 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-12T03:14:48.923 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:48.960 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:48 +0000 (0:00:00.065) 0:00:09.567 ******** 2020-03-12T03:14:48.989 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-12T03:14:48.990 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.032 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:49 +0000 (0:00:00.071) 0:00:09.639 ******** 2020-03-12T03:14:49.080 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-12T03:14:49.081 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.092 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.103 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.104 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.114 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.125 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.126 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.137 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.148 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.160 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.171 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.171 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.182 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.193 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.194 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.205 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.216 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.217 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.228 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.239 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.250 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.251 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.262 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.263 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.275 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.275 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.287 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.288 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.300 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.301 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.312 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.313 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.325 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.340 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.341 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.352 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.353 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.354 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.364 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.365 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.376 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.377 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.379 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:49.388 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.389 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.400 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.411 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.412 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.425 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.431 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.436 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.436 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.448 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.450 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.461 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.462 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.473 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.473 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.484 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.484 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.495 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.496 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.506 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.517 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.518 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.529 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.530 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.541 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.541 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.552 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.553 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.563 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.564 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.575 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.575 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.586 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.587 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.598 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.598 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.609 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.610 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.620 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.635 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.636 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.647 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.648 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.659 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.659 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.670 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.671 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.682 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.682 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.693 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.693 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.704 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.715 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.716 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.717 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.730 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.739 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.739 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.750 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.752 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.761 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.762 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.778 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.784 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.785 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.795 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.796 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.807 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.861 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:49 +0000 (0:00:00.829) 0:00:10.469 ******** 2020-03-12T03:14:49.888 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-12T03:14:49.926 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:49 +0000 (0:00:00.064) 0:00:10.533 ******** 2020-03-12T03:14:49.954 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-12T03:14:49.955 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:49.996 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:49 +0000 (0:00:00.070) 0:00:10.603 ******** 2020-03-12T03:14:50.024 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-12T03:14:50.026 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.065 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:50 +0000 (0:00:00.068) 0:00:10.672 ******** 2020-03-12T03:14:50.095 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.140 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:50 +0000 (0:00:00.075) 0:00:10.747 ******** 2020-03-12T03:14:50.167 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-12T03:14:50.168 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.205 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:50 +0000 (0:00:00.064) 0:00:10.812 ******** 2020-03-12T03:14:50.233 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.267 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:50 +0000 (0:00:00.061) 0:00:10.874 ******** 2020-03-12T03:14:50.302 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-12T03:14:50.303 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.313 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.314 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.325 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.325 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.336 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.336 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.347 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.349 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.349 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.365 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.371 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.372 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.382 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.383 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.394 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.394 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.405 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.416 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.416 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.427 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.428 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.438 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.449 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.450 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.451 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.471 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.472 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.473 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.484 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.484 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.495 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.519 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.520 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.521 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.531 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.533 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.534 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-12T03:14:50.575 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:50 +0000 (0:00:00.308) 0:00:11.182 ******** 2020-03-12T03:14:51.327 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-12T03:14:51.328 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-12T03:14:51.363 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:51 +0000 (0:00:00.788) 0:00:11.970 ******** 2020-03-12T03:14:51.398 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-12T03:14:51.400 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => { "msg": "Host smithi180.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-12T03:14:51.437 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:51 +0000 (0:00:00.073) 0:00:12.044 ******** 2020-03-12T03:14:51.473 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-12T03:14:51.474 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-12T03:14:51.510 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:51 +0000 (0:00:00.073) 0:00:12.117 ******** 2020-03-12T03:14:51.535 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-12T03:14:51.537 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:51.572 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:51 +0000 (0:00:00.061) 0:00:12.179 ******** 2020-03-12T03:14:51.845 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:51.847 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.073593", "end": "2020-03-12 03:14:51.821209", "rc": 0, "start": "2020-03-12 03:14:51.747616", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-12T03:14:51.886 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:51 +0000 (0:00:00.313) 0:00:12.493 ******** 2020-03-12T03:14:51.887 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:51.912 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:51.913 INFO:teuthology.task.ansible.out:TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:51.953 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:51 +0000 (0:00:00.067) 0:00:12.560 ******** 2020-03-12T03:14:51.979 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-12T03:14:51.981 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.021 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.067) 0:00:12.628 ******** 2020-03-12T03:14:52.047 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-12T03:14:52.048 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.089 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.068) 0:00:12.696 ******** 2020-03-12T03:14:52.115 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-12T03:14:52.117 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.163 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.073) 0:00:12.770 ******** 2020-03-12T03:14:52.200 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-12T03:14:52.202 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.250 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.086) 0:00:12.857 ******** 2020-03-12T03:14:52.287 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-12T03:14:52.289 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.332 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.082) 0:00:12.939 ******** 2020-03-12T03:14:52.369 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-12T03:14:52.370 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi180.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-12T03:14:52.412 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.080) 0:00:13.019 ******** 2020-03-12T03:14:52.449 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-12T03:14:52.450 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.494 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.082) 0:00:13.101 ******** 2020-03-12T03:14:52.533 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-12T03:14:52.534 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.575 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.080) 0:00:13.182 ******** 2020-03-12T03:14:52.614 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-12T03:14:52.615 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-12T03:14:52.656 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.080) 0:00:13.263 ******** 2020-03-12T03:14:52.693 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-12T03:14:52.694 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.735 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.079) 0:00:13.342 ******** 2020-03-12T03:14:52.762 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-12T03:14:52.763 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-12T03:14:52.801 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.065) 0:00:13.408 ******** 2020-03-12T03:14:52.826 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-12T03:14:52.827 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.863 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.062) 0:00:13.470 ******** 2020-03-12T03:14:52.889 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-12T03:14:52.890 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.928 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.064) 0:00:13.535 ******** 2020-03-12T03:14:52.954 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-12T03:14:52.955 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:52.992 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:52 +0000 (0:00:00.063) 0:00:13.599 ******** 2020-03-12T03:14:53.017 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-12T03:14:53.018 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.056 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.063) 0:00:13.663 ******** 2020-03-12T03:14:53.082 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-12T03:14:53.083 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-12T03:14:53.122 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.065) 0:00:13.729 ******** 2020-03-12T03:14:53.147 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-12T03:14:53.148 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.184 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.062) 0:00:13.791 ******** 2020-03-12T03:14:53.220 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-12T03:14:53.221 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.261 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.077) 0:00:13.869 ******** 2020-03-12T03:14:53.297 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-12T03:14:53.299 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.350 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.088) 0:00:13.958 ******** 2020-03-12T03:14:53.386 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-12T03:14:53.387 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.425 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.074) 0:00:14.032 ******** 2020-03-12T03:14:53.426 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:53.462 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:53.464 INFO:teuthology.task.ansible.out:TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.500 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.075) 0:00:14.107 ******** 2020-03-12T03:14:53.501 INFO:teuthology.task.ansible.out: 2020-03-12T03:14:53.528 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-12T03:14:53.530 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.565 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.064) 0:00:14.172 ******** 2020-03-12T03:14:53.601 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-12T03:14:53.603 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-12T03:14:53.637 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.071) 0:00:14.244 ******** 2020-03-12T03:14:53.662 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-12T03:14:53.700 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.063) 0:00:14.307 ******** 2020-03-12T03:14:53.725 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-12T03:14:53.727 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:53.763 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:53 +0000 (0:00:00.062) 0:00:14.370 ******** 2020-03-12T03:14:53.997 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-12T03:14:54.006 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-12T03:14:54.034 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:54 +0000 (0:00:00.271) 0:00:14.641 ******** 2020-03-12T03:14:54.565 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-12T03:14:54.566 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-12T03:14:55.042 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-12T03:14:55.090 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:55 +0000 (0:00:01.056) 0:00:15.697 ******** 2020-03-12T03:14:55.579 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-12T03:14:55.579 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-12T03:14:56.035 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-12T03:14:56.075 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.984) 0:00:16.682 ******** 2020-03-12T03:14:56.110 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-12T03:14:56.112 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.147 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.072) 0:00:16.754 ******** 2020-03-12T03:14:56.183 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-12T03:14:56.184 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.222 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.074) 0:00:16.829 ******** 2020-03-12T03:14:56.248 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-12T03:14:56.249 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.283 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.061) 0:00:16.891 ******** 2020-03-12T03:14:56.308 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-12T03:14:56.310 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.347 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.063) 0:00:16.954 ******** 2020-03-12T03:14:56.372 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-12T03:14:56.374 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.409 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.061) 0:00:17.016 ******** 2020-03-12T03:14:56.434 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-12T03:14:56.435 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.471 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.061) 0:00:17.078 ******** 2020-03-12T03:14:56.496 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-12T03:14:56.497 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.533 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.062) 0:00:17.140 ******** 2020-03-12T03:14:56.558 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-12T03:14:56.559 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.598 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.065) 0:00:17.205 ******** 2020-03-12T03:14:56.623 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-12T03:14:56.625 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:14:56.660 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:56 +0000 (0:00:00.062) 0:00:17.267 ******** 2020-03-12T03:14:57.521 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-12T03:14:57.523 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.666018", "end": "2020-03-12 03:14:57.500363", "rc": 0, "start": "2020-03-12 03:14:56.834345", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:14:57.561 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:14:57 +0000 (0:00:00.901) 0:00:18.169 ******** 2020-03-12T03:15:00.865 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-12T03:15:00.866 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-12T03:15:00.903 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:00 +0000 (0:00:03.340) 0:00:21.509 ******** 2020-03-12T03:15:00.927 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:00.963 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:00 +0000 (0:00:00.060) 0:00:21.570 ******** 2020-03-12T03:15:00.989 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-12T03:15:00.990 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:01.024 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:01 +0000 (0:00:00.060) 0:00:21.631 ******** 2020-03-12T03:15:01.049 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-12T03:15:01.050 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:01.087 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:01 +0000 (0:00:00.062) 0:00:21.694 ******** 2020-03-12T03:15:01.112 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-12T03:15:01.113 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:01.146 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:01 +0000 (0:00:00.059) 0:00:21.754 ******** 2020-03-12T03:15:01.626 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-12T03:15:01.626 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-12T03:15:01.667 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:01 +0000 (0:00:00.520) 0:00:22.274 ******** 2020-03-12T03:15:02.586 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-12T03:15:02.586 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-12T03:15:02.645 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:02 +0000 (0:00:00.977) 0:00:23.251 ******** 2020-03-12T03:15:03.155 INFO:teuthology.task.ansible.out: 2020-03-12T03:15:03.157 INFO:teuthology.task.ansible.out:TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi180.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-12T03:15:03.646 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-12T03:15:04.121 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-12T03:15:04.169 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:04 +0000 (0:00:01.524) 0:00:24.776 ******** 2020-03-12T03:15:04.363 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-12T03:15:04.364 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-12T03:15:04.413 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:04 +0000 (0:00:00.244) 0:00:25.020 ******** 2020-03-12T03:15:04.884 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-12T03:15:04.885 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-12T03:15:05.316 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-12T03:15:05.756 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-12T03:15:05.795 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:05 +0000 (0:00:01.382) 0:00:26.402 ******** 2020-03-12T03:15:05.823 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:05.864 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:05 +0000 (0:00:00.068) 0:00:26.471 ******** 2020-03-12T03:15:05.902 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-12T03:15:05.904 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-12T03:15:05.939 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:05 +0000 (0:00:00.074) 0:00:26.546 ******** 2020-03-12T03:15:06.141 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-12T03:15:06.142 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-12T03:15:06.186 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:06 +0000 (0:00:00.247) 0:00:26.793 ******** 2020-03-12T03:15:06.223 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-12T03:15:06.306 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Thursday 12 March 2020 03:15:06 +0000 (0:00:00.111) 0:00:26.904 ******** 2020-03-12T03:15:07.206 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-12T03:15:07.206 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-12T03:15:07.246 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:07 +0000 (0:00:00.949) 0:00:27.853 ******** 2020-03-12T03:15:07.273 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-12T03:15:07.275 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:07.312 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:07 +0000 (0:00:00.065) 0:00:27.919 ******** 2020-03-12T03:15:07.786 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-12T03:15:07.787 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-12T03:15:07.839 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:07 +0000 (0:00:00.526) 0:00:28.445 ******** 2020-03-12T03:15:08.321 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-12T03:15:08.322 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-12T03:15:08.357 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:08 +0000 (0:00:00.518) 0:00:28.964 ******** 2020-03-12T03:15:08.385 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-12T03:15:08.387 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:08.422 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:08 +0000 (0:00:00.064) 0:00:29.029 ******** 2020-03-12T03:15:08.643 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-12T03:15:08.644 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-12T03:15:08.680 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:08 +0000 (0:00:00.258) 0:00:29.287 ******** 2020-03-12T03:15:08.924 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-12T03:15:08.926 INFO:teuthology.task.ansible.out:fatal: [smithi180.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-12T03:15:08.978 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:08 +0000 (0:00:00.298) 0:00:29.585 ******** 2020-03-12T03:15:09.004 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-12T03:15:09.006 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:09.040 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:09 +0000 (0:00:00.061) 0:00:29.647 ******** 2020-03-12T03:15:09.508 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-12T03:15:09.509 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-12T03:15:09.547 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:09 +0000 (0:00:00.507) 0:00:30.154 ******** 2020-03-12T03:15:10.332 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-12T03:15:10.333 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-03-12 03:14:26 UTC", "ActiveEnterTimestampMonotonic": "11764720", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target var-run.mount systemd-tmpfiles-setup.service -.mount local-fs.target systemd-journald.socket sysinit.target system.slice basic.target tmp.mount network.target time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-03-12 03:14:26 UTC", "AssertTimestampMonotonic": "11763870", "Before": "getty@tty1.service multi-user.target xdm.service plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-03-12 03:14:26 UTC", "ConditionTimestampMonotonic": "11763870", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-12T03:15:10.335 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1613", "ExecMainStartTimestamp": "Thu 2020-03-12 03:14:26 UTC", "ExecMainStartTimestampMonotonic": "11764694", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Thu 2020-03-12 03:14:26 UTC] ; stop_time=[n/a] ; pid=1613 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-03-12 03:14:26 UTC", "InactiveExitTimestampMonotonic": "11764720", "InvocationID": "6f41705cdaeb4b0682841476f8f042e2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-12T03:15:10.337 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "62925", "LimitNPROCSoft": "62925", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "62925", "LimitSIGPENDINGSoft": "62925", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1613", "MemoryAccounting": "yes", "MemoryCurrent": "1171456", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe" 2020-03-12T03:15:10.338 INFO:teuthology.task.ansible.out:, "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-03-12 03:14:26 UTC", "StateChangeTimestampMonotonic": "11764720", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-03-12 03:14:26 UTC", "WatchdogTimestampMonotonic": "11764719", "WatchdogUSec": "0"}} 2020-03-12T03:15:10.375 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:10 +0000 (0:00:00.827) 0:00:30.982 ******** 2020-03-12T03:15:10.578 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-12T03:15:10.578 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001708", "end": "2020-03-12 03:15:10.552950", "rc": 0, "start": "2020-03-12 03:15:10.551242", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-12T03:15:10.617 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:10 +0000 (0:00:00.242) 0:00:31.224 ******** 2020-03-12T03:15:11.546 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-12T03:15:11.548 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-12T03:15:11.586 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:11 +0000 (0:00:00.968) 0:00:32.193 ******** 2020-03-12T03:15:12.510 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-12T03:15:12.511 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-12T03:15:12.551 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:12 +0000 (0:00:00.963) 0:00:33.157 ******** 2020-03-12T03:15:14.054 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-12T03:15:14.055 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-12T03:15:14.096 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:14 +0000 (0:00:01.545) 0:00:34.703 ******** 2020-03-12T03:15:30.600 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-12T03:15:30.601 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.303145", "end": "2020-03-12 03:15:30.577174", "failed_when_result": false, "rc": 0, "start": "2020-03-12 03:15:14.274029", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-12T03:15:30.640 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:30 +0000 (0:00:16.543) 0:00:51.246 ******** 2020-03-12T03:15:31.130 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-12T03:15:31.131 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-12T03:15:31.183 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:31 +0000 (0:00:00.544) 0:00:51.790 ******** 2020-03-12T03:15:31.404 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-12T03:15:31.405 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.019765", "end": "2020-03-12 03:15:31.380230", "rc": 0, "start": "2020-03-12 03:15:31.360465", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:15:31.441 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:31 +0000 (0:00:00.258) 0:00:52.048 ******** 2020-03-12T03:15:31.640 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-12T03:15:31.642 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002068", "end": "2020-03-12 03:15:31.615007", "rc": 0, "start": "2020-03-12 03:15:31.612939", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:15:31.678 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:31 +0000 (0:00:00.236) 0:00:52.285 ******** 2020-03-12T03:15:48.099 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-12T03:15:48.100 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.229053", "end": "2020-03-12 03:15:48.078891", "rc": 0, "start": "2020-03-12 03:15:31.849838", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:15:48.142 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:48 +0000 (0:00:16.464) 0:01:08.749 ******** 2020-03-12T03:15:48.354 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-12T03:15:48.356 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-12T03:15:48.395 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:48 +0000 (0:00:00.253) 0:01:09.002 ******** 2020-03-12T03:15:48.422 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-12T03:15:48.422 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-12T03:15:48.460 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:48 +0000 (0:00:00.064) 0:01:09.067 ******** 2020-03-12T03:15:48.495 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-12T03:15:48.497 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-12T03:15:48.534 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:48 +0000 (0:00:00.074) 0:01:09.141 ******** 2020-03-12T03:15:48.582 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-12T03:15:48.584 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-12T03:15:48.626 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:48 +0000 (0:00:00.091) 0:01:09.233 ******** 2020-03-12T03:15:48.672 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-12T03:15:48.674 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-12T03:15:48.722 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:48 +0000 (0:00:00.095) 0:01:09.329 ******** 2020-03-12T03:15:48.937 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-12T03:15:48.937 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-12T03:15:48.999 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:48 +0000 (0:00:00.277) 0:01:09.606 ******** 2020-03-12T03:15:49.189 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-12T03:15:49.191 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-12T03:15:49.228 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:49 +0000 (0:00:00.229) 0:01:09.835 ******** 2020-03-12T03:15:49.257 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-12T03:15:49.259 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:49.293 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:49 +0000 (0:00:00.064) 0:01:09.900 ******** 2020-03-12T03:15:49.318 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-12T03:15:49.320 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:15:49.355 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:49 +0000 (0:00:00.061) 0:01:09.962 ******** 2020-03-12T03:15:49.576 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-12T03:15:49.578 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-12T03:15:49.613 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:49 +0000 (0:00:00.258) 0:01:10.220 ******** 2020-03-12T03:15:50.177 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-12T03:15:50.177 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-12T03:15:50.218 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:50 +0000 (0:00:00.605) 0:01:10.825 ******** 2020-03-12T03:15:50.254 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-12T03:15:50.255 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi180"}, "changed": false} 2020-03-12T03:15:50.294 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:50 +0000 (0:00:00.075) 0:01:10.901 ******** 2020-03-12T03:15:51.302 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-12T03:15:51.304 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi180.front.sepia.ceph.com", "ansible_hostname": "smithi180", "ansible_nodename": "smithi180"}, "changed": true, "name": "smithi180"} 2020-03-12T03:15:51.339 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:51 +0000 (0:00:01.045) 0:01:11.946 ******** 2020-03-12T03:15:51.814 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-12T03:15:51.816 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-12T03:15:51.858 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:51 +0000 (0:00:00.518) 0:01:12.465 ******** 2020-03-12T03:15:52.339 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-12T03:15:52.340 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-12T03:15:52.388 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:52 +0000 (0:00:00.530) 0:01:12.995 ******** 2020-03-12T03:15:53.220 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-12T03:15:53.222 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-12T03:15:53.264 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:53 +0000 (0:00:00.875) 0:01:13.871 ******** 2020-03-12T03:15:53.764 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-12T03:15:53.764 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-12T03:15:53.802 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:53 +0000 (0:00:00.538) 0:01:14.410 ******** 2020-03-12T03:15:54.007 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-12T03:15:54.007 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-12T03:15:54.044 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:54 +0000 (0:00:00.241) 0:01:14.651 ******** 2020-03-12T03:15:54.246 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-12T03:15:54.248 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-12T03:15:54.290 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:54 +0000 (0:00:00.245) 0:01:14.897 ******** 2020-03-12T03:15:54.516 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-12T03:15:54.518 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-12T03:15:54.559 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:54 +0000 (0:00:00.268) 0:01:15.166 ******** 2020-03-12T03:15:55.051 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-12T03:15:55.058 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-12T03:15:55.104 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:55 +0000 (0:00:00.545) 0:01:15.711 ******** 2020-03-12T03:15:55.580 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-12T03:15:55.581 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-12T03:15:55.620 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:55 +0000 (0:00:00.516) 0:01:16.227 ******** 2020-03-12T03:15:57.482 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-12T03:15:57.483 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.667782", "end": "2020-03-12 03:15:57.459402", "rc": 0, "start": "2020-03-12 03:15:55.791620", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:15:57.519 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:57 +0000 (0:00:01.899) 0:01:18.126 ******** 2020-03-12T03:15:58.023 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-12T03:15:58.024 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-12T03:15:58.500 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-12T03:15:58.973 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-12T03:15:59.012 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:59 +0000 (0:00:01.493) 0:01:19.620 ******** 2020-03-12T03:15:59.533 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-12T03:15:59.534 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-12T03:15:59.593 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:15:59 +0000 (0:00:00.579) 0:01:20.199 ******** 2020-03-12T03:16:00.073 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-12T03:16:00.074 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-12T03:16:00.538 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-12T03:16:01.002 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-12T03:16:01.043 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:01 +0000 (0:00:01.450) 0:01:21.650 ******** 2020-03-12T03:16:01.604 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-12T03:16:01.605 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.341822", "end": "2020-03-12 03:16:01.582579", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-12 03:16:01.240757", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-12T03:16:01.606 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-12T03:16:01.643 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:01 +0000 (0:00:00.600) 0:01:22.250 ******** 2020-03-12T03:16:01.670 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:01.707 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:01 +0000 (0:00:00.064) 0:01:22.314 ******** 2020-03-12T03:16:01.744 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-12T03:16:01.746 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:01.757 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:01.793 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:01 +0000 (0:00:00.085) 0:01:22.400 ******** 2020-03-12T03:16:01.829 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-12T03:16:01.831 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:01.886 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:01 +0000 (0:00:00.092) 0:01:22.493 ******** 2020-03-12T03:16:01.924 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-12T03:16:01.926 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-12T03:16:01.967 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:01 +0000 (0:00:00.081) 0:01:22.574 ******** 2020-03-12T03:16:02.003 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-12T03:16:02.005 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:02.045 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:02 +0000 (0:00:00.077) 0:01:22.652 ******** 2020-03-12T03:16:02.082 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-12T03:16:02.083 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:02.120 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:02 +0000 (0:00:00.075) 0:01:22.727 ******** 2020-03-12T03:16:02.335 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-12T03:16:02.336 INFO:teuthology.task.ansible.out:fatal: [smithi180.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006716", "end": "2020-03-12 03:16:02.306854", "msg": "non-zero return code", "rc": 1, "start": "2020-03-12 03:16:02.300138", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-12T03:16:02.399 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:02 +0000 (0:00:00.278) 0:01:23.006 ******** 2020-03-12T03:16:02.425 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-12T03:16:02.427 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:02.464 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:02 +0000 (0:00:00.064) 0:01:23.071 ******** 2020-03-12T03:16:05.054 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-12T03:16:05.055 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-12T03:16:05.098 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:05 +0000 (0:00:02.634) 0:01:25.705 ******** 2020-03-12T03:16:08.069 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-12T03:16:08.070 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-12T03:16:08.106 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:08 +0000 (0:00:03.007) 0:01:28.713 ******** 2020-03-12T03:16:13.244 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-12T03:16:13.245 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:04.900373", "end": "2020-03-12 03:16:13.213935", "rc": 0, "start": "2020-03-12 03:16:08.313562", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\""], "stdout": "CentOS-8 - AppStream 142 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 131 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 77 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 137 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:03 ago on Thu 12 Mar 2020 03:16:08 AM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-12T03:16:13.246 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 142 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 131 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 77 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 137 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:03 ago on Thu 12 Mar 2020 03:16:08 AM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-12T03:16:13.294 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:13 +0000 (0:00:05.187) 0:01:33.901 ******** 2020-03-12T03:16:21.048 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-12T03:16:21.049 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-12T03:16:21.102 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:21 +0000 (0:00:07.808) 0:01:41.709 ******** 2020-03-12T03:16:21.128 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-12T03:16:21.129 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:21.170 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:21 +0000 (0:00:00.067) 0:01:41.777 ******** 2020-03-12T03:16:22.093 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-12T03:16:22.094 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-12T03:16:22.138 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:22 +0000 (0:00:00.968) 0:01:42.745 ******** 2020-03-12T03:16:22.187 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-12T03:16:22.189 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:22.230 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:22 +0000 (0:00:00.091) 0:01:42.837 ******** 2020-03-12T03:16:22.580 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-12T03:16:22.584 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target polkit.service basic.target dbus.socket system.slice dbus.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service ebtables.service ipset.service shutdown.target iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-12T03:16:22.585 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "62925", "LimitNPROCSoft": "62925", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "62925", "Li 2020-03-12T03:16:22.586 INFO:teuthology.task.ansible.out:mitSIGPENDINGSoft": "62925", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", " 2020-03-12T03:16:22.588 INFO:teuthology.task.ansible.out:SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-12T03:16:22.644 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:22 +0000 (0:00:00.413) 0:01:43.251 ******** 2020-03-12T03:16:22.956 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-12T03:16:22.958 INFO:teuthology.task.ansible.out:fatal: [smithi180.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-12T03:16:23.003 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:23 +0000 (0:00:00.358) 0:01:43.610 ******** 2020-03-12T03:16:23.774 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-12T03:16:23.813 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:23 +0000 (0:00:00.809) 0:01:44.420 ******** 2020-03-12T03:16:23.848 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-12T03:16:23.850 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:23.886 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:23 +0000 (0:00:00.073) 0:01:44.493 ******** 2020-03-12T03:16:23.922 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-12T03:16:23.975 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:23 +0000 (0:00:00.088) 0:01:44.582 ******** 2020-03-12T03:16:24.001 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-12T03:16:24.003 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.041 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.065) 0:01:44.647 ******** 2020-03-12T03:16:24.066 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-12T03:16:24.067 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.104 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.063) 0:01:44.711 ******** 2020-03-12T03:16:24.140 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-12T03:16:24.141 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.151 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.190 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.086) 0:01:44.797 ******** 2020-03-12T03:16:24.215 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-12T03:16:24.216 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.256 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.065) 0:01:44.863 ******** 2020-03-12T03:16:24.282 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-12T03:16:24.319 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.062) 0:01:44.926 ******** 2020-03-12T03:16:24.344 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-12T03:16:24.345 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.383 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.064) 0:01:44.990 ******** 2020-03-12T03:16:24.409 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-12T03:16:24.411 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.450 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.066) 0:01:45.057 ******** 2020-03-12T03:16:24.475 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-12T03:16:24.477 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.516 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.066) 0:01:45.123 ******** 2020-03-12T03:16:24.543 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-12T03:16:24.544 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.585 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.068) 0:01:45.192 ******** 2020-03-12T03:16:24.612 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-12T03:16:24.613 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.658 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.073) 0:01:45.265 ******** 2020-03-12T03:16:24.686 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.728 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.069) 0:01:45.335 ******** 2020-03-12T03:16:24.754 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-12T03:16:24.755 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.800 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.072) 0:01:45.407 ******** 2020-03-12T03:16:24.829 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.865 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.065) 0:01:45.472 ******** 2020-03-12T03:16:24.890 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-12T03:16:24.892 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.931 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.065) 0:01:45.538 ******** 2020-03-12T03:16:24.957 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-12T03:16:24.958 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:24.997 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:24 +0000 (0:00:00.066) 0:01:45.604 ******** 2020-03-12T03:16:25.023 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-12T03:16:25.024 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.066 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.068) 0:01:45.673 ******** 2020-03-12T03:16:25.092 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-12T03:16:25.093 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.130 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.064) 0:01:45.737 ******** 2020-03-12T03:16:25.155 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-12T03:16:25.156 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.196 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.066) 0:01:45.803 ******** 2020-03-12T03:16:25.222 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-12T03:16:25.223 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.268 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.071) 0:01:45.875 ******** 2020-03-12T03:16:25.304 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-12T03:16:25.305 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.350 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.082) 0:01:45.957 ******** 2020-03-12T03:16:25.376 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-12T03:16:25.377 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.414 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.063) 0:01:46.021 ******** 2020-03-12T03:16:25.451 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-12T03:16:25.452 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.490 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.076) 0:01:46.097 ******** 2020-03-12T03:16:25.515 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-12T03:16:25.517 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.554 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.063) 0:01:46.161 ******** 2020-03-12T03:16:25.579 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-12T03:16:25.581 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.617 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.063) 0:01:46.224 ******** 2020-03-12T03:16:25.643 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-12T03:16:25.645 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.682 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.064) 0:01:46.289 ******** 2020-03-12T03:16:25.708 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-12T03:16:25.709 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.745 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.063) 0:01:46.352 ******** 2020-03-12T03:16:25.772 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.811 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.065) 0:01:46.418 ******** 2020-03-12T03:16:25.837 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-12T03:16:25.839 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.883 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.071) 0:01:46.490 ******** 2020-03-12T03:16:25.909 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-12T03:16:25.910 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:25.967 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:25 +0000 (0:00:00.083) 0:01:46.574 ******** 2020-03-12T03:16:25.992 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-12T03:16:25.994 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:26.028 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:26 +0000 (0:00:00.061) 0:01:46.635 ******** 2020-03-12T03:16:26.058 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-12T03:16:26.059 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:26.100 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:26 +0000 (0:00:00.071) 0:01:46.707 ******** 2020-03-12T03:16:26.323 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-12T03:16:26.325 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-12T03:16:26.362 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:26 +0000 (0:00:00.262) 0:01:46.969 ******** 2020-03-12T03:16:26.568 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-12T03:16:26.569 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-12T03:16:26.623 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:26 +0000 (0:00:00.261) 0:01:47.230 ******** 2020-03-12T03:16:26.825 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-12T03:16:26.826 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-12T03:16:26.869 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:26 +0000 (0:00:00.245) 0:01:47.476 ******** 2020-03-12T03:16:26.894 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-12T03:16:26.896 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:26.930 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:26 +0000 (0:00:00.061) 0:01:47.537 ******** 2020-03-12T03:16:26.967 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-12T03:16:26.968 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.018 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.085) 0:01:47.623 ******** 2020-03-12T03:16:27.044 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-12T03:16:27.045 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.081 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.064) 0:01:47.688 ******** 2020-03-12T03:16:27.117 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-12T03:16:27.119 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.163 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.081) 0:01:47.770 ******** 2020-03-12T03:16:27.188 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.225 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.062) 0:01:47.832 ******** 2020-03-12T03:16:27.251 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-12T03:16:27.253 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.288 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.062) 0:01:47.895 ******** 2020-03-12T03:16:27.325 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-12T03:16:27.326 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.380 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.092) 0:01:47.988 ******** 2020-03-12T03:16:27.406 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-12T03:16:27.407 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.445 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.064) 0:01:48.052 ******** 2020-03-12T03:16:27.472 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-12T03:16:27.474 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.518 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.072) 0:01:48.125 ******** 2020-03-12T03:16:27.546 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-12T03:16:27.547 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.586 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.068) 0:01:48.193 ******** 2020-03-12T03:16:27.627 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.664 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.077) 0:01:48.271 ******** 2020-03-12T03:16:27.691 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-12T03:16:27.692 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.749 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.085) 0:01:48.356 ******** 2020-03-12T03:16:27.774 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-12T03:16:27.776 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.813 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.063) 0:01:48.420 ******** 2020-03-12T03:16:27.841 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-12T03:16:27.842 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.878 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.065) 0:01:48.485 ******** 2020-03-12T03:16:27.903 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-12T03:16:27.905 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:27.943 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:27 +0000 (0:00:00.064) 0:01:48.550 ******** 2020-03-12T03:16:27.968 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-12T03:16:27.969 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.006 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.063) 0:01:48.613 ******** 2020-03-12T03:16:28.038 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-12T03:16:28.039 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.094 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.087) 0:01:48.701 ******** 2020-03-12T03:16:28.122 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-12T03:16:28.123 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.164 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.070) 0:01:48.771 ******** 2020-03-12T03:16:28.200 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-12T03:16:28.202 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.238 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.074) 0:01:48.845 ******** 2020-03-12T03:16:28.284 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-12T03:16:28.286 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.344 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.105) 0:01:48.951 ******** 2020-03-12T03:16:28.370 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-12T03:16:28.371 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.429 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.085) 0:01:49.036 ******** 2020-03-12T03:16:28.457 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-12T03:16:28.459 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.501 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.072) 0:01:49.108 ******** 2020-03-12T03:16:28.527 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-12T03:16:28.529 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-12T03:16:28.565 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.063) 0:01:49.172 ******** 2020-03-12T03:16:28.605 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-12T03:16:28.607 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.618 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-12T03:16:28.629 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.640 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.650 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.651 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.662 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.706 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.141) 0:01:49.313 ******** 2020-03-12T03:16:28.846 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-12T03:16:28.848 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003279", "end": "2020-03-12 03:16:28.829339", "rc": 0, "start": "2020-03-12 03:16:28.826060", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-12T03:16:28.889 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.183) 0:01:49.496 ******** 2020-03-12T03:16:28.925 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-12T03:16:28.926 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:28.966 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:28 +0000 (0:00:00.076) 0:01:49.573 ******** 2020-03-12T03:16:29.002 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-12T03:16:29.004 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:29.039 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:29 +0000 (0:00:00.073) 0:01:49.646 ******** 2020-03-12T03:16:29.064 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-12T03:16:29.066 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:29.103 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:29 +0000 (0:00:00.063) 0:01:49.710 ******** 2020-03-12T03:16:29.129 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-12T03:16:29.131 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:29.168 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:29 +0000 (0:00:00.065) 0:01:49.775 ******** 2020-03-12T03:16:30.118 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-12T03:16:30.164 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:30 +0000 (0:00:00.995) 0:01:50.771 ******** 2020-03-12T03:16:30.199 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-12T03:16:30.200 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item={u'block_used': 12964182, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130926, u'size_available': 930328272896, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989858, u'device': u'/dev/sda1', u'inode_used': 65118, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130926, "block_size": 4096, "block_total": 240095108, "block_used": 12964182, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989858, "inode_total": 61054976, "inode_used": 65118, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328272896, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-12T03:16:30.238 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:30 +0000 (0:00:00.074) 0:01:50.845 ******** 2020-03-12T03:16:30.437 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-12T03:16:30.438 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005119", "end": "2020-03-12 03:16:30.415419", "rc": 0, "start": "2020-03-12 03:16:30.410300", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:30.475 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:30 +0000 (0:00:00.236) 0:01:51.082 ******** 2020-03-12T03:16:30.551 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-12T03:16:30.552 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'block_used': 12964182, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130926, u'size_available': 930328272896, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989858, u'device': u'/dev/sda1', u'inode_used': 65118, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130926, "block_size": 4096, "block_total": 240095108, "block_used": 12964182, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989858, "inode_total": 61054976, "inode_used": 65118, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328272896, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:30.591 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:30 +0000 (0:00:00.116) 0:01:51.198 ******** 2020-03-12T03:16:30.628 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-12T03:16:30.664 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:30 +0000 (0:00:00.072) 0:01:51.271 ******** 2020-03-12T03:16:30.818 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-12T03:16:30.819 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGGP', u'wwn-0x5000c50092930050'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XGGP', u'wwn': u'0x5000c50092930050', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGGP-part1', u'wwn-0x5000c50092930050-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGGP", "wwn-0x5000c50092930050"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGGP-part1", "wwn-0x5000c50092930050-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGGP", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-12T03:16:30.820 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50092930050"}}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:32.435 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-ryAU7v-0qmz-EZRW-rWV1-5Vnk-Yy2F-Um5Hd1', u'nvme-INTEL_SSDPED1D480GA_PHMB747600M7480DGN', u'nvme-nvme.8086-50484d423734373630304d3734383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.423782", "end": "2020-03-12 03:16:32.395354", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-ryAU7v-0qmz-EZRW-rWV1-5Vnk-Yy2F-Um5Hd1", "nvme-INTEL_SSDPED1D480GA_PHMB747600M7480DGN", "nvme-nvme.8086-50484d423734373630304d3734383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:30.971572", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGP 2020-03-12T03:16:32.436 INFO:teuthology.task.ansible.out:T data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-12T03:16:33.683 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRs9fOky2d8oPc5iemuEU5ZgFDXN0a1Ius'], u'uuids': [u'3d39af49-e36e-42de-a790-d8288cbae9ba']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.047412", "end": "2020-03-12 03:16:33.649065", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRs9fOky2d8oPc5iemuEU5ZgFDXN0a1Ius"], "labels": [], "masters": [], "uuids": ["3d39af49-e36e-42de-a790-d8288cbae9ba"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:32.601653", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-12T03:16:34.949 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRFQ9trFGTFk7r082y7HqFcAGw6dobPpdx'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.066641", "end": "2020-03-12 03:16:34.916997", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRFQ9trFGTFk7r082y7HqFcAGw6dobPpdx"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:33.850356", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-12T03:16:36.186 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRfp6orU8JWARLdv9F5Zyl8JsWNIMhral2'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.041586", "end": "2020-03-12 03:16:36.150028", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRfp6orU8JWARLdv9F5Zyl8JsWNIMhral2"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:35.108442", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-12T03:16:37.429 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRr3qH0JtI1Mfi718O3k9ZteAfRYn5O3hB'], u'uuids': [u'9d04b9ed-15e4-42ce-b368-ce6635274c21']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.050542", "end": "2020-03-12 03:16:37.401201", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRr3qH0JtI1Mfi718O3k9ZteAfRYn5O3hB"], "labels": [], "masters": [], "uuids": ["9d04b9ed-15e4-42ce-b368-ce6635274c21"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:36.350659", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-12T03:16:38.758 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRXmG8PCa0fMsO2ca9k9xHj1s3XLELS5O7'], u'uuids': [u'16c38a67-6e0b-4103-af44-536ae7be518f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.043018", "end": "2020-03-12 03:16:38.633012", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRXmG8PCa0fMsO2ca9k9xHj1s3XLELS5O7"], "labels": [], "masters": [], "uuids": ["16c38a67-6e0b-4103-af44-536ae7be518f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:37.589994", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-12T03:16:38.807 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:38 +0000 (0:00:08.142) 0:01:59.414 ******** 2020-03-12T03:16:38.854 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-12T03:16:38.855 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGGP', u'wwn-0x5000c50092930050'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XGGP', u'wwn': u'0x5000c50092930050', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XGGP-part1', u'wwn-0x5000c50092930050-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGGP", "wwn-0x5000c50092930050"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGGP-part1", "wwn-0x5000c50092930050-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGGP", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-12T03:16:38.857 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c50092930050"}}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:39.085 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-ryAU7v-0qmz-EZRW-rWV1-5Vnk-Yy2F-Um5Hd1', u'nvme-INTEL_SSDPED1D480GA_PHMB747600M7480DGN', u'nvme-nvme.8086-50484d423734373630304d3734383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037556", "end": "2020-03-12 03:16:39.055386", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-ryAU7v-0qmz-EZRW-rWV1-5Vnk-Yy2F-Um5Hd1", "nvme-INTEL_SSDPED1D480GA_PHMB747600M7480DGN", "nvme-nvme.8086-50484d423734373630304d3734383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:39.017830", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 2020-03-12T03:16:39.086 INFO:teuthology.task.ansible.out: 0.0348712 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0348712 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:39.375 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRs9fOky2d8oPc5iemuEU5ZgFDXN0a1Ius'], u'uuids': [u'3d39af49-e36e-42de-a790-d8288cbae9ba']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.096676", "end": "2020-03-12 03:16:39.346741", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRs9fOky2d8oPc5iemuEU5ZgFDXN0a1Ius"], "labels": [], "masters": [], "uuids": ["3d39af49-e36e-42de-a790-d8288cbae9ba"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:39.250065", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0940096 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0940096 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:39.657 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRFQ9trFGTFk7r082y7HqFcAGw6dobPpdx'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.095916", "end": "2020-03-12 03:16:39.630700", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRFQ9trFGTFk7r082y7HqFcAGw6dobPpdx"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:39.534784", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0933472 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0933472 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:39.951 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRfp6orU8JWARLdv9F5Zyl8JsWNIMhral2'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.096520", "end": "2020-03-12 03:16:39.925764", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRfp6orU8JWARLdv9F5Zyl8JsWNIMhral2"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:39.829244", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0938546 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0938546 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:40.242 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRr3qH0JtI1Mfi718O3k9ZteAfRYn5O3hB'], u'uuids': [u'9d04b9ed-15e4-42ce-b368-ce6635274c21']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.095491", "end": "2020-03-12 03:16:40.212210", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRr3qH0JtI1Mfi718O3k9ZteAfRYn5O3hB"], "labels": [], "masters": [], "uuids": ["9d04b9ed-15e4-42ce-b368-ce6635274c21"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:40.116719", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0929029 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0929029 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:40.533 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRXmG8PCa0fMsO2ca9k9xHj1s3XLELS5O7'], u'uuids': [u'16c38a67-6e0b-4103-af44-536ae7be518f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.094371", "end": "2020-03-12 03:16:40.490793", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eyb7bZesNtw7g36nTwhcnkIOt97bgKgRXmG8PCa0fMsO2ca9k9xHj1s3XLELS5O7"], "labels": [], "masters": [], "uuids": ["16c38a67-6e0b-4103-af44-536ae7be518f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-12 03:16:40.396422", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0918568 s, 1.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0918568 s, 1.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:40.578 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:40 +0000 (0:00:01.771) 0:02:01.185 ******** 2020-03-12T03:16:40.847 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-12T03:16:40.847 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.063814", "end": "2020-03-12 03:16:40.809301", "rc": 0, "start": "2020-03-12 03:16:40.745487", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:40.884 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:40 +0000 (0:00:00.305) 0:02:01.491 ******** 2020-03-12T03:16:41.128 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-12T03:16:41.129 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043325", "end": "2020-03-12 03:16:41.105847", "rc": 0, "start": "2020-03-12 03:16:41.062522", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:41.176 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.292) 0:02:01.783 ******** 2020-03-12T03:16:41.201 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-12T03:16:41.243 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.067) 0:02:01.850 ******** 2020-03-12T03:16:41.269 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-12T03:16:41.271 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:41.312 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.068) 0:02:01.919 ******** 2020-03-12T03:16:41.337 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-12T03:16:41.339 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:41.386 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.074) 0:02:01.993 ******** 2020-03-12T03:16:41.413 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-12T03:16:41.415 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:41.456 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.070) 0:02:02.064 ******** 2020-03-12T03:16:41.483 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:41.535 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.077) 0:02:02.142 ******** 2020-03-12T03:16:41.736 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-12T03:16:41.738 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-12T03:16:41.775 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.240) 0:02:02.382 ******** 2020-03-12T03:16:41.821 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-12T03:16:41.822 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={u'block_used': 12964182, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130926, u'size_available': 930328272896, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989858, u'device': u'/dev/sda1', u'inode_used': 65118, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130926, "block_size": 4096, "block_total": 240095108, "block_used": 12964182, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989858, "inode_total": 61054976, "inode_used": 65118, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328272896, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:41.872 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.097) 0:02:02.479 ******** 2020-03-12T03:16:41.910 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-12T03:16:41.912 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:41.953 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:41 +0000 (0:00:00.077) 0:02:02.557 ******** 2020-03-12T03:16:41.976 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-12T03:16:41.978 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:42.013 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:42 +0000 (0:00:00.063) 0:02:02.620 ******** 2020-03-12T03:16:42.039 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-12T03:16:42.040 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:42.079 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:42 +0000 (0:00:00.065) 0:02:02.686 ******** 2020-03-12T03:16:42.105 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-12T03:16:42.107 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:42.157 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:42 +0000 (0:00:00.077) 0:02:02.764 ******** 2020-03-12T03:16:42.183 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-12T03:16:42.185 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:42.224 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:42 +0000 (0:00:00.066) 0:02:02.831 ******** 2020-03-12T03:16:42.254 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-12T03:16:42.255 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:42.295 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:42 +0000 (0:00:00.070) 0:02:02.902 ******** 2020-03-12T03:16:42.907 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-12T03:16:43.239 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} Thursday 12 March 2020 03:16:42 +0000 (0:00:00.655) 0:02:03.558 ******** 2020-03-12T03:16:43.479 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-12T03:16:43.481 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-12T03:16:43.802 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-12T03:16:44.177 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-12T03:16:44.549 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-12T03:16:44.879 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-12T03:16:44.922 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:44 +0000 (0:00:01.970) 0:02:05.529 ******** 2020-03-12T03:16:45.123 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-12T03:16:45.124 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-12T03:16:45.161 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:45 +0000 (0:00:00.239) 0:02:05.768 ******** 2020-03-12T03:16:45.208 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-12T03:16:45.209 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:45.397 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-12T03:16:45.585 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-12T03:16:45.763 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-12T03:16:45.953 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-12T03:16:45.994 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:45 +0000 (0:00:00.832) 0:02:06.601 ******** 2020-03-12T03:16:46.212 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-12T03:16:46.214 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-12T03:16:46.247 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:46 +0000 (0:00:00.252) 0:02:06.854 ******** 2020-03-12T03:16:46.284 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-12T03:16:46.285 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-12T03:16:46.336 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:46 +0000 (0:00:00.079) 0:02:06.934 ******** 2020-03-12T03:16:46.688 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-12T03:16:46.690 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true} 2020-03-12T03:16:46.736 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:46 +0000 (0:00:00.409) 0:02:07.343 ******** 2020-03-12T03:16:47.216 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-12T03:16:47.218 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-12T03:16:47.254 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.518) 0:02:07.861 ******** 2020-03-12T03:16:47.280 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-12T03:16:47.281 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.316 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.061) 0:02:07.923 ******** 2020-03-12T03:16:47.341 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-12T03:16:47.342 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.378 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.061) 0:02:07.985 ******** 2020-03-12T03:16:47.413 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-12T03:16:47.415 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.452 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.074) 0:02:08.059 ******** 2020-03-12T03:16:47.478 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-12T03:16:47.480 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.539 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.087) 0:02:08.146 ******** 2020-03-12T03:16:47.575 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-12T03:16:47.576 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.612 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.072) 0:02:08.219 ******** 2020-03-12T03:16:47.637 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-12T03:16:47.639 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.674 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.062) 0:02:08.281 ******** 2020-03-12T03:16:47.710 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-12T03:16:47.712 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.750 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.072) 0:02:08.353 ******** 2020-03-12T03:16:47.773 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-12T03:16:47.777 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.823 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.076) 0:02:08.430 ******** 2020-03-12T03:16:47.849 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-12T03:16:47.850 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.896 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.072) 0:02:08.503 ******** 2020-03-12T03:16:47.931 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-12T03:16:47.933 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:47.968 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:47 +0000 (0:00:00.071) 0:02:08.575 ******** 2020-03-12T03:16:48.005 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-12T03:16:48.006 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:48.068 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:48 +0000 (0:00:00.099) 0:02:08.675 ******** 2020-03-12T03:16:48.559 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-12T03:16:48.561 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-12T03:16:48.628 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:48 +0000 (0:00:00.560) 0:02:09.235 ******** 2020-03-12T03:16:48.947 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-03-12 03:14:26 UTC", "ActiveEnterTimestampMonotonic": "11699184", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service ntpd.service systemd-tmpfiles-setup.service -.mount sntp.service systemd-journald.socket tmp.mount system.slice sysinit.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-03-12 03:14:25 UTC", "AssertTimestampMonotonic": "10486656", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-03-12 03:14:25 UTC", "ConditionTimestampMonotonic": "10486624", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.servic 2020-03-12T03:16:48.949 INFO:teuthology.task.ansible.out:e", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1571", "ExecMainStartTimestamp": "Thu 2020-03-12 03:14:26 UTC", "ExecMainStartTimestampMonotonic": "11627164", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-03-12 03:14:25 UTC", "InactiveExitTimestampMonotonic": "10490769", "InvocationID": "f775a554a563419286883b89e91a336b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infini 2020-03-12T03:16:48.950 INFO:teuthology.task.ansible.out:ty", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "62925", "LimitNPROCSoft": "62925", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "62925", "LimitSIGPENDINGSoft": "62925", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1571", "MemoryAccounting": "yes", "MemoryCurrent": "2969600", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0 2020-03-12T03:16:48.951 INFO:teuthology.task.ansible.out:", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-03-12 03:14:26 UTC", "StateChangeTimestampMonotonic": "11699184", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-03-12 03:14:26 UTC", "WatchdogTimestampMonotonic": "11627179", "WatchdogUSec": "0"}} 2020-03-12T03:16:48.983 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:48 +0000 (0:00:00.355) 0:02:09.590 ******** 2020-03-12T03:16:49.021 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-12T03:16:49.022 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:49.081 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:49 +0000 (0:00:00.097) 0:02:09.688 ******** 2020-03-12T03:16:49.107 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-12T03:16:49.107 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:49.146 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:49 +0000 (0:00:00.064) 0:02:09.753 ******** 2020-03-12T03:16:49.172 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-12T03:16:49.172 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:49.216 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:49 +0000 (0:00:00.070) 0:02:09.823 ******** 2020-03-12T03:16:49.422 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-12T03:16:49.464 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:49 +0000 (0:00:00.248) 0:02:10.071 ******** 2020-03-12T03:16:49.957 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-12T03:16:49.958 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-12T03:16:49.994 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:49 +0000 (0:00:00.529) 0:02:10.601 ******** 2020-03-12T03:16:51.223 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-12T03:16:51.267 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:51 +0000 (0:00:01.271) 0:02:11.874 ******** 2020-03-12T03:16:51.292 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-12T03:16:51.293 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:51.330 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:51 +0000 (0:00:00.063) 0:02:11.938 ******** 2020-03-12T03:16:51.366 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-12T03:16:51.368 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-12T03:16:51.406 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:51 +0000 (0:00:00.075) 0:02:12.013 ******** 2020-03-12T03:16:51.407 INFO:teuthology.task.ansible.out: 2020-03-12T03:16:51.603 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-12T03:16:51.603 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-12T03:16:51.640 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:51 +0000 (0:00:00.234) 0:02:12.247 ******** 2020-03-12T03:16:51.667 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-12T03:16:51.668 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:51.706 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:51 +0000 (0:00:00.065) 0:02:12.313 ******** 2020-03-12T03:16:51.741 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-12T03:16:51.743 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-12T03:16:51.780 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:51 +0000 (0:00:00.074) 0:02:12.387 ******** 2020-03-12T03:16:51.838 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-12T03:16:51.840 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:51.887 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:51 +0000 (0:00:00.106) 0:02:12.494 ******** 2020-03-12T03:16:52.091 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-12T03:16:52.092 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-12T03:16:52.127 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.240) 0:02:12.735 ******** 2020-03-12T03:16:52.328 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-12T03:16:52.330 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-12T03:16:52.379 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.251) 0:02:12.986 ******** 2020-03-12T03:16:52.404 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-12T03:16:52.406 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.441 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.061) 0:02:13.048 ******** 2020-03-12T03:16:52.487 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-12T03:16:52.489 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi180.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.500 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.511 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.511 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.553 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.112) 0:02:13.160 ******** 2020-03-12T03:16:52.589 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-12T03:16:52.590 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.592 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.603 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.614 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.616 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.655 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.101) 0:02:13.262 ******** 2020-03-12T03:16:52.681 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-12T03:16:52.683 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.744 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.089) 0:02:13.351 ******** 2020-03-12T03:16:52.769 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-12T03:16:52.771 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.810 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.065) 0:02:13.417 ******** 2020-03-12T03:16:52.835 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-12T03:16:52.837 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.875 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.064) 0:02:13.482 ******** 2020-03-12T03:16:52.900 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-12T03:16:52.902 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:52.938 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:52 +0000 (0:00:00.063) 0:02:13.545 ******** 2020-03-12T03:16:52.963 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-12T03:16:52.964 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:53.003 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:53 +0000 (0:00:00.064) 0:02:13.610 ******** 2020-03-12T03:16:53.028 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-12T03:16:53.030 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:53.068 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:53 +0000 (0:00:00.064) 0:02:13.675 ******** 2020-03-12T03:16:53.104 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-12T03:16:53.105 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:53.143 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:53 +0000 (0:00:00.075) 0:02:13.750 ******** 2020-03-12T03:16:53.169 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-12T03:16:53.169 INFO:teuthology.task.ansible.out:skipping: [smithi180.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-12T03:16:53.229 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:53 +0000 (0:00:00.085) 0:02:13.836 ******** 2020-03-12T03:16:53.452 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-12T03:16:53.453 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004809", "end": "2020-03-12 03:16:53.422590", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-12 03:16:53.417781", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:53.645 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003066", "end": "2020-03-12 03:16:53.606695", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-12 03:16:53.603629", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:53.831 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002858", "end": "2020-03-12 03:16:53.791552", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-12 03:16:53.788694", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:54.036 INFO:teuthology.task.ansible.out:ok: [smithi180.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.025100", "end": "2020-03-12 03:16:54.002772", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-12 03:16:53.977672", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-12T03:16:54.075 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:54 +0000 (0:00:00.846) 0:02:14.682 ******** 2020-03-12T03:16:54.121 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-12T03:16:54.164 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:54 +0000 (0:00:00.088) 0:02:14.771 ******** 2020-03-12T03:16:54.365 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-12T03:16:54.366 INFO:teuthology.task.ansible.out:changed: [smithi180.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-12T03:16:54.470 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-03-12T03:16:54.473 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-03-12T03:16:54.474 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-12T03:16:54.475 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi180.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 2020-03-12T03:16:54.476 INFO:teuthology.task.ansible.out:Thursday 12 March 2020 03:16:54 +0000 (0:00:00.312) 0:02:15.083 ******** =============================================================================== 2020-03-12T03:16:54.477 INFO:teuthology.task.ansible.out:common : nrpe - Remove SELinux policy package -------------------------- 16.54s common : nrpe - Load SELinux policy package ---------------------------- 16.46s testnode : Zap all non-root disks --------------------------------------- 8.14s testnode : Install packages --------------------------------------------- 7.81s testnode : Base group workaround for BZ1782899 -------------------------- 5.19s Gathering Facts --------------------------------------------------------- 4.16s common : Install Kerberos Packages (RedHat) ----------------------------- 3.34s testnode : Ensure ceph dependency packages are not present. ------------- 3.01s testnode : Ensure ceph packages are not present. ------------------------ 2.63s testnode : Create logical volume(s) ------------------------------------- 1.97s testnode : Rebuild rpmdb ------------------------------------------------ 1.90s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.77s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.55s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.52s testnode : Configure local mirrorlists ---------------------------------- 1.49s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.45s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.38s testnode : Install packages via pip ------------------------------------- 1.27s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 1.14s common : Configure epel repos in /etc/yum.repos.d/ ---------------------- 1.06s 2020-03-12T03:16:54.743 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi180.front.sepia.ceph.com')] 2020-03-12T03:16:54.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-12T03:16:54.746 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi180.front.sepia.ceph.com', 'timeout': 60} 2020-03-12T03:16:54.817 INFO:teuthology.orchestra.run.smithi180:> true 2020-03-12T03:16:54.905 INFO:teuthology.run_tasks:Running task clock... 2020-03-12T03:16:54.920 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-12T03:16:54.920 INFO:teuthology.orchestra.run.smithi180:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-12T03:16:54.979 INFO:teuthology.orchestra.run.smithi180.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-12T03:16:54.996 INFO:teuthology.orchestra.run.smithi180.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-12T03:16:55.041 INFO:teuthology.orchestra.run.smithi180.stderr:sudo: ntpd: command not found 2020-03-12T03:16:55.055 INFO:teuthology.orchestra.run.smithi180.stdout:506 Cannot talk to daemon 2020-03-12T03:16:55.072 INFO:teuthology.orchestra.run.smithi180.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-12T03:16:55.089 INFO:teuthology.orchestra.run.smithi180.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-12T03:16:55.152 INFO:teuthology.orchestra.run.smithi180.stderr:bash: ntpq: command not found 2020-03-12T03:16:55.155 INFO:teuthology.orchestra.run.smithi180.stdout:210 Number of sources = 4 2020-03-12T03:16:55.155 INFO:teuthology.orchestra.run.smithi180.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-12T03:16:55.155 INFO:teuthology.orchestra.run.smithi180.stdout:=============================================================================== 2020-03-12T03:16:55.156 INFO:teuthology.orchestra.run.smithi180.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-12T03:16:55.156 INFO:teuthology.orchestra.run.smithi180.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-12T03:16:55.156 INFO:teuthology.orchestra.run.smithi180.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-12T03:16:55.156 INFO:teuthology.orchestra.run.smithi180.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-12T03:16:55.156 INFO:teuthology.run_tasks:Running task install... 2020-03-12T03:16:55.167 DEBUG:teuthology.task.install:project ceph 2020-03-12T03:16:55.167 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-12T03:16:55.167 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-12T03:16:55.168 INFO:teuthology.task.install:Using flavor: basic 2020-03-12T03:16:55.175 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-12T03:16:55.175 INFO:teuthology.task.install:extra packages: [] 2020-03-12T03:16:55.175 INFO:teuthology.orchestra.run.smithi180:> sudo lsb_release -is 2020-03-12T03:16:55.231 INFO:teuthology.orchestra.run.smithi180.stdout:CentOS 2020-03-12T03:16:55.232 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-12T03:16:55.233 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-12T03:16:55.233 INFO:teuthology.orchestra.run.smithi180:> sudo lsb_release -is 2020-03-12T03:16:55.307 INFO:teuthology.orchestra.run.smithi180.stdout:CentOS 2020-03-12T03:16:55.308 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-12T03:16:55.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-12T03:16:55.552 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-12T03:16:55.552 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-12T03:16:55.731 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-12T03:16:55.731 INFO:teuthology.orchestra.run.smithi180:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-12T03:16:55.776 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-12T03:16:55.777 INFO:teuthology.orchestra.run.smithi180:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-12T03:16:55.845 INFO:teuthology.orchestra.run.smithi180:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-12T03:16:55.926 INFO:teuthology.orchestra.run.smithi180:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-12T03:16:55.993 INFO:teuthology.orchestra.run.smithi180.stdout:check_obsoletes = 1 2020-03-12T03:16:55.995 INFO:teuthology.orchestra.run.smithi180:> sudo yum clean all 2020-03-12T03:16:56.336 INFO:teuthology.orchestra.run.smithi180.stdout:239 files removed 2020-03-12T03:16:56.365 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-radosgw 2020-03-12T03:16:57.032 INFO:teuthology.orchestra.run.smithi180.stdout:ceph packages for x86_64 211 kB/s | 69 kB 00:00 2020-03-12T03:16:57.342 INFO:teuthology.orchestra.run.smithi180.stdout:ceph noarch packages 48 kB/s | 14 kB 00:00 2020-03-12T03:16:57.663 INFO:teuthology.orchestra.run.smithi180.stdout:ceph source packages 4.7 kB/s | 1.4 kB 00:00 2020-03-12T03:16:57.814 INFO:teuthology.orchestra.run.smithi180.stdout:CentOS-8 - AppStream 49 MB/s | 6.5 MB 00:00 2020-03-12T03:16:58.990 INFO:teuthology.orchestra.run.smithi180.stdout:CentOS-8 - Base 42 MB/s | 5.0 MB 00:00 2020-03-12T03:16:59.785 INFO:teuthology.orchestra.run.smithi180.stdout:CentOS-8 - Extras 33 kB/s | 2.1 kB 00:00 2020-03-12T03:16:59.945 INFO:teuthology.orchestra.run.smithi180.stdout:Copr repo for ceph-el8 owned by ktdreyer 330 kB/s | 38 kB 00:00 2020-03-12T03:17:00.360 INFO:teuthology.orchestra.run.smithi180.stdout:Extra Packages for Enterprise Linux 16 MB/s | 6.0 MB 00:00 2020-03-12T03:17:01.597 INFO:teuthology.orchestra.run.smithi180.stdout:lab-extras 5.6 MB/s | 30 kB 00:00 2020-03-12T03:17:02.560 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:17:02.563 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:02.563 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repository Size 2020-03-12T03:17:02.563 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:02.564 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:17:02.564 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-12T03:17:02.564 INFO:teuthology.orchestra.run.smithi180.stdout:Upgrading: 2020-03-12T03:17:02.564 INFO:teuthology.orchestra.run.smithi180.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-12T03:17:02.564 INFO:teuthology.orchestra.run.smithi180.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-12T03:17:02.564 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:17:02.564 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-12T03:17:02.565 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-12T03:17:02.565 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-12T03:17:02.565 INFO:teuthology.orchestra.run.smithi180.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-12T03:17:02.565 INFO:teuthology.orchestra.run.smithi180.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-12T03:17:02.565 INFO:teuthology.orchestra.run.smithi180.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-12T03:17:02.565 INFO:teuthology.orchestra.run.smithi180.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-12T03:17:02.565 INFO:teuthology.orchestra.run.smithi180.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-12T03:17:02.566 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-12T03:17:02.566 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-12T03:17:02.566 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-12T03:17:02.566 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-12T03:17:02.566 INFO:teuthology.orchestra.run.smithi180.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-12T03:17:02.567 INFO:teuthology.orchestra.run.smithi180.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-12T03:17:02.567 INFO:teuthology.orchestra.run.smithi180.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-12T03:17:02.567 INFO:teuthology.orchestra.run.smithi180.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-12T03:17:02.567 INFO:teuthology.orchestra.run.smithi180.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-12T03:17:02.567 INFO:teuthology.orchestra.run.smithi180.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-12T03:17:02.567 INFO:teuthology.orchestra.run.smithi180.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-12T03:17:02.567 INFO:teuthology.orchestra.run.smithi180.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-12T03:17:02.568 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:02.568 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:17:02.568 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:02.568 INFO:teuthology.orchestra.run.smithi180.stdout:Install 21 Packages 2020-03-12T03:17:02.568 INFO:teuthology.orchestra.run.smithi180.stdout:Upgrade 2 Packages 2020-03-12T03:17:02.568 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:02.569 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 51 M 2020-03-12T03:17:02.569 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:17:04.085 INFO:teuthology.orchestra.run.smithi180.stdout:(1/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 3.6 MB/s | 5.5 MB 00:01 2020-03-12T03:17:04.131 INFO:teuthology.orchestra.run.smithi180.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 531 kB/s | 25 kB 00:00 2020-03-12T03:17:04.213 INFO:teuthology.orchestra.run.smithi180.stdout:(3/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 6.1 MB/s | 507 kB 00:00 2020-03-12T03:17:04.288 INFO:teuthology.orchestra.run.smithi180.stdout:(4/23): libradosstriper1-15.1.0-1941.g4acc112.e 4.9 MB/s | 377 kB 00:00 2020-03-12T03:17:04.574 INFO:teuthology.orchestra.run.smithi180.stdout:(5/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 4.9 MB/s | 9.8 MB 00:02 2020-03-12T03:17:04.628 INFO:teuthology.orchestra.run.smithi180.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 741 kB/s | 40 kB 00:00 2020-03-12T03:17:04.669 INFO:teuthology.orchestra.run.smithi180.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 1.2 MB/s | 49 kB 00:00 2020-03-12T03:17:04.716 INFO:teuthology.orchestra.run.smithi180.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 3.4 MB/s | 162 kB 00:00 2020-03-12T03:17:04.772 INFO:teuthology.orchestra.run.smithi180.stdout:(9/23): python3-rados-15.1.0-1941.g4acc112.el8. 6.1 MB/s | 352 kB 00:00 2020-03-12T03:17:04.829 INFO:teuthology.orchestra.run.smithi180.stdout:(10/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 5.4 MB/s | 309 kB 00:00 2020-03-12T03:17:04.880 INFO:teuthology.orchestra.run.smithi180.stdout:(11/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 2.2 MB/s | 113 kB 00:00 2020-03-12T03:17:04.926 INFO:teuthology.orchestra.run.smithi180.stdout:(12/23): librgw2-15.1.0-1941.g4acc112.el8.x86_6 9.8 MB/s | 6.2 MB 00:00 2020-03-12T03:17:04.935 INFO:teuthology.orchestra.run.smithi180.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 6.3 MB/s | 353 kB 00:00 2020-03-12T03:17:04.937 INFO:teuthology.orchestra.run.smithi180.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 31 MB/s | 44 kB 00:00 2020-03-12T03:17:04.938 INFO:teuthology.orchestra.run.smithi180.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 33 MB/s | 47 kB 00:00 2020-03-12T03:17:04.958 INFO:teuthology.orchestra.run.smithi180.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.1 MB/s | 259 kB 00:00 2020-03-12T03:17:05.020 INFO:teuthology.orchestra.run.smithi180.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 952 kB/s | 59 kB 00:00 2020-03-12T03:17:05.047 INFO:teuthology.orchestra.run.smithi180.stdout:(18/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.8 MB/s | 315 kB 00:00 2020-03-12T03:17:05.050 INFO:teuthology.orchestra.run.smithi180.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.4 MB/s | 75 kB 00:00 2020-03-12T03:17:05.052 INFO:teuthology.orchestra.run.smithi180.stdout:(20/23): leveldb-1.21-2.el8.x86_64.rpm 38 MB/s | 179 kB 00:00 2020-03-12T03:17:05.340 INFO:teuthology.orchestra.run.smithi180.stdout:(21/23): ceph-common-15.1.0-1941.g4acc112.el8.x 7.5 MB/s | 21 MB 00:02 2020-03-12T03:17:05.350 INFO:teuthology.orchestra.run.smithi180.stdout:(22/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 6.4 MB/s | 1.9 MB 00:00 2020-03-12T03:17:05.390 INFO:teuthology.orchestra.run.smithi180.stdout:(23/23): librados2-15.1.0-1941.g4acc112.el8.x86 11 MB/s | 3.6 MB 00:00 2020-03-12T03:17:05.392 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:17:05.393 INFO:teuthology.orchestra.run.smithi180.stdout:Total 18 MB/s | 51 MB 00:02 2020-03-12T03:17:05.399 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:17:05.441 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:17:05.441 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:17:05.820 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:17:05.820 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:17:06.881 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:17:07.064 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-12T03:17:07.236 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-12T03:17:07.633 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-12T03:17:07.764 INFO:teuthology.orchestra.run.smithi180.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-12T03:17:07.854 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-12T03:17:08.058 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-12T03:17:08.189 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-12T03:17:08.425 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-12T03:17:08.663 INFO:teuthology.orchestra.run.smithi180.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-12T03:17:08.735 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-12T03:17:08.905 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-12T03:17:09.030 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-12T03:17:09.656 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-12T03:17:09.797 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-12T03:17:09.865 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-12T03:17:09.997 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-12T03:17:10.133 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-12T03:17:10.326 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-12T03:17:10.555 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-12T03:17:10.688 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-12T03:17:10.760 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-12T03:17:10.942 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-12T03:17:11.138 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-12T03:17:11.304 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-12T03:17:11.453 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-12T03:17:11.613 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-12T03:17:15.200 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-12T03:17:15.396 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-12T03:17:16.079 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-12T03:17:16.376 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-12T03:17:16.463 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-12T03:17:16.587 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-12T03:17:33.036 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-12T03:17:33.036 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /sys 2020-03-12T03:17:33.037 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /proc 2020-03-12T03:17:33.037 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /dev 2020-03-12T03:17:33.037 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /run 2020-03-12T03:17:33.037 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /mnt 2020-03-12T03:17:33.037 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /var/tmp 2020-03-12T03:17:33.037 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /home 2020-03-12T03:17:33.037 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /tmp 2020-03-12T03:17:33.038 INFO:teuthology.orchestra.run.smithi180.stdout:skipping the directory /dev 2020-03-12T03:17:33.038 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:34.310 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-12T03:17:34.425 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-12T03:17:34.425 INFO:teuthology.orchestra.run.smithi180.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-12T03:17:34.603 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-12T03:17:34.603 INFO:teuthology.orchestra.run.smithi180.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-12T03:17:35.237 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-12T03:17:35.238 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-12T03:17:35.238 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-12T03:17:35.238 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-12T03:17:35.238 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-12T03:17:35.238 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-12T03:17:35.239 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-12T03:17:35.240 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-12T03:17:35.240 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-12T03:17:35.240 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-12T03:17:35.240 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-12T03:17:35.240 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-12T03:17:35.240 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-12T03:17:35.240 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-12T03:17:35.241 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-12T03:17:35.241 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-12T03:17:35.241 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-12T03:17:35.241 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-12T03:17:35.869 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-12T03:17:35.869 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:35.869 INFO:teuthology.orchestra.run.smithi180.stdout:Upgraded: 2020-03-12T03:17:35.869 INFO:teuthology.orchestra.run.smithi180.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.869 INFO:teuthology.orchestra.run.smithi180.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.870 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:35.870 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:17:35.870 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.870 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.870 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.870 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.870 INFO:teuthology.orchestra.run.smithi180.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi180.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi180.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi180.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi180.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.871 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-12T03:17:35.872 INFO:teuthology.orchestra.run.smithi180.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-12T03:17:35.873 INFO:teuthology.orchestra.run.smithi180.stdout: leveldb-1.21-2.el8.x86_64 2020-03-12T03:17:35.873 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:35.873 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:17:36.192 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-test 2020-03-12T03:17:36.600 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:00:35 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:17:36.958 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:17:36.959 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:36.959 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repository Size 2020-03-12T03:17:36.959 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:36.959 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:17:36.960 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-12T03:17:36.960 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:17:36.960 INFO:teuthology.orchestra.run.smithi180.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-12T03:17:36.960 INFO:teuthology.orchestra.run.smithi180.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-12T03:17:36.960 INFO:teuthology.orchestra.run.smithi180.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-12T03:17:36.960 INFO:teuthology.orchestra.run.smithi180.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-12T03:17:36.961 INFO:teuthology.orchestra.run.smithi180.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-12T03:17:36.961 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:36.961 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:17:36.961 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:36.961 INFO:teuthology.orchestra.run.smithi180.stdout:Install 6 Packages 2020-03-12T03:17:36.961 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:36.962 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 43 M 2020-03-12T03:17:36.962 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 184 M 2020-03-12T03:17:36.962 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:17:36.994 INFO:teuthology.orchestra.run.smithi180.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.8 MB/s | 161 kB 00:00 2020-03-12T03:17:37.004 INFO:teuthology.orchestra.run.smithi180.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2020-03-12T03:17:37.005 INFO:teuthology.orchestra.run.smithi180.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 26 MB/s | 298 kB 00:00 2020-03-12T03:17:37.009 INFO:teuthology.orchestra.run.smithi180.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 47 MB/s | 249 kB 00:00 2020-03-12T03:17:37.082 INFO:teuthology.orchestra.run.smithi180.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 905 kB/s | 69 kB 00:00 2020-03-12T03:17:39.061 INFO:teuthology.orchestra.run.smithi180.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 20 MB/s | 42 MB 00:02 2020-03-12T03:17:39.064 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:17:39.065 INFO:teuthology.orchestra.run.smithi180.stdout:Total 20 MB/s | 43 MB 00:02 2020-03-12T03:17:39.071 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:17:39.092 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:17:39.093 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:17:39.362 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:17:39.362 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:17:40.119 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:17:40.543 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-12T03:17:40.721 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-12T03:17:40.930 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-12T03:17:41.066 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-12T03:17:41.136 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-12T03:17:45.140 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-12T03:17:46.455 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-12T03:17:46.961 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-12T03:17:46.961 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-12T03:17:46.961 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-12T03:17:46.961 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-12T03:17:46.962 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-12T03:17:46.962 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-12T03:17:47.231 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-12T03:17:47.232 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:47.232 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:17:47.232 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-12T03:17:47.232 INFO:teuthology.orchestra.run.smithi180.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-12T03:17:47.233 INFO:teuthology.orchestra.run.smithi180.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-12T03:17:47.233 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:47.233 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:17:47.511 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph 2020-03-12T03:17:47.910 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:00:46 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:17:48.330 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:17:48.334 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:48.334 INFO:teuthology.orchestra.run.smithi180.stdout: Package ArchVersion Repository Size 2020-03-12T03:17:48.334 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:48.335 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:17:48.335 INFO:teuthology.orchestra.run.smithi180.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-12T03:17:48.335 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:17:48.335 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-12T03:17:48.336 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-12T03:17:48.336 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-12T03:17:48.336 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-12T03:17:48.336 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-12T03:17:48.336 INFO:teuthology.orchestra.run.smithi180.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-12T03:17:48.337 INFO:teuthology.orchestra.run.smithi180.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-12T03:17:48.337 INFO:teuthology.orchestra.run.smithi180.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-12T03:17:48.337 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-12T03:17:48.337 INFO:teuthology.orchestra.run.smithi180.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-12T03:17:48.338 INFO:teuthology.orchestra.run.smithi180.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-12T03:17:48.338 INFO:teuthology.orchestra.run.smithi180.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-12T03:17:48.338 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-12T03:17:48.338 INFO:teuthology.orchestra.run.smithi180.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-12T03:17:48.339 INFO:teuthology.orchestra.run.smithi180.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-12T03:17:48.339 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-12T03:17:48.339 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-12T03:17:48.339 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-12T03:17:48.339 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-12T03:17:48.340 INFO:teuthology.orchestra.run.smithi180.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-12T03:17:48.340 INFO:teuthology.orchestra.run.smithi180.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-12T03:17:48.340 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-12T03:17:48.340 INFO:teuthology.orchestra.run.smithi180.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-12T03:17:48.341 INFO:teuthology.orchestra.run.smithi180.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-12T03:17:48.341 INFO:teuthology.orchestra.run.smithi180.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-12T03:17:48.341 INFO:teuthology.orchestra.run.smithi180.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-12T03:17:48.341 INFO:teuthology.orchestra.run.smithi180.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-12T03:17:48.341 INFO:teuthology.orchestra.run.smithi180.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-12T03:17:48.342 INFO:teuthology.orchestra.run.smithi180.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-12T03:17:48.342 INFO:teuthology.orchestra.run.smithi180.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-12T03:17:48.342 INFO:teuthology.orchestra.run.smithi180.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-12T03:17:48.342 INFO:teuthology.orchestra.run.smithi180.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-12T03:17:48.343 INFO:teuthology.orchestra.run.smithi180.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-12T03:17:48.343 INFO:teuthology.orchestra.run.smithi180.stdout:Installing weak dependencies: 2020-03-12T03:17:48.343 INFO:teuthology.orchestra.run.smithi180.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-12T03:17:48.343 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-12T03:17:48.344 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:48.344 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:17:48.344 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:17:48.344 INFO:teuthology.orchestra.run.smithi180.stdout:Install 36 Packages 2020-03-12T03:17:48.344 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:17:48.345 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 29 M 2020-03-12T03:17:48.345 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 105 M 2020-03-12T03:17:48.345 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:17:48.473 INFO:teuthology.orchestra.run.smithi180.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 48 kB/s | 6.4 kB 00:00 2020-03-12T03:17:48.702 INFO:teuthology.orchestra.run.smithi180.stdout:(2/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 3.9 MB/s | 1.4 MB 00:00 2020-03-12T03:17:48.843 INFO:teuthology.orchestra.run.smithi180.stdout:(3/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 4.0 MB/s | 2.0 MB 00:00 2020-03-12T03:17:48.922 INFO:teuthology.orchestra.run.smithi180.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.1 MB/s | 165 kB 00:00 2020-03-12T03:17:48.962 INFO:teuthology.orchestra.run.smithi180.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 5.3 MB/s | 214 kB 00:00 2020-03-12T03:17:48.976 INFO:teuthology.orchestra.run.smithi180.stdout:(6/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_6 8.7 MB/s | 4.4 MB 00:00 2020-03-12T03:17:48.994 INFO:teuthology.orchestra.run.smithi180.stdout:(7/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 48 MB/s | 1.5 MB 00:00 2020-03-12T03:17:48.995 INFO:teuthology.orchestra.run.smithi180.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 29 MB/s | 39 kB 00:00 2020-03-12T03:17:48.996 INFO:teuthology.orchestra.run.smithi180.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2020-03-12T03:17:48.998 INFO:teuthology.orchestra.run.smithi180.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 25 MB/s | 27 kB 00:00 2020-03-12T03:17:49.004 INFO:teuthology.orchestra.run.smithi180.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 71 MB/s | 457 kB 00:00 2020-03-12T03:17:49.009 INFO:teuthology.orchestra.run.smithi180.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 56 MB/s | 195 kB 00:00 2020-03-12T03:17:49.010 INFO:teuthology.orchestra.run.smithi180.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 24 MB/s | 34 kB 00:00 2020-03-12T03:17:49.010 INFO:teuthology.orchestra.run.smithi180.stdout:(14/36): python3-mako-1.0.6-13.el8.noarch.rpm 4.5 MB/s | 157 kB 00:00 2020-03-12T03:17:49.011 INFO:teuthology.orchestra.run.smithi180.stdout:(15/36): python3-requests-2.20.0-2.1.el8_1.noar 60 MB/s | 123 kB 00:00 2020-03-12T03:17:49.013 INFO:teuthology.orchestra.run.smithi180.stdout:(16/36): python3-urllib3-1.24.2-2.el8.noarch.rp 55 MB/s | 176 kB 00:00 2020-03-12T03:17:49.067 INFO:teuthology.orchestra.run.smithi180.stdout:(17/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.6 MB/s | 145 kB 00:00 2020-03-12T03:17:49.075 INFO:teuthology.orchestra.run.smithi180.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.5 MB/s | 11 kB 00:00 2020-03-12T03:17:49.084 INFO:teuthology.orchestra.run.smithi180.stdout:(19/36): python3-cherrypy-18.4.0-1.el8.noarch.r 5.4 MB/s | 385 kB 00:00 2020-03-12T03:17:49.084 INFO:teuthology.orchestra.run.smithi180.stdout:(20/36): python3-jaraco-functools-2.0-4.el8.noa 1.8 MB/s | 18 kB 00:00 2020-03-12T03:17:49.092 INFO:teuthology.orchestra.run.smithi180.stdout:(21/36): python3-logutils-0.3.5-11.el8.noarch.r 5.6 MB/s | 49 kB 00:00 2020-03-12T03:17:49.094 INFO:teuthology.orchestra.run.smithi180.stdout:(22/36): python3-more-itertools-7.2.0-3.el8.noa 6.2 MB/s | 59 kB 00:00 2020-03-12T03:17:49.102 INFO:teuthology.orchestra.run.smithi180.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-03-12T03:17:49.104 INFO:teuthology.orchestra.run.smithi180.stdout:(24/36): python3-pecan-1.3.2-9.el8.noarch.rpm 23 MB/s | 283 kB 00:00 2020-03-12T03:17:49.111 INFO:teuthology.orchestra.run.smithi180.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 2.1 MB/s | 19 kB 00:00 2020-03-12T03:17:49.113 INFO:teuthology.orchestra.run.smithi180.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.8 MB/s | 24 kB 00:00 2020-03-12T03:17:49.119 INFO:teuthology.orchestra.run.smithi180.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.5 MB/s | 30 kB 00:00 2020-03-12T03:17:49.121 INFO:teuthology.orchestra.run.smithi180.stdout:(28/36): python3-trustme-0.5.2-4.el8.noarch.rpm 3.1 MB/s | 27 kB 00:00 2020-03-12T03:17:49.128 INFO:teuthology.orchestra.run.smithi180.stdout:(29/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.6 MB/s | 23 kB 00:00 2020-03-12T03:17:49.197 INFO:teuthology.orchestra.run.smithi180.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 582 kB/s | 44 kB 00:00 2020-03-12T03:17:49.216 INFO:teuthology.orchestra.run.smithi180.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 2.1 MB/s | 40 kB 00:00 2020-03-12T03:17:49.223 INFO:teuthology.orchestra.run.smithi180.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-12T03:17:49.255 INFO:teuthology.orchestra.run.smithi180.stdout:(33/36): python3-waitress-1.2.1-2.el8.1.noarch. 6.0 MB/s | 241 kB 00:00 2020-03-12T03:17:49.261 INFO:teuthology.orchestra.run.smithi180.stdout:(34/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.5 MB/s | 251 kB 00:00 2020-03-12T03:17:49.275 INFO:teuthology.orchestra.run.smithi180.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.2 MB/s | 85 kB 00:00 2020-03-12T03:17:49.549 INFO:teuthology.orchestra.run.smithi180.stdout:(36/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 20 MB/s | 17 MB 00:00 2020-03-12T03:17:49.553 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:17:49.553 INFO:teuthology.orchestra.run.smithi180.stdout:Total 24 MB/s | 29 MB 00:01 2020-03-12T03:17:49.573 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:17:49.610 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:17:49.611 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:17:49.941 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:17:49.941 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:17:51.213 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:17:51.392 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-12T03:17:51.561 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-12T03:17:51.783 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-12T03:17:51.915 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-12T03:17:52.105 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-12T03:17:52.247 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-12T03:17:52.429 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-12T03:17:52.552 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-12T03:17:52.682 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-12T03:17:52.834 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-12T03:17:52.970 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-12T03:17:53.177 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-12T03:17:53.350 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-12T03:17:53.531 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-12T03:17:53.796 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-12T03:17:54.018 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-12T03:17:54.231 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-12T03:17:54.451 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-12T03:17:54.661 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-12T03:17:54.901 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-12T03:17:55.206 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-12T03:17:55.435 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-12T03:17:55.619 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-12T03:17:55.774 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-12T03:17:55.902 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-12T03:17:56.128 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-12T03:17:56.320 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-12T03:17:56.558 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-12T03:17:56.895 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-12T03:17:57.136 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-12T03:17:57.468 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-12T03:17:57.643 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-12T03:17:59.074 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-12T03:17:59.263 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-12T03:17:59.656 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-12T03:17:59.780 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-12T03:17:59.974 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-12T03:18:00.121 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-12T03:18:00.140 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-12T03:18:00.253 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-12T03:18:00.591 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-12T03:18:00.591 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-12T03:18:00.592 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-12T03:18:00.592 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-12T03:18:00.592 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-12T03:18:00.592 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-12T03:18:00.592 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-12T03:18:00.592 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-12T03:18:00.592 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-12T03:18:00.593 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-12T03:18:00.593 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-12T03:18:00.593 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-12T03:18:00.593 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-12T03:18:00.593 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-12T03:18:00.593 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-12T03:18:00.593 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-12T03:18:00.594 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-12T03:18:00.595 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-12T03:18:00.596 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-12T03:18:00.596 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-12T03:18:00.596 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-12T03:18:00.596 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-12T03:18:01.321 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-12T03:18:01.321 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:01.321 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:01.321 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:01.321 INFO:teuthology.orchestra.run.smithi180.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-12T03:18:01.321 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-12T03:18:01.321 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:01.322 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:01.322 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:01.322 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:01.322 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-12T03:18:01.322 INFO:teuthology.orchestra.run.smithi180.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-12T03:18:01.322 INFO:teuthology.orchestra.run.smithi180.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-12T03:18:01.322 INFO:teuthology.orchestra.run.smithi180.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-12T03:18:01.323 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-12T03:18:01.323 INFO:teuthology.orchestra.run.smithi180.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-12T03:18:01.323 INFO:teuthology.orchestra.run.smithi180.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-12T03:18:01.323 INFO:teuthology.orchestra.run.smithi180.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-12T03:18:01.323 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-12T03:18:01.323 INFO:teuthology.orchestra.run.smithi180.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-12T03:18:01.323 INFO:teuthology.orchestra.run.smithi180.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-12T03:18:01.324 INFO:teuthology.orchestra.run.smithi180.stdout: python3-portend-2.6-1.el8.noarch 2020-03-12T03:18:01.325 INFO:teuthology.orchestra.run.smithi180.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-12T03:18:01.325 INFO:teuthology.orchestra.run.smithi180.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-12T03:18:01.325 INFO:teuthology.orchestra.run.smithi180.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-12T03:18:01.325 INFO:teuthology.orchestra.run.smithi180.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-12T03:18:01.325 INFO:teuthology.orchestra.run.smithi180.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-12T03:18:01.325 INFO:teuthology.orchestra.run.smithi180.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-12T03:18:01.325 INFO:teuthology.orchestra.run.smithi180.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-12T03:18:01.326 INFO:teuthology.orchestra.run.smithi180.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-12T03:18:01.326 INFO:teuthology.orchestra.run.smithi180.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-12T03:18:01.326 INFO:teuthology.orchestra.run.smithi180.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-12T03:18:01.326 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:01.326 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:01.629 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-base 2020-03-12T03:18:02.033 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:01 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:02.323 INFO:teuthology.orchestra.run.smithi180.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:02.431 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:02.432 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:02.432 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:02.561 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install cephadm 2020-03-12T03:18:02.956 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:01 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:03.331 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:03.332 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:03.333 INFO:teuthology.orchestra.run.smithi180.stdout: Package Architecture Version Repository Size 2020-03-12T03:18:03.333 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:03.333 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:03.333 INFO:teuthology.orchestra.run.smithi180.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-12T03:18:03.333 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:03.334 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:03.334 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:03.334 INFO:teuthology.orchestra.run.smithi180.stdout:Install 1 Package 2020-03-12T03:18:03.334 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:03.335 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 36 k 2020-03-12T03:18:03.335 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 129 k 2020-03-12T03:18:03.335 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:03.496 INFO:teuthology.orchestra.run.smithi180.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 218 kB/s | 36 kB 00:00 2020-03-12T03:18:03.496 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:03.497 INFO:teuthology.orchestra.run.smithi180.stdout:Total 216 kB/s | 36 kB 00:00 2020-03-12T03:18:03.499 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:03.509 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:03.510 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:03.510 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:03.510 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:03.548 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:04.815 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:05.055 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:05.280 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:05.280 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:05.280 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:05.280 INFO:teuthology.orchestra.run.smithi180.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:05.280 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:05.280 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:05.411 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-immutable-object-cache 2020-03-12T03:18:05.818 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:04 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:06.186 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:06.186 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:06.187 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repo Size 2020-03-12T03:18:06.187 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:06.187 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:06.187 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-12T03:18:06.187 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:06.187 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:06.187 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:06.188 INFO:teuthology.orchestra.run.smithi180.stdout:Install 1 Package 2020-03-12T03:18:06.188 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:06.188 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 203 k 2020-03-12T03:18:06.188 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 743 k 2020-03-12T03:18:06.189 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:06.480 INFO:teuthology.orchestra.run.smithi180.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 694 kB/s | 203 kB 00:00 2020-03-12T03:18:06.481 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:06.481 INFO:teuthology.orchestra.run.smithi180.stdout:Total 692 kB/s | 203 kB 00:00 2020-03-12T03:18:06.483 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:06.495 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:06.495 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:06.541 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:06.541 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:06.681 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:06.812 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-12T03:18:07.126 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-12T03:18:07.360 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-12T03:18:07.360 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:07.360 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:07.360 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:07.361 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:07.361 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:07.678 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-mgr 2020-03-12T03:18:08.076 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:07 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:08.391 INFO:teuthology.orchestra.run.smithi180.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:08.498 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:08.499 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:08.499 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:08.674 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-mgr-dashboard 2020-03-12T03:18:09.073 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:08 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:09.436 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:09.437 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:09.437 INFO:teuthology.orchestra.run.smithi180.stdout: Package ArchVersion Repository Size 2020-03-12T03:18:09.437 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:09.437 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:09.438 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:09.439 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:09.439 INFO:teuthology.orchestra.run.smithi180.stdout:Install 5 Packages 2020-03-12T03:18:09.439 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:09.439 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 4.2 M 2020-03-12T03:18:09.440 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 19 M 2020-03-12T03:18:09.440 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:09.470 INFO:teuthology.orchestra.run.smithi180.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2020-03-12T03:18:09.521 INFO:teuthology.orchestra.run.smithi180.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 667 kB/s | 34 kB 00:00 2020-03-12T03:18:09.545 INFO:teuthology.orchestra.run.smithi180.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 8.0 MB/s | 197 kB 00:00 2020-03-12T03:18:09.603 INFO:teuthology.orchestra.run.smithi180.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 136 kB/s | 22 kB 00:00 2020-03-12T03:18:10.021 INFO:teuthology.orchestra.run.smithi180.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 6.6 MB/s | 3.9 MB 00:00 2020-03-12T03:18:10.024 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:10.024 INFO:teuthology.orchestra.run.smithi180.stdout:Total 7.1 MB/s | 4.2 MB 00:00 2020-03-12T03:18:10.029 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:10.037 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:10.037 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:10.170 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:10.170 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:10.401 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:10.601 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-12T03:18:10.855 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-12T03:18:11.283 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-12T03:18:12.013 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-12T03:18:12.311 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-12T03:18:12.513 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-12T03:18:12.513 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-12T03:18:12.513 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-12T03:18:12.514 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-12T03:18:12.514 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-12T03:18:12.762 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-12T03:18:12.762 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:12.762 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:12.762 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-12T03:18:12.762 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-12T03:18:12.762 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-12T03:18:12.762 INFO:teuthology.orchestra.run.smithi180.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-12T03:18:12.763 INFO:teuthology.orchestra.run.smithi180.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-12T03:18:12.763 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:12.763 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:12.898 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-12T03:18:13.296 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:12 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:14.205 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:14.206 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:14.206 INFO:teuthology.orchestra.run.smithi180.stdout: Package ArchVersion Repository Size 2020-03-12T03:18:14.206 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:14.207 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:14.207 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-12T03:18:14.207 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:18:14.207 INFO:teuthology.orchestra.run.smithi180.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-12T03:18:14.207 INFO:teuthology.orchestra.run.smithi180.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-12T03:18:14.207 INFO:teuthology.orchestra.run.smithi180.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-12T03:18:14.207 INFO:teuthology.orchestra.run.smithi180.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-12T03:18:14.208 INFO:teuthology.orchestra.run.smithi180.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-12T03:18:14.208 INFO:teuthology.orchestra.run.smithi180.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-12T03:18:14.208 INFO:teuthology.orchestra.run.smithi180.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-12T03:18:14.208 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:14.208 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:14.208 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:14.208 INFO:teuthology.orchestra.run.smithi180.stdout:Install 8 Packages 2020-03-12T03:18:14.209 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:14.209 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 37 M 2020-03-12T03:18:14.209 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 205 M 2020-03-12T03:18:14.209 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:14.378 INFO:teuthology.orchestra.run.smithi180.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 25 MB/s | 4.3 MB 00:00 2020-03-12T03:18:14.488 INFO:teuthology.orchestra.run.smithi180.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2020-03-12T03:18:14.525 INFO:teuthology.orchestra.run.smithi180.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 6.0 MB/s | 225 kB 00:00 2020-03-12T03:18:14.608 INFO:teuthology.orchestra.run.smithi180.stdout:(4/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2020-03-12T03:18:14.629 INFO:teuthology.orchestra.run.smithi180.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 30 MB/s | 638 kB 00:00 2020-03-12T03:18:14.713 INFO:teuthology.orchestra.run.smithi180.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 2.0 MB/s | 169 kB 00:00 2020-03-12T03:18:15.021 INFO:teuthology.orchestra.run.smithi180.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 29 MB/s | 15 MB 00:00 2020-03-12T03:18:15.386 INFO:teuthology.orchestra.run.smithi180.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 7.3 MB/s | 8.6 MB 00:01 2020-03-12T03:18:15.388 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:15.388 INFO:teuthology.orchestra.run.smithi180.stdout:Total 31 MB/s | 37 MB 00:01 2020-03-12T03:18:15.400 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:15.419 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:15.419 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:15.735 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:15.735 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:16.259 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:16.385 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-12T03:18:16.526 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-12T03:18:16.660 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-12T03:18:17.194 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-12T03:18:17.327 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-12T03:18:17.867 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-12T03:18:18.060 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-12T03:18:18.746 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-12T03:18:19.054 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-12T03:18:19.268 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-12T03:18:21.215 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-12T03:18:22.508 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-12T03:18:22.692 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-12T03:18:22.943 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-12T03:18:22.943 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-12T03:18:22.943 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-12T03:18:22.943 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-12T03:18:22.944 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-12T03:18:22.944 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-12T03:18:22.944 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-12T03:18:22.944 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-12T03:18:23.226 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-12T03:18:23.226 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:23.227 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:23.227 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-12T03:18:23.227 INFO:teuthology.orchestra.run.smithi180.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-12T03:18:23.227 INFO:teuthology.orchestra.run.smithi180.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-12T03:18:23.227 INFO:teuthology.orchestra.run.smithi180.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-12T03:18:23.227 INFO:teuthology.orchestra.run.smithi180.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-12T03:18:23.227 INFO:teuthology.orchestra.run.smithi180.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-12T03:18:23.228 INFO:teuthology.orchestra.run.smithi180.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-12T03:18:23.228 INFO:teuthology.orchestra.run.smithi180.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-12T03:18:23.228 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:23.228 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:23.353 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-mgr-rook 2020-03-12T03:18:23.754 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:22 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:24.137 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:24.139 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:24.139 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repository Size 2020-03-12T03:18:24.139 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:24.140 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:24.140 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-12T03:18:24.140 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-noarch 24 k 2020-03-12T03:18:24.140 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:18:24.140 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-12T03:18:24.141 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-12T03:18:24.141 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-12T03:18:24.141 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-12T03:18:24.141 INFO:teuthology.orchestra.run.smithi180.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-12T03:18:24.141 INFO:teuthology.orchestra.run.smithi180.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-12T03:18:24.142 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-12T03:18:24.142 INFO:teuthology.orchestra.run.smithi180.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-12T03:18:24.142 INFO:teuthology.orchestra.run.smithi180.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-12T03:18:24.142 INFO:teuthology.orchestra.run.smithi180.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-12T03:18:24.142 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-12T03:18:24.143 INFO:teuthology.orchestra.run.smithi180.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-12T03:18:24.143 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:24.143 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:24.143 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:24.144 INFO:teuthology.orchestra.run.smithi180.stdout:Install 13 Packages 2020-03-12T03:18:24.144 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:24.145 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 1.9 M 2020-03-12T03:18:24.145 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 23 M 2020-03-12T03:18:24.145 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:24.172 INFO:teuthology.orchestra.run.smithi180.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 889 kB/s | 27 kB 00:00 2020-03-12T03:18:24.175 INFO:teuthology.orchestra.run.smithi180.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 38 MB/s | 126 kB 00:00 2020-03-12T03:18:24.176 INFO:teuthology.orchestra.run.smithi180.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 586 kB/s | 20 kB 00:00 2020-03-12T03:18:24.178 INFO:teuthology.orchestra.run.smithi180.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 37 MB/s | 110 kB 00:00 2020-03-12T03:18:24.184 INFO:teuthology.orchestra.run.smithi180.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 19 MB/s | 155 kB 00:00 2020-03-12T03:18:24.185 INFO:teuthology.orchestra.run.smithi180.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 6.8 MB/s | 43 kB 00:00 2020-03-12T03:18:24.232 INFO:teuthology.orchestra.run.smithi180.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 702 kB/s | 33 kB 00:00 2020-03-12T03:18:24.242 INFO:teuthology.orchestra.run.smithi180.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 288 kB/s | 16 kB 00:00 2020-03-12T03:18:24.263 INFO:teuthology.orchestra.run.smithi180.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.6 MB/s | 81 kB 00:00 2020-03-12T03:18:24.279 INFO:teuthology.orchestra.run.smithi180.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.4 MB/s | 73 kB 00:00 2020-03-12T03:18:24.295 INFO:teuthology.orchestra.run.smithi180.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.8 MB/s | 61 kB 00:00 2020-03-12T03:18:24.330 INFO:teuthology.orchestra.run.smithi180.stdout:(12/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 130 kB/s | 24 kB 00:00 2020-03-12T03:18:24.375 INFO:teuthology.orchestra.run.smithi180.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 8.4 MB/s | 1.1 MB 00:00 2020-03-12T03:18:24.377 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:24.377 INFO:teuthology.orchestra.run.smithi180.stdout:Total 7.9 MB/s | 1.9 MB 00:00 2020-03-12T03:18:24.383 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:24.398 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:24.398 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:24.473 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:24.474 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:24.954 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:25.211 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-12T03:18:25.414 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-12T03:18:25.603 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-12T03:18:26.123 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-12T03:18:26.289 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-12T03:18:26.511 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-12T03:18:26.769 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-12T03:18:27.012 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-12T03:18:28.044 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-12T03:18:28.273 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-12T03:18:28.515 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-12T03:18:28.715 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-12T03:18:28.916 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-12T03:18:29.121 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-12T03:18:29.121 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-12T03:18:29.121 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-12T03:18:29.122 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-12T03:18:29.122 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-12T03:18:29.122 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-12T03:18:29.122 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-12T03:18:29.122 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-12T03:18:29.122 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-12T03:18:29.122 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-12T03:18:29.123 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-12T03:18:29.123 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-12T03:18:29.123 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-12T03:18:29.456 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-12T03:18:29.457 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:29.457 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:29.457 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-12T03:18:29.457 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-12T03:18:29.457 INFO:teuthology.orchestra.run.smithi180.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-12T03:18:29.458 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-12T03:18:29.458 INFO:teuthology.orchestra.run.smithi180.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-12T03:18:29.458 INFO:teuthology.orchestra.run.smithi180.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-12T03:18:29.458 INFO:teuthology.orchestra.run.smithi180.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-12T03:18:29.458 INFO:teuthology.orchestra.run.smithi180.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-12T03:18:29.458 INFO:teuthology.orchestra.run.smithi180.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-12T03:18:29.458 INFO:teuthology.orchestra.run.smithi180.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-12T03:18:29.459 INFO:teuthology.orchestra.run.smithi180.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-12T03:18:29.459 INFO:teuthology.orchestra.run.smithi180.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-12T03:18:29.459 INFO:teuthology.orchestra.run.smithi180.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-12T03:18:29.459 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:29.459 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:29.586 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-mgr-cephadm 2020-03-12T03:18:29.984 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:28 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:30.360 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:30.360 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:30.361 INFO:teuthology.orchestra.run.smithi180.stdout: Package ArchVersion Repository Size 2020-03-12T03:18:30.361 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:30.361 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:30.361 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-12T03:18:30.361 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:18:30.362 INFO:teuthology.orchestra.run.smithi180.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-12T03:18:30.362 INFO:teuthology.orchestra.run.smithi180.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-12T03:18:30.362 INFO:teuthology.orchestra.run.smithi180.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-12T03:18:30.362 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:30.363 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:30.363 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:30.363 INFO:teuthology.orchestra.run.smithi180.stdout:Install 4 Packages 2020-03-12T03:18:30.363 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:30.364 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 461 k 2020-03-12T03:18:30.364 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 1.4 M 2020-03-12T03:18:30.365 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:30.413 INFO:teuthology.orchestra.run.smithi180.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 968 kB/s | 48 kB 00:00 2020-03-12T03:18:30.430 INFO:teuthology.orchestra.run.smithi180.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.2 MB/s | 356 kB 00:00 2020-03-12T03:18:30.461 INFO:teuthology.orchestra.run.smithi180.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 443 kB/s | 21 kB 00:00 2020-03-12T03:18:30.558 INFO:teuthology.orchestra.run.smithi180.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 185 kB/s | 36 kB 00:00 2020-03-12T03:18:30.559 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:30.560 INFO:teuthology.orchestra.run.smithi180.stdout:Total 2.3 MB/s | 461 kB 00:00 2020-03-12T03:18:30.564 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:30.571 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:30.571 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:30.594 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:30.595 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:30.745 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:31.009 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-12T03:18:31.419 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-12T03:18:32.289 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-12T03:18:32.490 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-12T03:18:32.703 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-12T03:18:32.704 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-12T03:18:32.704 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-12T03:18:32.704 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-12T03:18:32.950 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-12T03:18:32.950 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:32.950 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:32.950 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-12T03:18:32.950 INFO:teuthology.orchestra.run.smithi180.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-12T03:18:32.950 INFO:teuthology.orchestra.run.smithi180.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-12T03:18:32.951 INFO:teuthology.orchestra.run.smithi180.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-12T03:18:32.951 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:32.951 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:33.078 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install ceph-fuse 2020-03-12T03:18:33.478 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:32 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:33.850 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:33.851 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:33.851 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repository Size 2020-03-12T03:18:33.851 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:33.851 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:33.851 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-12T03:18:33.852 INFO:teuthology.orchestra.run.smithi180.stdout:Installing dependencies: 2020-03-12T03:18:33.852 INFO:teuthology.orchestra.run.smithi180.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-12T03:18:33.852 INFO:teuthology.orchestra.run.smithi180.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-12T03:18:33.852 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:33.853 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:33.853 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:33.853 INFO:teuthology.orchestra.run.smithi180.stdout:Install 3 Packages 2020-03-12T03:18:33.853 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:33.854 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 676 k 2020-03-12T03:18:33.854 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 2.1 M 2020-03-12T03:18:33.855 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:33.884 INFO:teuthology.orchestra.run.smithi180.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2020-03-12T03:18:33.891 INFO:teuthology.orchestra.run.smithi180.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 545 kB/s | 21 kB 00:00 2020-03-12T03:18:34.145 INFO:teuthology.orchestra.run.smithi180.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.9 MB/s | 572 kB 00:00 2020-03-12T03:18:34.146 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:34.147 INFO:teuthology.orchestra.run.smithi180.stdout:Total 2.2 MB/s | 676 kB 00:00 2020-03-12T03:18:34.150 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:34.171 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:34.171 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:34.234 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:34.234 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:34.394 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:34.394 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-12T03:18:34.394 INFO:teuthology.orchestra.run.smithi180.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-12T03:18:34.394 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:34.789 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-12T03:18:34.997 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-12T03:18:35.304 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-12T03:18:35.304 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-12T03:18:35.304 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-12T03:18:35.658 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-12T03:18:35.658 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:35.658 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:35.659 INFO:teuthology.orchestra.run.smithi180.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-12T03:18:35.659 INFO:teuthology.orchestra.run.smithi180.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-12T03:18:35.659 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:35.659 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:35.930 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install librados-devel 2020-03-12T03:18:36.338 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:35 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:36.766 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:36.767 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:36.767 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repo Size 2020-03-12T03:18:36.767 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:36.767 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:36.768 INFO:teuthology.orchestra.run.smithi180.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-12T03:18:36.768 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:36.768 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:36.768 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:36.768 INFO:teuthology.orchestra.run.smithi180.stdout:Install 1 Package 2020-03-12T03:18:36.768 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:36.769 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 141 k 2020-03-12T03:18:36.769 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 487 k 2020-03-12T03:18:36.769 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:36.995 INFO:teuthology.orchestra.run.smithi180.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 621 kB/s | 141 kB 00:00 2020-03-12T03:18:36.995 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:36.996 INFO:teuthology.orchestra.run.smithi180.stdout:Total 616 kB/s | 141 kB 00:00 2020-03-12T03:18:36.998 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:37.005 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:37.005 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:37.054 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:37.054 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:37.164 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:37.371 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:37.895 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:38.149 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:38.149 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:38.149 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:38.150 INFO:teuthology.orchestra.run.smithi180.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:38.150 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:38.150 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:38.406 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install libcephfs2 2020-03-12T03:18:38.820 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:37 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:39.094 INFO:teuthology.orchestra.run.smithi180.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:39.198 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:39.200 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:39.200 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:39.361 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install libcephfs-devel 2020-03-12T03:18:39.763 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:38 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:40.134 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:40.135 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:40.135 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repo Size 2020-03-12T03:18:40.135 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:40.135 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:40.135 INFO:teuthology.orchestra.run.smithi180.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-12T03:18:40.136 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:40.136 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:40.136 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:40.136 INFO:teuthology.orchestra.run.smithi180.stdout:Install 1 Package 2020-03-12T03:18:40.136 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:40.137 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 21 k 2020-03-12T03:18:40.137 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 76 k 2020-03-12T03:18:40.137 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:40.385 INFO:teuthology.orchestra.run.smithi180.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 86 kB/s | 21 kB 00:00 2020-03-12T03:18:40.386 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:40.386 INFO:teuthology.orchestra.run.smithi180.stdout:Total 86 kB/s | 21 kB 00:00 2020-03-12T03:18:40.386 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:40.395 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:40.396 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:40.396 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:40.396 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:40.465 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:40.795 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:41.029 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:41.030 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:41.030 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:41.030 INFO:teuthology.orchestra.run.smithi180.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:41.030 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:41.030 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:41.171 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install librados2 2020-03-12T03:18:41.569 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:40 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:41.826 INFO:teuthology.orchestra.run.smithi180.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:41.931 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:41.933 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:41.933 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:42.096 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install librbd1 2020-03-12T03:18:42.494 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:41 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:42.758 INFO:teuthology.orchestra.run.smithi180.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:42.865 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:42.866 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:42.866 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:42.993 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install python3-rados 2020-03-12T03:18:43.396 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:42 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:43.683 INFO:teuthology.orchestra.run.smithi180.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:43.788 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:43.789 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:43.789 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:43.916 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install python3-rgw 2020-03-12T03:18:44.315 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:43 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:44.583 INFO:teuthology.orchestra.run.smithi180.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:44.693 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:44.694 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:44.694 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:44.857 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install python3-cephfs 2020-03-12T03:18:45.257 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:44 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:45.524 INFO:teuthology.orchestra.run.smithi180.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:45.631 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:45.632 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:45.632 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:45.760 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install python3-rbd 2020-03-12T03:18:46.159 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:45 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:46.449 INFO:teuthology.orchestra.run.smithi180.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-12T03:18:46.555 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:46.556 INFO:teuthology.orchestra.run.smithi180.stdout:Nothing to do. 2020-03-12T03:18:46.556 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:46.699 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install rbd-fuse 2020-03-12T03:18:47.107 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:46 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:47.482 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:47.483 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:47.483 INFO:teuthology.orchestra.run.smithi180.stdout: Package Architecture Version Repository Size 2020-03-12T03:18:47.483 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:47.483 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:47.483 INFO:teuthology.orchestra.run.smithi180.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-12T03:18:47.483 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:47.484 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:47.484 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:47.484 INFO:teuthology.orchestra.run.smithi180.stdout:Install 1 Package 2020-03-12T03:18:47.484 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:47.484 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 72 k 2020-03-12T03:18:47.484 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 230 k 2020-03-12T03:18:47.484 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:47.756 INFO:teuthology.orchestra.run.smithi180.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 265 kB/s | 72 kB 00:00 2020-03-12T03:18:47.756 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:47.757 INFO:teuthology.orchestra.run.smithi180.stdout:Total 264 kB/s | 72 kB 00:00 2020-03-12T03:18:47.758 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:47.769 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:47.770 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:47.818 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:47.818 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:47.915 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:48.086 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:48.326 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:49.366 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:49.366 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:49.366 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:49.366 INFO:teuthology.orchestra.run.smithi180.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:49.367 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:49.367 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:49.645 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install rbd-mirror 2020-03-12T03:18:50.051 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:49 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:50.447 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:50.448 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:50.448 INFO:teuthology.orchestra.run.smithi180.stdout: Package Arch Version Repo Size 2020-03-12T03:18:50.448 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:50.448 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:50.448 INFO:teuthology.orchestra.run.smithi180.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-12T03:18:50.449 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:50.449 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:50.449 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:50.449 INFO:teuthology.orchestra.run.smithi180.stdout:Install 1 Package 2020-03-12T03:18:50.449 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:50.450 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 2.7 M 2020-03-12T03:18:50.450 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 12 M 2020-03-12T03:18:50.450 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:50.884 INFO:teuthology.orchestra.run.smithi180.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 6.2 MB/s | 2.7 MB 00:00 2020-03-12T03:18:50.885 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:50.885 INFO:teuthology.orchestra.run.smithi180.stdout:Total 6.2 MB/s | 2.7 MB 00:00 2020-03-12T03:18:50.887 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:50.900 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:50.900 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:50.958 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:50.958 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:51.327 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:51.529 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:51.858 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:52.090 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:52.090 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:52.090 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:52.090 INFO:teuthology.orchestra.run.smithi180.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:52.090 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:52.090 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:52.377 INFO:teuthology.orchestra.run.smithi180:> sudo yum -y install rbd-nbd 2020-03-12T03:18:52.787 INFO:teuthology.orchestra.run.smithi180.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Mar 2020 03:17:01 AM UTC. 2020-03-12T03:18:53.181 INFO:teuthology.orchestra.run.smithi180.stdout:Dependencies resolved. 2020-03-12T03:18:53.181 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:53.181 INFO:teuthology.orchestra.run.smithi180.stdout: Package Architecture Version Repository Size 2020-03-12T03:18:53.182 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:53.182 INFO:teuthology.orchestra.run.smithi180.stdout:Installing: 2020-03-12T03:18:53.182 INFO:teuthology.orchestra.run.smithi180.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-12T03:18:53.182 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:53.182 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction Summary 2020-03-12T03:18:53.182 INFO:teuthology.orchestra.run.smithi180.stdout:================================================================================ 2020-03-12T03:18:53.182 INFO:teuthology.orchestra.run.smithi180.stdout:Install 1 Package 2020-03-12T03:18:53.183 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:53.183 INFO:teuthology.orchestra.run.smithi180.stdout:Total download size: 141 k 2020-03-12T03:18:53.183 INFO:teuthology.orchestra.run.smithi180.stdout:Installed size: 526 k 2020-03-12T03:18:53.183 INFO:teuthology.orchestra.run.smithi180.stdout:Downloading Packages: 2020-03-12T03:18:53.410 INFO:teuthology.orchestra.run.smithi180.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 619 kB/s | 141 kB 00:00 2020-03-12T03:18:53.411 INFO:teuthology.orchestra.run.smithi180.stdout:-------------------------------------------------------------------------------- 2020-03-12T03:18:53.411 INFO:teuthology.orchestra.run.smithi180.stdout:Total 616 kB/s | 141 kB 00:00 2020-03-12T03:18:53.413 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction check 2020-03-12T03:18:53.424 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction check succeeded. 2020-03-12T03:18:53.424 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction test 2020-03-12T03:18:53.472 INFO:teuthology.orchestra.run.smithi180.stdout:Transaction test succeeded. 2020-03-12T03:18:53.472 INFO:teuthology.orchestra.run.smithi180.stdout:Running transaction 2020-03-12T03:18:53.583 INFO:teuthology.orchestra.run.smithi180.stdout: Preparing : 1/1 2020-03-12T03:18:53.743 INFO:teuthology.orchestra.run.smithi180.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:53.975 INFO:teuthology.orchestra.run.smithi180.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:54.205 INFO:teuthology.orchestra.run.smithi180.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-12T03:18:54.205 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:54.205 INFO:teuthology.orchestra.run.smithi180.stdout:Installed: 2020-03-12T03:18:54.206 INFO:teuthology.orchestra.run.smithi180.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-12T03:18:54.206 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-03-12T03:18:54.206 INFO:teuthology.orchestra.run.smithi180.stdout:Complete! 2020-03-12T03:18:54.543 DEBUG:teuthology.parallel:result is None 2020-03-12T03:18:54.543 INFO:teuthology.orchestra.run.smithi180:> sudo lsb_release -is 2020-03-12T03:18:54.579 INFO:teuthology.orchestra.run.smithi180.stdout:CentOS 2020-03-12T03:18:54.580 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-12T03:18:54.580 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-12T03:18:54.817 INFO:teuthology.orchestra.run.smithi180:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-12T03:18:54.836 INFO:teuthology.orchestra.run.smithi180.stdout:15.1.0-1941.g4acc112.el8 2020-03-12T03:18:54.836 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-12T03:18:54.836 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-12T03:18:54.837 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-12T03:18:54.843 INFO:teuthology.orchestra.run.smithi180:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-12T03:18:54.904 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-12T03:18:54.904 INFO:teuthology.orchestra.run.smithi180:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-12T03:18:54.972 INFO:teuthology.orchestra.run.smithi180:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-12T03:18:55.040 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-12T03:18:55.040 INFO:teuthology.orchestra.run.smithi180:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-12T03:18:55.127 INFO:teuthology.orchestra.run.smithi180:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-12T03:18:55.208 INFO:teuthology.run_tasks:Running task workunit... 2020-03-12T03:18:55.234 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-12T03:18:55.234 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-12T03:18:55.235 INFO:tasks.workunit:timeout=3h 2020-03-12T03:18:55.235 INFO:tasks.workunit:cleanup=True 2020-03-12T03:18:55.235 INFO:teuthology.orchestra.run.smithi180:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-12T03:18:55.262 INFO:teuthology.orchestra.run.smithi180.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-12T03:18:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-12T03:18:55.263 INFO:teuthology.orchestra.run.smithi180:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-12T03:18:55.316 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-12T03:18:55.316 INFO:teuthology.orchestra.run.smithi180:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-12T03:18:55.372 INFO:teuthology.orchestra.run.smithi180:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-12T03:18:56.095 INFO:tasks.workunit.client.0.smithi180.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-12T03:20:40.938 INFO:tasks.workunit.client.0.smithi180.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-12T03:20:40.939 INFO:tasks.workunit.client.0.smithi180.stderr: 2020-03-12T03:20:40.939 INFO:tasks.workunit.client.0.smithi180.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-12T03:20:40.939 INFO:tasks.workunit.client.0.smithi180.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-12T03:20:40.939 INFO:tasks.workunit.client.0.smithi180.stderr:state without impacting any branches by performing another checkout. 2020-03-12T03:20:40.939 INFO:tasks.workunit.client.0.smithi180.stderr: 2020-03-12T03:20:40.939 INFO:tasks.workunit.client.0.smithi180.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-12T03:20:40.939 INFO:tasks.workunit.client.0.smithi180.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-12T03:20:40.940 INFO:tasks.workunit.client.0.smithi180.stderr: 2020-03-12T03:20:40.940 INFO:tasks.workunit.client.0.smithi180.stderr: git checkout -b 2020-03-12T03:20:40.940 INFO:tasks.workunit.client.0.smithi180.stderr: 2020-03-12T03:20:40.940 INFO:tasks.workunit.client.0.smithi180.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-12T03:20:40.943 INFO:teuthology.orchestra.run.smithi180:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-12T03:20:41.014 DEBUG:teuthology.orchestra.remote:smithi180:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-12T03:20:41.061 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-12T03:20:41.061 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-12T03:20:41.061 INFO:teuthology.orchestra.run.smithi180:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-12T03:20:41.092 INFO:tasks.workunit.client.0.smithi180.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-12T03:20:41.101 INFO:tasks.workunit.client.0.smithi180.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-12T03:20:41.101 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-12T03:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-12T03:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-12T03:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-12T03:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-12T03:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-12T03:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-12T03:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-12T03:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-12T03:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-12T03:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-12T03:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-12T03:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-12T03:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-12T03:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-12T03:20:41.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-12T03:20:41.104 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-12T03:20:41.104 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-12T03:20:41.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-12T03:20:41.106 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-12T03:20:41.106 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-12T03:20:41.106 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-12T03:20:41.106 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-12T03:20:41.106 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-12T03:20:41.107 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-12T03:20:41.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-12T03:20:41.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-12T03:20:41.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-12T03:20:41.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-12T03:20:41.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-12T03:20:41.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-12T03:20:41.108 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-12T03:20:41.108 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-12T03:20:41.108 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-12T03:20:41.108 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-12T03:20:41.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none ' 2020-03-12T03:20:41.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-12T03:20:41.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-12T03:20:41.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-12T03:20:41.110 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:20:41.110 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-12T03:20:41.110 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-12T03:20:41.111 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-12T03:20:41.112 INFO:tasks.workunit.client.0.smithi180.stderr:TEST_divergent_2 2020-03-12T03:20:41.112 INFO:tasks.workunit.client.0.smithi180.stderr:TEST_divergent_3 2020-03-12T03:20:41.112 INFO:tasks.workunit.client.0.smithi180.stderr:TEST_divergent_ec' 2020-03-12T03:20:41.112 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-12T03:20:41.112 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-12T03:20:41.112 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-12T03:20:41.113 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-12T03:20:41.113 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-12T03:20:41.113 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-12T03:20:41.113 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-12T03:20:41.113 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:20:41.113 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:20:41.113 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:20:41.114 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:20:41.114 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:20:41.114 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:20:41.114 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-12T03:20:41.115 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-12T03:20:41.115 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-12T03:20:41.115 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-12T03:20:41.116 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-12T03:20:41.116 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-12T03:20:41.116 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:20:41.116 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-12T03:20:41.116 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-12T03:20:41.116 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:20:41.117 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-12T03:20:41.117 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-12T03:20:41.117 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-12T03:20:41.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-12T03:20:41.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:41.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:41.118 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.16628 2020-03-12T03:20:41.119 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-12T03:20:41.119 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-12T03:20:41.119 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-12T03:20:41.120 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-12T03:20:41.120 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:41.120 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:41.120 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.16628 2020-03-12T03:20:41.122 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-12T03:20:41.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-12T03:20:41.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-12T03:20:41.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-12T03:20:41.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-12T03:20:41.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-12T03:20:41.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-12T03:20:41.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-12T03:20:41.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-12T03:20:41.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-12T03:20:41.123 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-12T03:20:41.124 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-12T03:20:41.128 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-12T03:20:41.128 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:20:41.128 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:20:41.128 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-12T03:20:41.129 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-12T03:20:41.129 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-12T03:20:41.129 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-12T03:20:41.129 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-12T03:20:41.129 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-12T03:20:41.130 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-12T03:20:41.488 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-12T03:20:41.488 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:20:41.488 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:20:41.489 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:20:41.489 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:41.489 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:41.489 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:41.489 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-12T03:20:41.639 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-12T03:20:41.642 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-12T03:20:41.642 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-12T03:20:41.643 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-12T03:20:41.643 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-12T03:20:41.643 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-12T03:20:41.643 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-12T03:20:41.644 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-12T03:20:41.644 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-12T03:20:41.644 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:20:41.644 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:41.644 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:41.645 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-mon.a.asok 2020-03-12T03:20:41.645 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-12T03:20:41.645 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.16628/ceph-mon.a.asok config get fsid 2020-03-12T03:20:41.728 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-12T03:20:41.728 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-12T03:20:41.729 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-12T03:20:41.729 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-12T03:20:41.729 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-12T03:20:41.729 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-12T03:20:41.729 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-12T03:20:41.729 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-12T03:20:41.729 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:20:41.730 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:41.730 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:41.730 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-mon.a.asok 2020-03-12T03:20:41.730 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-12T03:20:41.730 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.16628/ceph-mon.a.asok config get mon_host 2020-03-12T03:20:41.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-12T03:20:41.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-12T03:20:41.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-12T03:20:41.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-12T03:20:41.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-12T03:20:41.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-12T03:20:41.820 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-12T03:20:42.720 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-12T03:20:42.720 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:20:42.721 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:20:42.721 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:20:42.721 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:42.721 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:42.721 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:42.722 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-12T03:20:42.722 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-12T03:20:42.749 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-12T03:20:42.749 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-12T03:20:42.749 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:20:42.749 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:20:42.749 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-12T03:20:42.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:20:42.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-12T03:20:42.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:20:42.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:20:42.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:20:42.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:20:42.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-12T03:20:42.751 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-12T03:20:42.751 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:20:42.751 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:20:42.751 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:20:42.751 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:20:42.751 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:20:42.751 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:20:42.752 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:20:42.753 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:20:42.753 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:20:42.753 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:20:42.753 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:20:42.753 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:20:42.753 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:20:42.753 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:20:42.754 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-12T03:20:42.754 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:20:42.754 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=99ca4acf-0655-4af6-81b1-279a966fc6aa 2020-03-12T03:20:42.755 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 99ca4acf-0655-4af6-81b1-279a966fc6aa' 2020-03-12T03:20:42.755 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:20:42.755 INFO:tasks.workunit.client.0.smithi180.stdout:add osd0 99ca4acf-0655-4af6-81b1-279a966fc6aa 2020-03-12T03:20:42.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCKqmleeGKwLRAA5sVwFva7K2qQlzRtXI3fcw== 2020-03-12T03:20:42.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCKqmleeGKwLRAA5sVwFva7K2qQlzRtXI3fcw=="}' 2020-03-12T03:20:42.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 99ca4acf-0655-4af6-81b1-279a966fc6aa -i td/divergent-priors/0/new.json 2020-03-12T03:20:43.146 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:20:43.156 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-12T03:20:43.157 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCKqmleeGKwLRAA5sVwFva7K2qQlzRtXI3fcw== --osd-uuid 99ca4acf-0655-4af6-81b1-279a966fc6aa 2020-03-12T03:20:43.186 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:43.184+0000 7fdfd4a74ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:43.187 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:43.186+0000 7fdfd4a74ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:43.190 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:43.189+0000 7fdfd4a74ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:43.190 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:43.189+0000 7fdfd4a74ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:20:43.191 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:43.189+0000 7fdfd4a74ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:20:43.191 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:43.189+0000 7fdfd4a74ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:20:43.244 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:43.243+0000 7fdfd4a74ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-12T03:20:46.241 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-12T03:20:46.241 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:20:46.242 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-12T03:20:46.242 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:20:46.242 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd0 key to auth repository 2020-03-12T03:20:46.506 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.0 2020-03-12T03:20:46.506 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-12T03:20:46.506 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:20:46.506 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:20:46.506 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:20:46.507 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:20:46.533 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:46.531+0000 7f2edf411ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:46.565 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:46.563+0000 7f2edf411ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:46.567 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:46.565+0000 7f2edf411ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:46.790 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-12T03:20:46.790 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:20:46.791 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:20:46.791 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:20:46.791 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:20:46.791 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:20:46.791 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:20:46.791 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:20:46.792 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:20:46.792 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:20:47.082 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:47.080+0000 7f2edf411ec0 -1 Falling back to public interface 2020-03-12T03:20:47.137 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:20:47.854 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:47.852+0000 7f2edf411ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-12T03:20:48.139 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:20:48.139 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:20:48.139 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:20:48.139 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:20:48.139 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:20:48.139 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:20:48.494 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17010,v1:127.0.0.1:6803/17010] [v2:127.0.0.1:6804/17010,v1:127.0.0.1:6805/17010] exists,up 99ca4acf-0655-4af6-81b1-279a966fc6aa 2020-03-12T03:20:48.494 INFO:tasks.workunit.client.0.smithi180.stdout:add osd1 b9daa4e6-aceb-4cc6-be00-95297a67fffc 2020-03-12T03:20:48.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:20:48.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:20:48.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-12T03:20:48.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:20:48.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:20:48.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:20:48.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:20:48.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T03:20:48.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T03:20:48.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:20:48.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:20:48.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:20:48.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:20:48.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:20:48.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:20:48.497 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:20:48.497 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:48.498 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:48.498 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:48.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:48.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:20:48.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:20:48.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:20:48.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:20:48.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:20:48.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:20:48.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:20:48.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:20:48.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:20:48.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:20:48.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-12T03:20:48.500 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:20:48.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b9daa4e6-aceb-4cc6-be00-95297a67fffc 2020-03-12T03:20:48.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 b9daa4e6-aceb-4cc6-be00-95297a67fffc' 2020-03-12T03:20:48.500 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:20:48.511 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCQqmleU49fHhAAGUTWK01Nx4GV65dq3E8XKg== 2020-03-12T03:20:48.511 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCQqmleU49fHhAAGUTWK01Nx4GV65dq3E8XKg=="}' 2020-03-12T03:20:48.511 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b9daa4e6-aceb-4cc6-be00-95297a67fffc -i td/divergent-priors/1/new.json 2020-03-12T03:20:49.062 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:20:49.071 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-12T03:20:49.071 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCQqmleU49fHhAAGUTWK01Nx4GV65dq3E8XKg== --osd-uuid b9daa4e6-aceb-4cc6-be00-95297a67fffc 2020-03-12T03:20:49.104 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:49.089+0000 7f0bc922cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:49.113 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:49.100+0000 7f0bc922cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:49.115 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:49.102+0000 7f0bc922cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:49.115 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:49.103+0000 7f0bc922cec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:20:49.115 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:49.103+0000 7f0bc922cec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:20:49.115 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:49.103+0000 7f0bc922cec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:20:49.211 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:49.200+0000 7f0bc922cec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-12T03:20:52.177 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-12T03:20:52.177 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:20:52.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-12T03:20:52.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:20:52.178 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd1 key to auth repository 2020-03-12T03:20:52.632 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T03:20:52.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-12T03:20:52.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:20:52.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:20:52.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:20:52.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:20:52.658 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:52.656+0000 7f676783fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:52.660 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:52.658+0000 7f676783fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:52.661 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:52.659+0000 7f676783fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:52.978 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-12T03:20:52.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:20:52.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:20:52.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:20:52.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:20:52.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:20:52.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:20:52.980 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:20:52.980 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:20:52.980 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:20:53.176 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:53.174+0000 7f676783fec0 -1 Falling back to public interface 2020-03-12T03:20:53.322 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:20:53.884 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:53.882+0000 7f676783fec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-12T03:20:54.323 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:20:54.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:20:54.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:20:54.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:20:54.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:20:54.324 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:20:54.708 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/17420,v1:127.0.0.1:6811/17420] [v2:127.0.0.1:6812/17420,v1:127.0.0.1:6813/17420] exists,up b9daa4e6-aceb-4cc6-be00-95297a67fffc 2020-03-12T03:20:54.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:20:54.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:20:54.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:20:54.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-12T03:20:54.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-12T03:20:54.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:20:54.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:20:54.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-12T03:20:54.710 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:20:54.710 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-12T03:20:54.710 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:20:54.710 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:20:54.710 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:20:54.710 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:20:54.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-12T03:20:54.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-12T03:20:54.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:20:54.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:20:54.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:20:54.711 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:20:54.711 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:20:54.712 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:20:54.712 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:20:54.712 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:20:54.712 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:20:54.712 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:54.713 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:20:54.713 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:20:54.713 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:20:54.713 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:20:54.713 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:20:54.713 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:20:54.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:20:54.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:20:54.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:20:54.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:20:54.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:20:54.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-12T03:20:54.715 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:20:54.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b12f5e98-07a1-4a02-bc55-25f7acbfe28e 2020-03-12T03:20:54.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 b12f5e98-07a1-4a02-bc55-25f7acbfe28e' 2020-03-12T03:20:54.716 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:20:54.716 INFO:tasks.workunit.client.0.smithi180.stdout:add osd2 b12f5e98-07a1-4a02-bc55-25f7acbfe28e 2020-03-12T03:20:54.727 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCWqmleLJpEKxAA0ZrnpRaWgggYQRVky6B68w== 2020-03-12T03:20:54.727 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCWqmleLJpEKxAA0ZrnpRaWgggYQRVky6B68w=="}' 2020-03-12T03:20:54.727 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b12f5e98-07a1-4a02-bc55-25f7acbfe28e -i td/divergent-priors/2/new.json 2020-03-12T03:20:55.173 INFO:tasks.workunit.client.0.smithi180.stdout:2 2020-03-12T03:20:55.183 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-12T03:20:55.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCWqmleLJpEKxAA0ZrnpRaWgggYQRVky6B68w== --osd-uuid b12f5e98-07a1-4a02-bc55-25f7acbfe28e 2020-03-12T03:20:55.213 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:55.211+0000 7ffa41f62ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:55.215 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:55.213+0000 7ffa41f62ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:55.216 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:55.214+0000 7ffa41f62ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:55.217 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:55.215+0000 7ffa41f62ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:20:55.217 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:55.215+0000 7ffa41f62ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:20:55.217 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:55.215+0000 7ffa41f62ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:20:55.324 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:55.322+0000 7ffa41f62ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-12T03:20:58.352 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-12T03:20:58.352 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:20:58.355 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-12T03:20:58.355 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:20:58.356 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd2 key to auth repository 2020-03-12T03:20:58.845 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.2 2020-03-12T03:20:58.845 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-12T03:20:58.845 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:20:58.846 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:20:58.846 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:20:58.846 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:20:58.871 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:58.869+0000 7efe785f0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:58.873 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:58.871+0000 7efe785f0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:58.876 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:58.874+0000 7efe785f0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:20:59.200 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-12T03:20:59.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:20:59.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:20:59.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:20:59.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:20:59.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:20:59.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:20:59.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:20:59.202 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:20:59.202 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:20:59.392 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:20:59.390+0000 7efe785f0ec0 -1 Falling back to public interface 2020-03-12T03:20:59.543 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:21:00.156 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:00.154+0000 7efe785f0ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-12T03:21:00.545 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:21:00.545 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:21:00.545 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:21:00.546 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:21:00.546 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:21:00.546 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:21:00.927 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/17775,v1:127.0.0.1:6819/17775] [v2:127.0.0.1:6820/17775,v1:127.0.0.1:6821/17775] exists,up b12f5e98-07a1-4a02-bc55-25f7acbfe28e 2020-03-12T03:21:00.927 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:21:00.927 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:21:00.928 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:21:00.928 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-12T03:21:01.624 INFO:tasks.workunit.client.0.smithi180.stderr:noout is set 2020-03-12T03:21:01.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-12T03:21:02.442 INFO:tasks.workunit.client.0.smithi180.stderr:noin is set 2020-03-12T03:21:02.450 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-12T03:21:03.110 INFO:tasks.workunit.client.0.smithi180.stderr:nodown is set 2020-03-12T03:21:03.118 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-12T03:21:03.118 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-12T03:21:03.599 INFO:tasks.workunit.client.0.smithi180.stderr:pool 'test' already exists 2020-03-12T03:21:03.607 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-12T03:21:04.608 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-12T03:21:05.222 INFO:tasks.workunit.client.0.smithi180.stderr:set pool 1 size to 3 2020-03-12T03:21:05.230 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-12T03:21:05.884 INFO:tasks.workunit.client.0.smithi180.stderr:set pool 1 min_size to 2 2020-03-12T03:21:05.892 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-12T03:21:05.892 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:21:05.893 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:21:06.270 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:21:06.271 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:21:06.271 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:21:06.271 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:21:06.271 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:06.271 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:21:06.400 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-12T03:21:06.400 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-12T03:21:06.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:06.401 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:21:06.531 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-12T03:21:06.531 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-12T03:21:06.531 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:06.532 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:21:06.660 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-12T03:21:06.661 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-03-12T03:21:06.661 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:06.661 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-12T03:21:06.661 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:06.662 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:21:06.663 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-12T03:21:06.663 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:06.664 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-12T03:21:06.664 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-12T03:21:06.664 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836486 2020-03-12T03:21:06.664 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:21:07.009 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-12T03:21:07.009 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:07.009 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-12T03:21:07.010 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:07.010 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:21:07.011 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-12T03:21:07.011 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:07.011 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-12T03:21:07.012 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-12T03:21:07.012 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 42949672965 2020-03-12T03:21:07.012 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:21:07.353 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-12T03:21:07.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:07.462 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-12T03:21:07.462 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:07.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:21:07.462 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-12T03:21:07.462 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:07.463 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-12T03:21:07.463 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-12T03:21:07.463 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:21:07.463 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 64424509444 2020-03-12T03:21:07.703 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509442 -lt 64424509444 2020-03-12T03:21:07.703 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:21:08.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:21:08.704 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:21:09.069 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-12T03:21:09.070 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-12T03:21:09.070 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:21:09.070 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:21:09.070 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:21:09.070 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:21:09.071 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-12T03:21:09.071 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:21:09.071 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:21:09.071 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:21:09.071 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:21:09.071 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:21:09.157 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:21:09.157 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:21:09.157 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:21:09.157 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:21:09.158 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:21:09.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:21:09.498 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:21:09.498 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:21:09.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:21:09.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:09.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:21:09.627 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-12T03:21:09.627 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-12T03:21:09.627 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:09.628 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:21:09.757 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-12T03:21:09.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2020-03-12T03:21:09.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:09.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:21:09.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-12T03:21:09.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446' 2020-03-12T03:21:09.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:09.888 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-12T03:21:09.888 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:09.888 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:21:09.889 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-12T03:21:09.889 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:09.890 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-12T03:21:09.890 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-12T03:21:09.890 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836489 2020-03-12T03:21:09.891 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:21:10.235 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836489 2020-03-12T03:21:10.235 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:21:11.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:21:11.236 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:21:11.588 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-12T03:21:11.589 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:11.589 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-12T03:21:11.590 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:11.590 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:21:11.591 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-12T03:21:11.591 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:11.592 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-12T03:21:11.592 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-12T03:21:11.593 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:21:11.593 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 42949672967 2020-03-12T03:21:11.970 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672967 2020-03-12T03:21:11.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:11.971 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-12T03:21:11.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:11.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:21:11.973 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-12T03:21:11.973 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:11.974 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-12T03:21:11.974 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-12T03:21:11.975 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:21:11.975 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 64424509446 2020-03-12T03:21:12.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-12T03:21:12.317 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:21:12.317 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:12.318 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:12.940 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:21:12.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:12.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:12.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:12.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:12.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:12.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:12.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:13.273 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-12T03:21:13.273 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:13.274 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:13.274 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:13.840 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-12T03:21:13.840 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-12T03:21:13.840 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-12T03:21:13.840 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-12T03:21:13.841 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-12T03:21:14.167 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:21:14.176 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-12T03:21:14.176 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-12T03:21:14.176 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-12T03:21:14.176 INFO:tasks.workunit.client.0.smithi180.stdout:primary and soon to be divergent is 1 2020-03-12T03:21:14.497 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:21:14.497 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-12T03:21:04.349756+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:21:14.497 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:21:14.497 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:21:14.499 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:21:14.507 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-12T03:21:14.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-12T03:21:14.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-12T03:21:14.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-12T03:21:14.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-12T03:21:14.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-12T03:21:14.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-12T03:21:14.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-12T03:21:14.509 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-12T03:21:14.509 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-12T03:21:14.509 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-12T03:21:14.509 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-12T03:21:14.509 INFO:tasks.workunit.client.0.smithi180.stdout:writing initial objects 2020-03-12T03:21:14.510 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.510 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-12T03:21:14.583 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.584 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-12T03:21:14.646 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.646 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-12T03:21:14.716 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-12T03:21:14.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-12T03:21:14.797 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.797 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-12T03:21:14.833 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.834 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-12T03:21:14.909 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.909 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-12T03:21:14.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:14.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-12T03:21:15.076 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.076 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-12T03:21:15.159 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.160 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-12T03:21:15.226 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.227 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-12T03:21:15.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-12T03:21:15.308 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.308 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-12T03:21:15.350 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.350 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-12T03:21:15.390 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.390 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-12T03:21:15.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-12T03:21:15.480 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.480 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-12T03:21:15.517 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.517 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-12T03:21:15.602 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.602 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-12T03:21:15.642 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.642 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-12T03:21:15.727 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.727 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-12T03:21:15.832 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.832 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-12T03:21:15.875 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.875 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-12T03:21:15.914 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.914 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-12T03:21:15.950 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.951 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-12T03:21:15.989 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:15.989 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-12T03:21:16.025 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.026 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-12T03:21:16.067 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.067 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-12T03:21:16.105 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.106 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-12T03:21:16.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-12T03:21:16.244 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.245 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-12T03:21:16.286 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.287 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-12T03:21:16.387 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.387 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-12T03:21:16.426 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.426 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-12T03:21:16.467 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.467 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-12T03:21:16.509 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.509 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-12T03:21:16.551 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.551 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-12T03:21:16.592 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.593 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-12T03:21:16.634 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.634 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-12T03:21:16.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.700 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-12T03:21:16.775 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.775 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-12T03:21:16.817 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-12T03:21:16.912 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:16.913 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-12T03:21:17.016 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.016 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-12T03:21:17.105 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.105 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-12T03:21:17.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-12T03:21:17.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-12T03:21:17.226 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.226 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-12T03:21:17.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.268 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-12T03:21:17.309 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.309 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-12T03:21:17.352 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.352 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-12T03:21:17.411 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.411 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-12T03:21:17.451 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.451 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-12T03:21:17.501 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.501 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-12T03:21:17.588 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.588 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-12T03:21:17.681 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.681 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-12T03:21:17.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-12T03:21:17.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-12T03:21:17.859 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.859 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-12T03:21:17.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:17.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-12T03:21:18.040 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.040 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-12T03:21:18.084 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.084 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-12T03:21:18.126 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.126 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-12T03:21:18.168 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.168 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-12T03:21:18.276 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.276 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-12T03:21:18.357 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.357 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-12T03:21:18.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-12T03:21:18.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-12T03:21:18.543 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.543 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-12T03:21:18.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.610 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-12T03:21:18.651 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.651 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-12T03:21:18.726 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.727 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-12T03:21:18.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-12T03:21:18.810 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.810 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-12T03:21:18.851 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.851 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-12T03:21:18.893 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.893 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-12T03:21:18.935 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:18.935 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-12T03:21:19.033 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.033 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-12T03:21:19.117 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.117 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-12T03:21:19.200 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.200 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-12T03:21:19.242 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.242 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-12T03:21:19.285 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.285 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-12T03:21:19.326 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.327 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-12T03:21:19.393 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.393 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-12T03:21:19.435 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.435 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-12T03:21:19.476 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.477 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-12T03:21:19.518 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.518 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-12T03:21:19.560 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.560 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-12T03:21:19.642 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.642 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-12T03:21:19.684 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.684 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-12T03:21:19.826 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.826 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-12T03:21:19.910 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.910 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-12T03:21:19.960 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:19.960 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-12T03:21:20.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:20.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-12T03:21:20.043 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:20.044 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-12T03:21:20.085 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:20.085 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-12T03:21:20.126 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:20.127 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-12T03:21:20.185 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:20.185 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-12T03:21:20.225 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-12T03:21:20.226 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-12T03:21:20.269 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T03:21:20.269 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-12T03:21:20.269 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:21:20.269 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:21:20.269 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:21:20.269 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:21:20.270 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T03:21:20.270 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:21:20.271 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:21:20.271 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:21:20.272 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:21:20.273 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:21:20.323 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:21:20.323 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:21:20.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:21:20.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:21:20.324 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:21:20.663 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:21:20.664 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:21:20.664 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:21:20.664 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:21:20.664 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:20.664 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:21:20.794 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-12T03:21:20.794 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-12T03:21:20.794 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:20.795 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:21:20.925 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-12T03:21:20.925 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2020-03-12T03:21:20.926 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:20.926 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:21:21.055 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-12T03:21:21.055 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509450' 2020-03-12T03:21:21.056 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:21.056 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-12T03:21:21.056 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:21.057 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:21:21.057 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-12T03:21:21.057 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:21.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-12T03:21:21.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-12T03:21:21.059 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836493 2020-03-12T03:21:21.059 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:21:21.400 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-12T03:21:21.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:21:22.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:21:22.402 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:21:22.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-12T03:21:22.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:22.756 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-12T03:21:22.756 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:22.757 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:21:22.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-12T03:21:22.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:22.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-12T03:21:22.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-12T03:21:22.759 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 42949672972 2020-03-12T03:21:22.759 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:21:23.104 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-12T03:21:23.104 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:23.105 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-12T03:21:23.105 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:23.105 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:21:23.106 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-12T03:21:23.107 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:23.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-12T03:21:23.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-12T03:21:23.107 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 64424509450 2020-03-12T03:21:23.108 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:21:23.474 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509450 2020-03-12T03:21:23.475 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:21:23.475 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:23.475 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:23.919 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:21:23.919 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:23.920 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:23.921 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:23.921 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:23.921 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:23.921 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:23.921 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:24.251 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-12T03:21:24.252 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:24.252 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:24.252 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:24.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-12T03:21:24.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-12T03:21:24.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-12T03:21:24.698 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-12T03:21:24.698 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-12T03:21:24.699 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-12T03:21:24.699 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-12T03:21:24.699 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-12T03:21:25.047 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-12T03:21:25.047 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-12T03:21:25.048 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-12T03:21:25.048 INFO:tasks.workunit.client.0.smithi180.stdout:blackholing osds 0 2 2020-03-12T03:21:25.369 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:21:25.455 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-12T03:21:04.349756+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:21:25.455 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:21:25.455 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:21:25.456 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:21:25.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:25.457 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-12T03:21:25.457 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-12T03:21:25.457 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-12T03:21:25.457 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:21:25.457 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:21:25.458 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:21:25.458 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.0.asok 2020-03-12T03:21:25.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-12T03:21:25.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-12T03:21:25.467 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:21:25.468 INFO:tasks.workunit.client.0.smithi180.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-12T03:21:25.468 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:21:25.475 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:25.476 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-12T03:21:25.476 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-12T03:21:25.476 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-12T03:21:25.477 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:21:25.477 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:21:25.477 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:21:25.477 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.2.asok 2020-03-12T03:21:25.477 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-12T03:21:25.478 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-12T03:21:25.562 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:21:25.562 INFO:tasks.workunit.client.0.smithi180.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-12T03:21:25.562 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:21:25.569 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-12T03:21:25.570 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-12T03:21:25.572 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-12T03:21:25.573 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-12T03:21:25.573 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-12T03:21:25.573 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-12T03:21:25.573 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-12T03:21:25.573 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-12T03:21:25.573 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-12T03:21:25.574 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-12T03:21:25.574 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-12T03:21:25.574 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-12T03:21:25.574 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-12T03:21:25.574 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-12T03:21:25.574 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-12T03:21:25.574 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-12T03:21:25.575 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-12T03:21:25.576 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-12T03:21:25.576 INFO:tasks.workunit.client.0.smithi180.stdout:writing divergent object 2020-03-12T03:21:25.576 INFO:tasks.workunit.client.0.smithi180.stdout:create missing divergent object 2020-03-12T03:21:25.921 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-12T03:21:25.921 INFO:tasks.workunit.client.0.smithi180.stderr:0 2020-03-12T03:21:25.921 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:21:25.921 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-12T03:21:25.921 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-12T03:21:25.922 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-12T03:21:25.922 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-12T03:21:25.922 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-12T03:21:25.924 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-12T03:21:25.924 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-12T03:21:25.924 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-12T03:21:25.924 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-12T03:21:25.924 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-12T03:21:25.924 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-12T03:21:25.925 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-12T03:21:25.925 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-12T03:21:25.925 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-12T03:21:25.925 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-12T03:21:25.925 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-12T03:21:25.925 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:21:25.926 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:21:25.926 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:21:25.926 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.1.asok 2020-03-12T03:21:25.926 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.16628/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-12T03:21:26.018 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-12T03:21:26.018 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-12T03:21:26.019 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-12T03:21:26.020 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-12T03:21:26.020 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-12T03:21:26.020 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-12T03:21:26.020 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:21:26.020 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:21:26.020 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:21:26.020 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.1.asok 2020-03-12T03:21:26.021 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-12T03:21:26.021 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.1.asok injectdataerr test existing_99 2020-03-12T03:21:26.116 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-12T03:21:26.116 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-12T03:21:36.117 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-12T03:21:36.139 INFO:tasks.workunit.client.0.smithi180.stderr:rados: no process found 2020-03-12T03:21:36.139 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-12T03:21:36.140 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-12T03:21:36.140 INFO:tasks.workunit.client.0.smithi180.stdout:killing all the osds 2020-03-12T03:21:36.467 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:21:36.467 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-12T03:21:26.363842+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:21:36.467 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:21:36.467 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:21:36.468 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:21:36.476 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-12T03:21:36.477 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:21:36.477 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:21:36.478 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:21:36.478 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:21:36.478 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:21:36.584 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:21:36.584 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-12T03:21:36.584 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-12T03:21:37.141 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already down. 2020-03-12T03:21:37.149 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-12T03:21:37.150 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-12T03:21:37.674 INFO:tasks.workunit.client.0.smithi180.stderr:osd.1 is already down. 2020-03-12T03:21:37.682 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-12T03:21:37.682 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-12T03:21:38.202 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already down. 2020-03-12T03:21:38.210 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:38.211 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-12T03:21:38.703 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already out. 2020-03-12T03:21:38.710 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:38.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-12T03:21:39.212 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already out. 2020-03-12T03:21:39.220 INFO:tasks.workunit.client.0.smithi180.stdout:bringing up non_divergent 0 2 2020-03-12T03:21:39.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-12T03:21:39.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-12T03:21:39.538 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:21:39.539 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-12T03:21:26.363842+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:21:39.539 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:21:39.539 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:21:39.540 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:21:39.551 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:39.552 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-12T03:21:39.552 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:21:39.552 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:21:39.552 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-12T03:21:39.552 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:21:39.552 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-12T03:21:39.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:21:39.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:21:39.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:21:39.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:21:39.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-12T03:21:39.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-12T03:21:39.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:21:39.554 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:21:39.554 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:21:39.554 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:21:39.554 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:21:39.554 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:21:39.555 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:21:39.555 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:21:39.555 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:21:39.555 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:21:39.555 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:21:39.555 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:21:39.555 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:21:39.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:21:39.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:21:39.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:21:39.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:21:39.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:21:39.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:21:39.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-12T03:21:39.557 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-12T03:21:39.557 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:21:39.557 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-12T03:21:39.557 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.0 2020-03-12T03:21:39.558 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-12T03:21:39.558 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:21:39.558 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:21:39.558 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:21:39.578 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:39.576+0000 7f3bab77bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:21:39.580 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:39.578+0000 7f3bab77bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:21:39.582 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:39.579+0000 7f3bab77bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:21:39.900 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-12T03:21:39.900 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:21:39.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:21:39.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:21:39.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:21:39.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:21:39.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:21:39.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:21:39.901 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:21:39.902 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:21:40.099 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:40.096+0000 7f3bab77bec0 -1 Falling back to public interface 2020-03-12T03:21:40.244 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:21:41.002 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:41.000+0000 7f3bab77bec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-12T03:21:41.246 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:21:41.246 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:21:41.247 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:21:41.247 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:21:41.247 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:21:41.247 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:21:41.594 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/21762,v1:127.0.0.1:6803/21762] [v2:127.0.0.1:6804/21762,v1:127.0.0.1:6805/21762] exists,up 99ca4acf-0655-4af6-81b1-279a966fc6aa 2020-03-12T03:21:41.594 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.2 2020-03-12T03:21:41.594 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:21:41.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:21:41.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:21:41.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:41.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-12T03:21:41.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:21:41.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:21:41.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-12T03:21:41.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:21:41.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-12T03:21:41.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:21:41.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:21:41.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:21:41.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:21:41.597 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-12T03:21:41.597 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-12T03:21:41.597 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:21:41.597 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:21:41.597 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:21:41.598 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:21:41.598 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:21:41.598 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:21:41.598 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:21:41.598 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:21:41.598 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:21:41.598 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:21:41.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:21:41.600 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:21:41.600 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-12T03:21:41.600 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-12T03:21:41.600 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:21:41.600 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-12T03:21:41.600 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-12T03:21:41.600 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:21:41.601 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:21:41.601 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:21:41.619 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:41.616+0000 7f41b0630ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:21:41.620 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:41.618+0000 7f41b0630ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:21:41.622 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:41.619+0000 7f41b0630ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:21:41.940 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-12T03:21:41.940 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:21:41.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:21:41.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:21:41.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:21:41.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:21:41.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:21:41.942 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:21:41.942 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:21:41.942 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:21:42.139 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:42.137+0000 7f41b0630ec0 -1 Falling back to public interface 2020-03-12T03:21:42.284 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:21:43.107 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:21:43.105+0000 7f41b0630ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-12T03:21:43.286 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:21:43.286 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:21:43.287 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:21:43.287 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:21:43.287 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:21:43.287 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:21:43.686 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/21956,v1:127.0.0.1:6811/21956] [v2:127.0.0.1:6812/21956,v1:127.0.0.1:6813/21956] exists,up b12f5e98-07a1-4a02-bc55-25f7acbfe28e 2020-03-12T03:21:43.687 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:21:43.687 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:21:43.687 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:21:43.687 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:43.687 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-12T03:21:44.179 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already in. 2020-03-12T03:21:44.187 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-12T03:21:44.187 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-12T03:21:44.659 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already in. 2020-03-12T03:21:44.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T03:21:44.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-12T03:21:44.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:21:44.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:21:44.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:21:44.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:21:44.669 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T03:21:44.669 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:21:44.669 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:21:44.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:21:44.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:21:44.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:21:44.723 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:21:44.723 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:21:44.723 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:21:44.724 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:21:44.724 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:21:45.084 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:21:45.084 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:21:45.084 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:21:45.085 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:21:45.085 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:45.085 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:21:45.213 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-12T03:21:45.214 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-12T03:21:45.214 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:45.214 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:21:45.305 INFO:tasks.workunit.client.0.smithi180.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-12T03:21:45.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-12T03:21:45.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-12T03:21:45.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:21:45.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:21:45.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-12T03:21:45.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-12T03:21:45.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:45.442 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-12T03:21:45.443 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:45.443 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:21:45.443 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-12T03:21:45.444 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:45.444 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-12T03:21:45.444 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-12T03:21:45.445 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:21:45.445 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 146028888067 2020-03-12T03:21:45.786 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-12T03:21:45.786 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:21:46.787 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:21:46.787 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:21:47.144 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-12T03:21:47.144 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:47.145 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-12T03:21:47.146 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:47.146 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:21:47.147 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-12T03:21:47.147 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:47.148 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-12T03:21:47.148 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-12T03:21:47.148 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:21:47.148 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 2020-03-12T03:21:47.493 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-12T03:21:47.493 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-12T03:21:47.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:21:47.494 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-12T03:21:47.494 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:21:47.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:21:47.495 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-12T03:21:47.495 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:21:47.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-12T03:21:47.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-12T03:21:47.496 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:21:47.496 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 150323855363 2020-03-12T03:21:47.836 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-12T03:21:47.836 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:21:47.836 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:47.837 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:48.303 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:21:48.303 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:48.304 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:48.304 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:48.304 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:48.304 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:48.305 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:48.305 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:48.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:21:48.633 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:48.634 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:48.634 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:49.073 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:21:49.073 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-12T03:21:49.073 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-12T03:21:49.073 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-12T03:21:49.074 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:21:49.074 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-12T03:21:49.174 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:21:49.174 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:49.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:49.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:49.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:49.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:49.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:49.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:49.503 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:21:49.503 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:49.503 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:49.504 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:49.952 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:21:49.953 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:21:49.953 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:21:49.953 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:21:49.953 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:21:49.953 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:21:49.953 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:21:49.954 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:21:49.954 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:21:50.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:21:50.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:21:50.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-12T03:21:50.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:21:50.399 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-12T03:21:50.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:21:50.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:50.600 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:50.600 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:50.600 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:50.600 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:50.600 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:50.600 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:50.930 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:21:50.930 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:50.933 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:50.933 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:51.378 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:21:51.378 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:21:51.378 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:21:51.378 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:21:51.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:21:51.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:21:51.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:21:51.379 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:21:51.379 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:21:51.820 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:21:51.820 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:21:51.820 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-12T03:21:51.820 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:21:51.820 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-12T03:21:52.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:21:52.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:52.221 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:52.221 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:52.222 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:52.222 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:52.222 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:52.222 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:52.556 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:21:52.557 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:52.557 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:52.557 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:53.015 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:21:53.015 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:21:53.015 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:21:53.015 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:21:53.015 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:21:53.015 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:21:53.016 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:21:53.016 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:21:53.016 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:21:53.456 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:21:53.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:21:53.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-12T03:21:53.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:21:53.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-12T03:21:54.257 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:21:54.258 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:54.258 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:54.259 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:54.259 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:54.259 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:54.259 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:54.259 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:54.603 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:21:54.603 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:54.604 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:54.604 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:55.057 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:21:55.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:21:55.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:21:55.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:21:55.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:21:55.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:21:55.058 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:21:55.058 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:21:55.059 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:21:55.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:21:55.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:21:55.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-12T03:21:55.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:21:55.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-12T03:21:57.097 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:21:57.098 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:21:57.099 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:21:57.099 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:21:57.099 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:21:57.099 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:21:57.099 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:21:57.099 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:21:57.436 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:21:57.437 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:21:57.437 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:21:57.437 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:21:57.882 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:21:57.883 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:21:57.883 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:21:57.883 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:21:57.883 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:21:57.883 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:21:57.883 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:21:57.884 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:21:57.884 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:21:58.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:21:58.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:21:58.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-12T03:21:58.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:21:58.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-12T03:22:01.528 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:22:01.529 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:22:01.529 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:22:01.530 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:22:01.530 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:22:01.530 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:22:01.530 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:22:01.530 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:22:01.870 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:22:01.870 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:22:01.871 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:22:01.871 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:22:02.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:22:02.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:22:02.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:22:02.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:22:02.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:22:02.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:22:02.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:22:02.315 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:22:02.315 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:22:02.755 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:22:02.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:22:02.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-12T03:22:02.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:22:02.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-12T03:22:09.156 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:22:09.156 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:22:09.157 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:22:09.157 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:22:09.157 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:22:09.158 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:22:09.158 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:22:09.158 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:22:09.492 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:22:09.492 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:22:09.493 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:22:09.493 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:22:09.936 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:22:09.936 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:22:09.936 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:22:09.936 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:22:09.936 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:22:09.937 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:22:09.937 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:22:09.937 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:22:09.937 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:22:10.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:22:10.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:22:10.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-12T03:22:10.380 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:22:10.380 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-12T03:22:17.688 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:22:17.688 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:22:17.688 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:22:17.688 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:22:17.688 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:22:17.688 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:22:17.689 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:22:17.689 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:22:18.019 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:22:18.020 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:22:18.020 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:22:18.020 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:22:18.459 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:22:18.459 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:22:18.459 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:22:18.459 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:22:18.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:22:18.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:22:18.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:22:18.460 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:22:18.460 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:22:18.902 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:22:18.902 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:22:18.902 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-12T03:22:18.902 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-12T03:22:19.336 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:22:19.336 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_fingerprint": "1c1392eb-aa5e-4a16-90ee-45f8fb0a3533", 2020-03-12T03:22:19.336 INFO:tasks.workunit.client.0.smithi180.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-12T03:22:19.336 INFO:tasks.workunit.client.0.smithi180.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-12T03:22:19.336 INFO:tasks.workunit.client.0.smithi180.stdout: "timestamp": "2020-03-12T03:22:19.334034+0000", 2020-03-12T03:22:19.336 INFO:tasks.workunit.client.0.smithi180.stdout: "tag": "", 2020-03-12T03:22:19.337 INFO:tasks.workunit.client.0.smithi180.stdout: "health": { 2020-03-12T03:22:19.337 INFO:tasks.workunit.client.0.smithi180.stdout: "status": "HEALTH_WARN", 2020-03-12T03:22:19.337 INFO:tasks.workunit.client.0.smithi180.stdout: "checks": { 2020-03-12T03:22:19.337 INFO:tasks.workunit.client.0.smithi180.stdout: "OSDMAP_FLAGS": { 2020-03-12T03:22:19.337 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:22:19.337 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:22:19.338 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-12T03:22:19.338 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 17 2020-03-12T03:22:19.338 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.338 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [], 2020-03-12T03:22:19.338 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:22:19.338 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.338 INFO:tasks.workunit.client.0.smithi180.stdout: "OSD_DOWN": { 2020-03-12T03:22:19.339 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:22:19.339 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:22:19.339 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 osds down", 2020-03-12T03:22:19.339 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:22:19.339 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.339 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:22:19.339 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "osd.1 (root=default,host=smithi180) is down" 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: "PG_DEGRADED": { 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:22:19.340 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:22:19.341 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-12T03:22:19.341 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:22:19.341 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.341 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:22:19.341 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.341 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-12T03:22:19.341 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.342 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.342 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:22:19.342 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.342 INFO:tasks.workunit.client.0.smithi180.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-12T03:22:19.342 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:22:19.342 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:22:19.342 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-12T03:22:19.343 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:22:19.343 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.343 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:22:19.343 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.343 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "application not enabled on pool 'test'" 2020-03-12T03:22:19.343 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.343 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.344 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-12T03:22:19.344 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.344 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.344 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:22:19.344 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.344 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.344 INFO:tasks.workunit.client.0.smithi180.stdout: "mutes": [] 2020-03-12T03:22:19.345 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.345 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_first_committed": 1, 2020-03-12T03:22:19.345 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_last_committed": 1, 2020-03-12T03:22:19.345 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap": { 2020-03-12T03:22:19.345 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:22:19.345 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:22:19.345 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:20:41.183319Z", 2020-03-12T03:22:19.348 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:20:41.183319Z", 2020-03-12T03:22:19.348 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release": 15, 2020-03-12T03:22:19.348 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release_name": "octopus", 2020-03-12T03:22:19.348 INFO:tasks.workunit.client.0.smithi180.stdout: "features": { 2020-03-12T03:22:19.348 INFO:tasks.workunit.client.0.smithi180.stdout: "persistent": [ 2020-03-12T03:22:19.348 INFO:tasks.workunit.client.0.smithi180.stdout: "kraken", 2020-03-12T03:22:19.348 INFO:tasks.workunit.client.0.smithi180.stdout: "luminous", 2020-03-12T03:22:19.349 INFO:tasks.workunit.client.0.smithi180.stdout: "mimic", 2020-03-12T03:22:19.349 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap-prune", 2020-03-12T03:22:19.349 INFO:tasks.workunit.client.0.smithi180.stdout: "nautilus", 2020-03-12T03:22:19.349 INFO:tasks.workunit.client.0.smithi180.stdout: "octopus" 2020-03-12T03:22:19.349 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.349 INFO:tasks.workunit.client.0.smithi180.stdout: "optional": [] 2020-03-12T03:22:19.349 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.350 INFO:tasks.workunit.client.0.smithi180.stdout: "mons": [ 2020-03-12T03:22:19.350 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.350 INFO:tasks.workunit.client.0.smithi180.stdout: "rank": 0, 2020-03-12T03:22:19.350 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "a", 2020-03-12T03:22:19.350 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:22:19.350 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.350 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.351 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.351 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115", 2020-03-12T03:22:19.351 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 0 2020-03-12T03:22:19.351 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.351 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.351 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.351 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115/0", 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: "priority": 0, 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 0 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: "quorum": [ 2020-03-12T03:22:19.352 INFO:tasks.workunit.client.0.smithi180.stdout: 0 2020-03-12T03:22:19.353 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.353 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap": { 2020-03-12T03:22:19.353 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 39, 2020-03-12T03:22:19.353 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:22:19.353 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:20:41.927514+0000", 2020-03-12T03:22:19.353 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:21:44.756416+0000", 2020-03-12T03:22:19.353 INFO:tasks.workunit.client.0.smithi180.stdout: "last_up_change": "2020-03-12T03:21:43.186818+0000", 2020-03-12T03:22:19.354 INFO:tasks.workunit.client.0.smithi180.stdout: "last_in_change": "2020-03-12T03:21:44.447151+0000", 2020-03-12T03:22:19.354 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-12T03:22:19.354 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_num": 5800384, 2020-03-12T03:22:19.354 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_set": [ 2020-03-12T03:22:19.354 INFO:tasks.workunit.client.0.smithi180.stdout: "nodown", 2020-03-12T03:22:19.354 INFO:tasks.workunit.client.0.smithi180.stdout: "noin", 2020-03-12T03:22:19.354 INFO:tasks.workunit.client.0.smithi180.stdout: "noout", 2020-03-12T03:22:19.355 INFO:tasks.workunit.client.0.smithi180.stdout: "pglog_hardlimit", 2020-03-12T03:22:19.355 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snapdirs", 2020-03-12T03:22:19.355 INFO:tasks.workunit.client.0.smithi180.stdout: "recovery_deletes", 2020-03-12T03:22:19.355 INFO:tasks.workunit.client.0.smithi180.stdout: "sortbitwise" 2020-03-12T03:22:19.355 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.355 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_version": 7, 2020-03-12T03:22:19.355 INFO:tasks.workunit.client.0.smithi180.stdout: "full_ratio": 0.99000000953674316, 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_snapshot": "", 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_max": 1, 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "max_osd": 3, 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "require_min_compat_client": "jewel", 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "min_compat_client": "jewel", 2020-03-12T03:22:19.356 INFO:tasks.workunit.client.0.smithi180.stdout: "require_osd_release": "octopus", 2020-03-12T03:22:19.357 INFO:tasks.workunit.client.0.smithi180.stdout: "pools": [ 2020-03-12T03:22:19.357 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.357 INFO:tasks.workunit.client.0.smithi180.stdout: "pool": 1, 2020-03-12T03:22:19.357 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_name": "test", 2020-03-12T03:22:19.357 INFO:tasks.workunit.client.0.smithi180.stdout: "create_time": "2020-03-12T03:21:03.374250+0000", 2020-03-12T03:22:19.357 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": 1, 2020-03-12T03:22:19.357 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_names": "hashpspool", 2020-03-12T03:22:19.358 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:22:19.358 INFO:tasks.workunit.client.0.smithi180.stdout: "size": 3, 2020-03-12T03:22:19.358 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 2, 2020-03-12T03:22:19.358 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_rule": 0, 2020-03-12T03:22:19.359 INFO:tasks.workunit.client.0.smithi180.stdout: "object_hash": 2, 2020-03-12T03:22:19.359 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_autoscale_mode": "off", 2020-03-12T03:22:19.359 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num": 1, 2020-03-12T03:22:19.359 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num": 1, 2020-03-12T03:22:19.360 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num_target": 1, 2020-03-12T03:22:19.360 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_target": 1, 2020-03-12T03:22:19.360 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_pending": 1, 2020-03-12T03:22:19.360 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pg_merge_meta": { 2020-03-12T03:22:19.360 INFO:tasks.workunit.client.0.smithi180.stdout: "source_pgid": "0.0", 2020-03-12T03:22:19.360 INFO:tasks.workunit.client.0.smithi180.stdout: "ready_epoch": 0, 2020-03-12T03:22:19.361 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_started": 0, 2020-03-12T03:22:19.361 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_clean": 0, 2020-03-12T03:22:19.361 INFO:tasks.workunit.client.0.smithi180.stdout: "source_version": "0'0", 2020-03-12T03:22:19.361 INFO:tasks.workunit.client.0.smithi180.stdout: "target_version": "0'0" 2020-03-12T03:22:19.363 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.363 INFO:tasks.workunit.client.0.smithi180.stdout: "last_change": "28", 2020-03-12T03:22:19.364 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend": "0", 2020-03-12T03:22:19.364 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-12T03:22:19.364 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_preluminous": "0", 2020-03-12T03:22:19.365 INFO:tasks.workunit.client.0.smithi180.stdout: "auid": 0, 2020-03-12T03:22:19.365 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_mode": "selfmanaged", 2020-03-12T03:22:19.365 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_seq": 0, 2020-03-12T03:22:19.365 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_epoch": 0, 2020-03-12T03:22:19.366 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_snaps": [], 2020-03-12T03:22:19.366 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps": "[]", 2020-03-12T03:22:19.366 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_bytes": 0, 2020-03-12T03:22:19.366 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_objects": 0, 2020-03-12T03:22:19.366 INFO:tasks.workunit.client.0.smithi180.stdout: "tiers": [], 2020-03-12T03:22:19.367 INFO:tasks.workunit.client.0.smithi180.stdout: "tier_of": -1, 2020-03-12T03:22:19.367 INFO:tasks.workunit.client.0.smithi180.stdout: "read_tier": -1, 2020-03-12T03:22:19.367 INFO:tasks.workunit.client.0.smithi180.stdout: "write_tier": -1, 2020-03-12T03:22:19.367 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_mode": "none", 2020-03-12T03:22:19.367 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_bytes": 0, 2020-03-12T03:22:19.368 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_objects": 0, 2020-03-12T03:22:19.368 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-12T03:22:19.368 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-12T03:22:19.368 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-12T03:22:19.368 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_flush_age": 0, 2020-03-12T03:22:19.369 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_evict_age": 0, 2020-03-12T03:22:19.369 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profile": "", 2020-03-12T03:22:19.369 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_params": { 2020-03-12T03:22:19.369 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "none" 2020-03-12T03:22:19.369 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.369 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_period": 0, 2020-03-12T03:22:19.370 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_count": 0, 2020-03-12T03:22:19.370 INFO:tasks.workunit.client.0.smithi180.stdout: "use_gmt_hitset": true, 2020-03-12T03:22:19.370 INFO:tasks.workunit.client.0.smithi180.stdout: "min_read_recency_for_promote": 0, 2020-03-12T03:22:19.370 INFO:tasks.workunit.client.0.smithi180.stdout: "min_write_recency_for_promote": 0, 2020-03-12T03:22:19.370 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_grade_decay_rate": 0, 2020-03-12T03:22:19.370 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_search_last_n": 0, 2020-03-12T03:22:19.370 INFO:tasks.workunit.client.0.smithi180.stdout: "grade_table": [], 2020-03-12T03:22:19.371 INFO:tasks.workunit.client.0.smithi180.stdout: "stripe_width": 0, 2020-03-12T03:22:19.371 INFO:tasks.workunit.client.0.smithi180.stdout: "expected_num_objects": 0, 2020-03-12T03:22:19.371 INFO:tasks.workunit.client.0.smithi180.stdout: "fast_read": false, 2020-03-12T03:22:19.371 INFO:tasks.workunit.client.0.smithi180.stdout: "options": {}, 2020-03-12T03:22:19.371 INFO:tasks.workunit.client.0.smithi180.stdout: "application_metadata": {} 2020-03-12T03:22:19.371 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: "osds": [ 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "99ca4acf-0655-4af6-81b1-279a966fc6aa", 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:22:19.372 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:22:19.373 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:22:19.373 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 5, 2020-03-12T03:22:19.373 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 28, 2020-03-12T03:22:19.373 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 34, 2020-03-12T03:22:19.373 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 38, 2020-03-12T03:22:19.373 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 29, 2020-03-12T03:22:19.374 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:22:19.374 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:22:19.374 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.374 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.374 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.374 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6802", 2020-03-12T03:22:19.375 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.375 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.375 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.375 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.375 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6803", 2020-03-12T03:22:19.375 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.376 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.376 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.376 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.376 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:22:19.376 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.376 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.376 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.377 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6804", 2020-03-12T03:22:19.377 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.377 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.377 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.377 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.377 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6805", 2020-03-12T03:22:19.377 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.378 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.378 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.378 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.378 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:22:19.378 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.378 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.378 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.379 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6808", 2020-03-12T03:22:19.379 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.379 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.379 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.379 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.379 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6809", 2020-03-12T03:22:19.379 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.380 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6806", 2020-03-12T03:22:19.381 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.381 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.381 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.381 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.381 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6807", 2020-03-12T03:22:19.381 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21762 2020-03-12T03:22:19.381 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.382 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.382 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.382 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6803/21762", 2020-03-12T03:22:19.382 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6805/21762", 2020-03-12T03:22:19.382 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6809/21762", 2020-03-12T03:22:19.382 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6807/21762", 2020-03-12T03:22:19.382 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:22:19.383 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:22:19.383 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:22:19.383 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.383 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.383 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.383 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:22:19.383 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "b9daa4e6-aceb-4cc6-be00-95297a67fffc", 2020-03-12T03:22:19.384 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 0, 2020-03-12T03:22:19.384 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:22:19.384 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:22:19.384 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:22:19.385 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 0, 2020-03-12T03:22:19.386 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 0, 2020-03-12T03:22:19.386 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 10, 2020-03-12T03:22:19.387 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 22, 2020-03-12T03:22:19.387 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 30, 2020-03-12T03:22:19.387 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:22:19.387 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:22:19.387 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.388 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.388 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.388 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:22:19.388 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.388 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.388 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.388 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.389 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:22:19.389 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.389 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.389 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.389 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.389 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:22:19.390 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.390 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.390 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.390 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:22:19.390 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.390 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.390 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:22:19.391 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.392 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.392 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.392 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:22:19.392 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.392 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.392 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.392 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.393 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:22:19.393 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.393 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.393 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.393 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.394 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:22:19.394 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.394 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.394 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.394 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:22:19.394 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.394 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.395 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.395 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.395 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:22:19.395 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 17420 2020-03-12T03:22:19.395 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.395 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.395 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.396 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/17420", 2020-03-12T03:22:19.396 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/17420", 2020-03-12T03:22:19.396 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/17420", 2020-03-12T03:22:19.396 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/17420", 2020-03-12T03:22:19.396 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:22:19.397 INFO:tasks.workunit.client.0.smithi180.stdout: "exists" 2020-03-12T03:22:19.397 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.397 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.397 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.397 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:22:19.398 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "b12f5e98-07a1-4a02-bc55-25f7acbfe28e", 2020-03-12T03:22:19.398 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:22:19.398 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:22:19.398 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:22:19.398 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:22:19.399 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 15, 2020-03-12T03:22:19.399 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 28, 2020-03-12T03:22:19.399 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 35, 2020-03-12T03:22:19.399 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 0, 2020-03-12T03:22:19.399 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 31, 2020-03-12T03:22:19.400 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:22:19.400 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:22:19.400 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.400 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.400 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.401 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:22:19.401 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.401 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.401 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.402 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.403 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:22:19.403 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.403 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.403 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.403 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.403 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:22:19.404 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.404 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.404 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.404 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:22:19.404 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.404 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.404 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.405 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.405 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:22:19.405 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.405 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.405 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.405 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.405 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.406 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.407 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:22:19.407 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.407 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.407 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.407 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.407 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:22:19.407 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:22:19.408 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:22:19.409 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 21956 2020-03-12T03:22:19.409 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.409 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.409 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.409 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/21956", 2020-03-12T03:22:19.409 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/21956", 2020-03-12T03:22:19.409 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/21956", 2020-03-12T03:22:19.410 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/21956", 2020-03-12T03:22:19.410 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:22:19.410 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:22:19.410 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:22:19.410 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.410 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.410 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_xinfo": [ 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:21:36.928677+0000", 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:22:19.411 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:22:19.412 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:20:48.819564+0000", 2020-03-12T03:22:19.412 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:22:19.412 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.412 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.412 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:22:19.412 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:21:37.460428+0000", 2020-03-12T03:22:19.412 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:20:54.834285+0000", 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.413 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:22:19.414 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:21:37.988126+0000", 2020-03-12T03:22:19.414 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:22:19.414 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:22:19.414 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:22:19.414 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:22:19.414 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:21:01.176720+0000", 2020-03-12T03:22:19.414 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:22:19.415 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.415 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.415 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap": [], 2020-03-12T03:22:19.415 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap_items": [], 2020-03-12T03:22:19.415 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_temp": [], 2020-03-12T03:22:19.415 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_temp": [], 2020-03-12T03:22:19.415 INFO:tasks.workunit.client.0.smithi180.stdout: "blacklist": {}, 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profiles": { 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: "default": { 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: "k": "2", 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: "m": "2", 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: "plugin": "jerasure", 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: "technique": "reed_sol_van" 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.416 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.417 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps_queue": [], 2020-03-12T03:22:19.417 INFO:tasks.workunit.client.0.smithi180.stdout: "new_removed_snaps": [], 2020-03-12T03:22:19.417 INFO:tasks.workunit.client.0.smithi180.stdout: "new_purged_snaps": [], 2020-03-12T03:22:19.417 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_node_flags": {}, 2020-03-12T03:22:19.417 INFO:tasks.workunit.client.0.smithi180.stdout: "device_class_flags": {} 2020-03-12T03:22:19.417 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.417 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_metadata": [ 2020-03-12T03:22:19.418 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.418 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:22:19.418 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:22:19.418 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6804/21762,v1:127.0.0.1:6805/21762]", 2020-03-12T03:22:19.418 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:22:19.418 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:22:19.418 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:22:19.419 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:22:19.419 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:22:19.419 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:22:19.419 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:22:19.419 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:22:19.419 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:22:19.420 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-12T03:22:19.420 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:22:19.420 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:22:19.420 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:22:19.420 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:22:19.420 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:22:19.420 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:22:19.421 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:22:19.421 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:22:19.421 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:22:19.421 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:22:19.421 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:22:19.421 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:22:19.421 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:22:19.422 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:22:19.422 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:22:19.422 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6802/21762,v1:127.0.0.1:6803/21762]", 2020-03-12T03:22:19.422 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:22:19.422 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/21762,v1:127.0.0.1:6809/21762]", 2020-03-12T03:22:19.422 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/21762,v1:127.0.0.1:6807/21762]", 2020-03-12T03:22:19.422 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:22:19.423 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:22:19.423 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:22:19.423 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:22:19.423 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:22:19.423 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:22:19.423 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:22:19.423 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/0", 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:22:19.424 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:22:19.425 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/17420,v1:127.0.0.1:6813/17420]", 2020-03-12T03:22:19.425 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:22:19.425 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:22:19.425 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:22:19.425 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:22:19.425 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:22:19.425 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:22:19.426 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:22:19.426 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:22:19.426 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:22:19.426 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-12T03:22:19.426 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:22:19.426 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:22:19.426 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:22:19.427 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:22:19.427 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:22:19.427 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:22:19.427 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:22:19.427 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:22:19.427 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:22:19.427 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/17420,v1:127.0.0.1:6811/17420]", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:22:19.428 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/17420,v1:127.0.0.1:6817/17420]", 2020-03-12T03:22:19.429 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/17420,v1:127.0.0.1:6815/17420]", 2020-03-12T03:22:19.429 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:22:19.429 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:22:19.429 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:22:19.429 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:22:19.429 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:22:19.429 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:22:19.430 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:22:19.430 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:22:19.430 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:22:19.430 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/1", 2020-03-12T03:22:19.430 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:22:19.430 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:22:19.430 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/21956,v1:127.0.0.1:6813/21956]", 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:22:19.431 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:22:19.432 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:22:19.432 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:22:19.432 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:22:19.432 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:22:19.432 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:22:19.432 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-12T03:22:19.432 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:22:19.433 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:22:19.433 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:22:19.434 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:22:19.434 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:22:19.434 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:22:19.434 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:22:19.434 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:22:19.434 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:22:19.434 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:22:19.435 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:22:19.435 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:22:19.435 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:22:19.435 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:22:19.435 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:22:19.435 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/21956,v1:127.0.0.1:6811/21956]", 2020-03-12T03:22:19.435 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/21956,v1:127.0.0.1:6817/21956]", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/21956,v1:127.0.0.1:6815/21956]", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:22:19.436 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:22:19.437 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:22:19.437 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:22:19.437 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:22:19.437 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/2", 2020-03-12T03:22:19.437 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:22:19.437 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:22:19.437 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.438 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.438 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_first_committed": 1, 2020-03-12T03:22:19.438 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_last_committed": 39, 2020-03-12T03:22:19.438 INFO:tasks.workunit.client.0.smithi180.stdout: "crushmap": { 2020-03-12T03:22:19.438 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": [ 2020-03-12T03:22:19.438 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.438 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:22:19.439 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.0", 2020-03-12T03:22:19.439 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:22:19.439 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.439 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.439 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:22:19.439 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.1", 2020-03-12T03:22:19.440 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:22:19.440 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.440 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.440 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:22:19.440 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.2", 2020-03-12T03:22:19.440 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:22:19.440 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.441 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.441 INFO:tasks.workunit.client.0.smithi180.stdout: "types": [ 2020-03-12T03:22:19.441 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.441 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 0, 2020-03-12T03:22:19.441 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd" 2020-03-12T03:22:19.441 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.441 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.442 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:22:19.442 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "host" 2020-03-12T03:22:19.442 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.442 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.442 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 2, 2020-03-12T03:22:19.442 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "chassis" 2020-03-12T03:22:19.442 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.443 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.443 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 3, 2020-03-12T03:22:19.443 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "rack" 2020-03-12T03:22:19.443 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.444 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.444 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 4, 2020-03-12T03:22:19.444 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "row" 2020-03-12T03:22:19.444 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.444 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.444 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 5, 2020-03-12T03:22:19.444 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pdu" 2020-03-12T03:22:19.445 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.445 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.445 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 6, 2020-03-12T03:22:19.445 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pod" 2020-03-12T03:22:19.445 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.445 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.445 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 7, 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "room" 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 8, 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "datacenter" 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.446 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 9, 2020-03-12T03:22:19.447 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "zone" 2020-03-12T03:22:19.447 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.447 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.447 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 10, 2020-03-12T03:22:19.447 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "region" 2020-03-12T03:22:19.447 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.447 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "root" 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: "buckets": [ 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -1, 2020-03-12T03:22:19.448 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default", 2020-03-12T03:22:19.449 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:22:19.449 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:22:19.449 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:22:19.449 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:22:19.449 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:22:19.449 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:22:19.449 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.450 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:22:19.450 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:22:19.450 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:22:19.450 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.450 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.450 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.450 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -2, 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default~hdd", 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:22:19.451 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:22:19.452 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.452 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:22:19.452 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:22:19.452 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:22:19.452 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.452 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.452 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180", 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:22:19.453 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:22:19.454 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:22:19.454 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.454 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:22:19.454 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:22:19.454 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:22:19.454 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.454 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.455 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:22:19.455 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:22:19.455 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:22:19.455 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.455 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.455 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:22:19.456 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:22:19.456 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:22:19.456 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.456 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.456 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.456 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.456 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180~hdd", 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:22:19.457 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.458 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:22:19.458 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:22:19.458 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:22:19.458 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.458 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.458 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:22:19.458 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:22:19.459 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:22:19.459 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.459 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.459 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:22:19.459 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:22:19.459 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:22:19.459 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: "rules": [ 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_id": 0, 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_name": "replicated_rule", 2020-03-12T03:22:19.460 INFO:tasks.workunit.client.0.smithi180.stdout: "ruleset": 0, 2020-03-12T03:22:19.461 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:22:19.461 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 1, 2020-03-12T03:22:19.461 INFO:tasks.workunit.client.0.smithi180.stdout: "max_size": 10, 2020-03-12T03:22:19.461 INFO:tasks.workunit.client.0.smithi180.stdout: "steps": [ 2020-03-12T03:22:19.461 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.461 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "take", 2020-03-12T03:22:19.462 INFO:tasks.workunit.client.0.smithi180.stdout: "item": -1, 2020-03-12T03:22:19.462 INFO:tasks.workunit.client.0.smithi180.stdout: "item_name": "default" 2020-03-12T03:22:19.462 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.462 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.462 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "choose_firstn", 2020-03-12T03:22:19.462 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 0, 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "osd" 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "emit" 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.463 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.464 INFO:tasks.workunit.client.0.smithi180.stdout: "tunables": { 2020-03-12T03:22:19.464 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_tries": 0, 2020-03-12T03:22:19.464 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_fallback_tries": 0, 2020-03-12T03:22:19.464 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_total_tries": 50, 2020-03-12T03:22:19.464 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_descend_once": 1, 2020-03-12T03:22:19.464 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_vary_r": 1, 2020-03-12T03:22:19.464 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_stable": 1, 2020-03-12T03:22:19.465 INFO:tasks.workunit.client.0.smithi180.stdout: "straw_calc_version": 1, 2020-03-12T03:22:19.465 INFO:tasks.workunit.client.0.smithi180.stdout: "allowed_bucket_algs": 54, 2020-03-12T03:22:19.465 INFO:tasks.workunit.client.0.smithi180.stdout: "profile": "jewel", 2020-03-12T03:22:19.465 INFO:tasks.workunit.client.0.smithi180.stdout: "optimal_tunables": 1, 2020-03-12T03:22:19.465 INFO:tasks.workunit.client.0.smithi180.stdout: "legacy_tunables": 0, 2020-03-12T03:22:19.465 INFO:tasks.workunit.client.0.smithi180.stdout: "minimum_required_version": "jewel", 2020-03-12T03:22:19.465 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables": 1, 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables2": 1, 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v2_rules": 0, 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables3": 1, 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v3_rules": 0, 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v4_buckets": 1, 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables5": 1, 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v5_rules": 0 2020-03-12T03:22:19.466 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.467 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_args": {} 2020-03-12T03:22:19.467 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.467 INFO:tasks.workunit.client.0.smithi180.stdout: "fsmap": { 2020-03-12T03:22:19.467 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:22:19.467 INFO:tasks.workunit.client.0.smithi180.stdout: "default_fscid": -1, 2020-03-12T03:22:19.467 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": { 2020-03-12T03:22:19.467 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": {}, 2020-03-12T03:22:19.468 INFO:tasks.workunit.client.0.smithi180.stdout: "ro_compat": {}, 2020-03-12T03:22:19.468 INFO:tasks.workunit.client.0.smithi180.stdout: "incompat": { 2020-03-12T03:22:19.468 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_1": "base v0.20", 2020-03-12T03:22:19.468 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_2": "client writeable ranges", 2020-03-12T03:22:19.468 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_3": "default file layouts on dirs", 2020-03-12T03:22:19.468 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_4": "dir inode in separate object", 2020-03-12T03:22:19.468 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_5": "mds uses versioned encoding", 2020-03-12T03:22:19.469 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-12T03:22:19.469 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_8": "no anchor table", 2020-03-12T03:22:19.469 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_9": "file layout v2", 2020-03-12T03:22:19.469 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_10": "snaprealm v2" 2020-03-12T03:22:19.469 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.469 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.469 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_flags": { 2020-03-12T03:22:19.470 INFO:tasks.workunit.client.0.smithi180.stdout: "enable_multiple": false, 2020-03-12T03:22:19.470 INFO:tasks.workunit.client.0.smithi180.stdout: "ever_enabled_multiple": false 2020-03-12T03:22:19.470 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.470 INFO:tasks.workunit.client.0.smithi180.stdout: "standbys": [], 2020-03-12T03:22:19.470 INFO:tasks.workunit.client.0.smithi180.stdout: "filesystems": [] 2020-03-12T03:22:19.470 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_first_committed": 1, 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_last_committed": 1, 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: "auth": { 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 5, 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: "num_secrets": 3 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.471 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 1, 2020-03-12T03:22:19.472 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_active": 1, 2020-03-12T03:22:19.472 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_unknown": 0, 2020-03-12T03:22:19.472 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osd": 3, 2020-03-12T03:22:19.472 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_sum": { 2020-03-12T03:22:19.472 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:22:19.472 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 48300, 2020-03-12T03:22:19.472 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:22:19.473 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:22:19.473 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:22:19.473 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:22:19.473 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:22:19.473 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:22:19.473 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:22:19.473 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 100, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 100, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:22:19.474 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:22:19.475 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:22:19.475 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:22:19.475 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:22:19.475 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:22:19.475 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:22:19.475 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:22:19.475 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:22:19.476 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:22:19.477 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:22:19.477 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:22:19.477 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:22:19.477 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:22:19.477 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:22:19.477 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:22:19.477 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:22:19.478 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:22:19.478 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.478 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:22:19.478 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:22:19.478 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:22:19.478 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:22:19.478 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 0, 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 0, 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.479 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 16, 2020-03-12T03:22:19.480 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 16, 2020-03-12T03:22:19.480 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:22:19.480 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:22:19.480 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 0 2020-03-12T03:22:19.480 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.480 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum": { 2020-03-12T03:22:19.480 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:22:19.481 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:22:19.481 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:22:19.481 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:22:19.481 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:22:19.481 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:22:19.481 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:22:19.481 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3179136, 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 33152, 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 14, 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145713, 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393664, 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867111936, 2020-03-12T03:22:19.482 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:22:19.483 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33947648, 2020-03-12T03:22:19.483 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 403763, 2020-03-12T03:22:19.483 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:22:19.483 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:22:19.483 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:22:19.483 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 14968, 2020-03-12T03:22:19.483 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221210504 2020-03-12T03:22:19.484 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.484 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:22:19.484 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:22:19.484 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:22:19.484 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 1, 2020-03-12T03:22:19.484 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:22:19.484 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:22:19.485 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:22:19.485 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.485 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:22:19.485 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:22:19.485 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:22:19.485 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:22:19.485 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum_by_class": { 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: "hdd": { 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:22:19.486 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:22:19.487 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:22:19.487 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:22:19.487 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:22:19.487 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:22:19.487 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:22:19.487 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3179136, 2020-03-12T03:22:19.488 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 33152, 2020-03-12T03:22:19.488 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 14, 2020-03-12T03:22:19.488 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145713, 2020-03-12T03:22:19.488 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393664, 2020-03-12T03:22:19.488 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:22:19.488 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:22:19.488 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867111936, 2020-03-12T03:22:19.489 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:22:19.489 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33947648, 2020-03-12T03:22:19.489 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 403763, 2020-03-12T03:22:19.489 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:22:19.489 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:22:19.489 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:22:19.490 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 14968, 2020-03-12T03:22:19.490 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221210504 2020-03-12T03:22:19.490 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.490 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:22:19.490 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:22:19.490 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:22:19.490 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 1, 2020-03-12T03:22:19.491 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:22:19.491 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:22:19.491 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:22:19.491 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.491 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:22:19.491 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:22:19.492 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:22:19.492 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:22:19.493 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:22:19.493 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.493 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:22:19.493 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:22:19.493 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.493 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_stats": [ 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: "poolid": 1, 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 1, 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 48300, 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:22:19.494 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:22:19.495 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:22:19.495 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:22:19.495 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:22:19.495 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:22:19.495 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:22:19.495 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:22:19.495 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:22:19.496 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:22:19.496 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:22:19.496 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:22:19.496 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 100, 2020-03-12T03:22:19.496 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 100, 2020-03-12T03:22:19.496 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:22:19.496 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:22:19.497 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:22:19.497 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:22:19.498 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:22:19.498 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:22:19.498 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:22:19.498 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:22:19.498 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:22:19.499 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:22:19.499 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:22:19.500 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:22:19.500 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:22:19.500 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:22:19.500 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:22:19.501 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:22:19.501 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:22:19.501 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:22:19.501 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:22:19.502 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:22:19.502 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:22:19.502 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:22:19.502 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:22:19.502 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:22:19.503 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:22:19.503 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.503 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:22:19.503 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:22:19.504 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:22:19.504 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:22:19.504 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 19660800, 2020-03-12T03:22:19.504 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 144900, 2020-03-12T03:22:19.505 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:22:19.505 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:22:19.505 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:22:19.505 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:22:19.506 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:22:19.506 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.506 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 16, 2020-03-12T03:22:19.506 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 16, 2020-03-12T03:22:19.507 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:22:19.507 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:22:19.507 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 3 2020-03-12T03:22:19.507 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.508 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.508 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_stats": [ 2020-03-12T03:22:19.508 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.508 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:22:19.509 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 146028888074 2020-03-12T03:22:19.509 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.509 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.509 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:22:19.510 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 42949672975 2020-03-12T03:22:19.510 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.510 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.510 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:22:19.511 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 150323855370 2020-03-12T03:22:19.511 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.511 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.512 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_state": [ 2020-03-12T03:22:19.512 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.512 INFO:tasks.workunit.client.0.smithi180.stdout: "state": "active+undersized+degraded", 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 1 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_osd": [ 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:22:19.513 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:22:19.514 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:22:19.514 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.514 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.514 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:22:19.514 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:22:19.514 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 0, 2020-03-12T03:22:19.514 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 1, 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.515 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snaps": [ 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: "pool": 1, 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snaps": {} 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: "servicemap": { 2020-03-12T03:22:19.516 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:22:19.517 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:20:41.927836+0000", 2020-03-12T03:22:19.518 INFO:tasks.workunit.client.0.smithi180.stdout: "services": {} 2020-03-12T03:22:19.518 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:22:19.518 INFO:tasks.workunit.client.0.smithi180.stdout: "paxos": { 2020-03-12T03:22:19.518 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:22:19.519 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 301, 2020-03-12T03:22:19.519 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pn": 0, 2020-03-12T03:22:19.519 INFO:tasks.workunit.client.0.smithi180.stdout: "accepted_pn": 0 2020-03-12T03:22:19.519 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:22:19.520 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:22:19.521 INFO:tasks.workunit.client.0.smithi180.stderr:report 2088821409 2020-03-12T03:22:19.522 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-12T03:22:19.522 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-12T03:22:19.522 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-12T03:22:19.522 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-12T03:22:19.522 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:19.523 INFO:tasks.workunit.client.0.smithi180.stdout:writing non-divergent object existing_4 2020-03-12T03:22:19.667 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:19.667 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-12T03:21:44.899000+0000 28'100 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:19.667 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:19.667 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:19.668 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:19.676 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-12T03:22:19.720 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-12T03:22:19.721 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:19.721 INFO:tasks.workunit.client.0.smithi180.stdout:delay recovery 2020-03-12T03:22:20.040 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:20.040 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-12T03:21:44.899000+0000 28'100 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:20.040 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:20.040 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:20.041 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:20.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-12T03:22:20.051 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-12T03:22:20.051 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-12T03:22:20.051 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-12T03:22:20.051 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:22:20.051 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:22:20.051 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:22:20.052 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.0.asok 2020-03-12T03:22:20.052 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-12T03:22:20.052 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.0.asok set_recovery_delay 100000 2020-03-12T03:22:20.142 INFO:tasks.workunit.client.0.smithi180.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-12T03:22:20.149 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-12T03:22:20.150 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-12T03:22:20.151 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-12T03:22:20.151 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-12T03:22:20.151 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:22:20.151 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:22:20.151 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:22:20.151 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.2.asok 2020-03-12T03:22:20.152 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-12T03:22:20.152 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.2.asok set_recovery_delay 100000 2020-03-12T03:22:20.241 INFO:tasks.workunit.client.0.smithi180.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-12T03:22:20.248 INFO:tasks.workunit.client.0.smithi180.stdout:revive divergent 1 2020-03-12T03:22:20.249 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-12T03:22:20.249 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:20.569 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:20.569 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-12T03:21:44.899000+0000 28'100 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:20.569 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:20.569 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:20.570 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:20.578 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-12T03:22:21.219 INFO:tasks.workunit.client.0.smithi180.stderr:noup is set 2020-03-12T03:22:21.232 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-12T03:22:21.232 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:22:21.232 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:22:21.232 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-12T03:22:21.232 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:22:21.232 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-12T03:22:21.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:22:21.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:22:21.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:22:21.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:22:21.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T03:22:21.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T03:22:21.234 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:22:21.234 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:22:21.234 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:22:21.234 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:22:21.234 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:22:21.235 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:22:21.235 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:22:21.235 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:22:21.235 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:22:21.235 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:22:21.235 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:22:21.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:22:21.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:22:21.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:22:21.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:22:21.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:22:21.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:22:21.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:22:21.237 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:22:21.237 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-12T03:22:21.237 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-12T03:22:21.237 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:22:21.237 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-12T03:22:21.237 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-12T03:22:21.237 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:22:21.238 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:22:21.238 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:22:21.238 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T03:22:21.257 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:21.255+0000 7fd130387ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:22:21.258 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:21.257+0000 7fd130387ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:22:21.260 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:21.259+0000 7fd130387ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:22:21.582 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-12T03:22:21.582 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-12T03:22:21.780 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:21.779+0000 7fd130387ec0 -1 Falling back to public interface 2020-03-12T03:22:22.617 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:22.616+0000 7fd130387ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-12T03:22:26.583 INFO:tasks.workunit.client.0.smithi180.stdout:delay recovery divergent 2020-03-12T03:22:26.583 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-12T03:22:26.584 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:26.925 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:26.925 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:21:44.899000+0000 39'101 39:125 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:26.926 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:26.926 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:26.927 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:26.936 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-12T03:22:26.936 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-12T03:22:26.936 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-12T03:22:26.936 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:22:26.937 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:22:26.937 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:22:26.937 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.1.asok 2020-03-12T03:22:26.937 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-12T03:22:26.937 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.1.asok set_recovery_delay 100000 2020-03-12T03:22:27.027 INFO:tasks.workunit.client.0.smithi180.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-12T03:22:27.034 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-12T03:22:27.798 INFO:tasks.workunit.client.0.smithi180.stderr:noup is unset 2020-03-12T03:22:27.807 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-12T03:22:27.807 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:22:27.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:22:27.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:22:27.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:22:27.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:22:27.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:22:27.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:22:27.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:22:27.809 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:22:28.179 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up in weight 1 up_from 34 up_thru 38 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/21762,v1:127.0.0.1:6803/21762] [v2:127.0.0.1:6804/21762,v1:127.0.0.1:6805/21762] exists,up 99ca4acf-0655-4af6-81b1-279a966fc6aa 2020-03-12T03:22:28.180 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:22:28.180 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:22:28.180 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:22:28.180 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:22:28.180 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-12T03:22:28.180 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:22:28.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:22:28.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:22:28.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:22:28.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:22:28.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:22:28.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:22:28.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:22:28.522 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/23417,v1:127.0.0.1:6819/23417] [v2:127.0.0.1:6820/23417,v1:127.0.0.1:6821/23417] exists,up b9daa4e6-aceb-4cc6-be00-95297a67fffc 2020-03-12T03:22:28.522 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:22:28.522 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:22:28.522 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:22:28.523 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:22:28.523 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-12T03:22:28.523 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:22:28.523 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:22:28.523 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:22:28.523 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:22:28.523 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:22:28.524 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:22:28.524 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:22:28.524 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:22:28.914 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/21956,v1:127.0.0.1:6811/21956] [v2:127.0.0.1:6812/21956,v1:127.0.0.1:6813/21956] exists,up b12f5e98-07a1-4a02-bc55-25f7acbfe28e 2020-03-12T03:22:28.914 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:22:28.914 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:22:28.914 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:22:28.914 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:29.236 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:29.237 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:21:44.899000+0000 39'101 39:125 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:29.237 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:29.237 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:29.238 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:29.246 INFO:tasks.workunit.client.0.smithi180.stdout:wait for peering 2020-03-12T03:22:29.247 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-12T03:22:29.247 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:29.565 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:29.565 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:21:44.899000+0000 39'101 39:125 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:29.565 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:29.566 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:29.567 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:29.575 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-12T03:22:40.926 INFO:tasks.workunit.client.0.smithi180.stdout:killing divergent 1 2020-03-12T03:22:40.926 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-12T03:22:40.927 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:41.251 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:41.251 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-12T03:22:27.955286+0000 39'101 44:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:41.251 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:41.252 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:41.253 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:41.261 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-12T03:22:41.261 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:22:41.262 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:22:41.262 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:22:41.262 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:22:41.262 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:22:41.366 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:22:41.367 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-12T03:22:41.367 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-12T03:22:41.367 INFO:tasks.workunit.client.0.smithi180.stdout:reviving divergent 1 2020-03-12T03:22:41.690 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:22:41.690 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-12T03:22:27.955286+0000 39'101 44:133 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:22:41.690 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:22:41.690 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:22:41.692 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:22:41.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-12T03:22:41.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:22:41.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:22:41.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-12T03:22:41.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:22:41.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-12T03:22:41.705 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:22:41.705 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:22:41.705 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:22:41.705 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:22:41.705 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T03:22:41.705 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T03:22:41.705 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:22:41.706 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:22:41.706 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:22:41.706 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:22:41.706 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:22:41.706 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:22:41.706 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:22:41.707 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:22:41.707 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:22:41.707 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:22:41.707 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:22:41.707 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:22:41.707 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:22:41.707 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:22:41.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:22:41.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:22:41.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:22:41.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:22:41.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:22:41.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-12T03:22:41.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-12T03:22:41.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:22:41.709 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-12T03:22:41.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-12T03:22:41.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:22:41.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:22:41.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:22:41.709 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T03:22:41.729 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:41.727+0000 7fbd9ef71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:22:41.731 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:41.729+0000 7fbd9ef71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:22:41.733 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:41.731+0000 7fbd9ef71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:22:42.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-12T03:22:42.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:22:42.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:22:42.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:22:42.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:22:42.051 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:22:42.051 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:22:42.051 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:22:42.051 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:22:42.051 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:22:42.248 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:42.245+0000 7fbd9ef71ec0 -1 Falling back to public interface 2020-03-12T03:22:42.395 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/23417,v1:127.0.0.1:6819/23417] [v2:127.0.0.1:6820/23417,v1:127.0.0.1:6821/23417] exists,up b9daa4e6-aceb-4cc6-be00-95297a67fffc 2020-03-12T03:22:42.395 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:22:42.395 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:22:42.395 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:22:42.396 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-12T03:22:43.164 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:22:43.161+0000 7fbd9ef71ec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-12T03:23:02.396 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-12T03:23:02.396 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-12T03:23:02.397 INFO:tasks.workunit.client.0.smithi180.stdout:allowing recovery 2020-03-12T03:23:02.725 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:23:02.725 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-12T03:22:44.038951+0000 44'102 47:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:23:02.726 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:23:02.726 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:23:02.727 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:23:02.735 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-12T03:23:02.735 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-12T03:23:02.856 INFO:tasks.workunit.client.0.smithi180.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-12T03:23:02.864 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-12T03:23:02.864 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-12T03:23:02.986 INFO:tasks.workunit.client.0.smithi180.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-12T03:23:02.994 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-12T03:23:02.994 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-12T03:23:03.115 INFO:tasks.workunit.client.0.smithi180.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-12T03:23:03.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-12T03:23:03.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-12T03:23:03.123 INFO:tasks.workunit.client.0.smithi180.stdout:reading divergent objects 2020-03-12T03:23:03.440 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:23:03.440 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-12T03:22:44.038951+0000 44'102 47:150 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:21:03.424415+0000 0'0 2020-03-12T03:21:03.424415+0000 0 2020-03-12T03:23:03.441 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:23:03.441 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:23:03.442 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:23:03.451 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-12T03:23:03.451 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-12T03:23:03.452 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:23:03.452 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-12T03:23:03.490 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:23:03.491 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-12T03:23:03.519 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:23:03.519 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-12T03:23:03.548 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:23:03.548 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-12T03:23:03.576 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-12T03:23:03.577 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-12T03:23:03.578 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-12T03:23:03.581 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-12T03:21:25.596621+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:23:03.581 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-12T03:21:25.596621+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:23:03.581 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-12T03:23:03.582 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-12T03:23:03.582 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-12T03:23:03.582 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-12T03:23:03.582 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2020-03-12T03:23:03.582 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-12T03:21:25.596621+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:23:03.582 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:22:27.921+0000 7fd10e542700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-12T03:23:03.583 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-12T03:23:03.583 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-12T03:23:03.585 INFO:tasks.workunit.client.0.smithi180.stdout:success 2020-03-12T03:23:03.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-12T03:23:03.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-12T03:23:03.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-12T03:23:03.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-12T03:23:04.053 INFO:tasks.workunit.client.0.smithi180.stderr:pool 'test' does not exist 2020-03-12T03:23:04.067 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-12T03:23:04.068 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:23:04.068 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:23:04.068 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:23:04.068 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:23:04.069 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:23:04.169 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:23:04.169 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-12T03:23:04.170 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-12T03:23:04.170 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-12T03:23:04.170 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-12T03:23:04.170 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:23:04.170 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:23:04.170 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:23:04.170 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:23:04.171 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:23:04.174 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:23:04.174 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-12T03:23:04.174 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-12T03:23:04.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-12T03:23:04.175 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-12T03:23:04.176 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-12T03:23:04.176 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-12T03:23:04.177 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:23:04.177 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-12T03:23:04.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-12T03:23:04.178 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:23:04.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-12T03:23:04.179 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-12T03:23:04.179 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-12T03:23:04.190 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-12T03:23:04.190 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:04.191 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:04.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.16628 2020-03-12T03:23:04.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-12T03:23:04.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-12T03:23:04.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-12T03:23:04.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-12T03:23:04.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-12T03:23:04.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-12T03:23:04.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-12T03:23:04.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-12T03:23:04.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-12T03:23:04.193 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:23:04.194 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:23:04.194 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:23:04.194 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:23:04.194 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:23:04.195 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:23:04.195 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-12T03:23:04.195 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-12T03:23:04.195 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-12T03:23:04.196 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-12T03:23:04.196 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-12T03:23:04.196 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-12T03:23:04.198 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:23:04.198 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-12T03:23:04.198 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-12T03:23:04.198 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:23:04.199 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-12T03:23:04.199 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-12T03:23:04.199 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-12T03:23:04.200 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-12T03:23:04.200 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:04.200 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:04.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.16628 2020-03-12T03:23:04.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-12T03:23:04.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-12T03:23:04.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-12T03:23:04.202 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-12T03:23:04.202 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:04.203 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:04.203 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.16628 2020-03-12T03:23:04.204 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-12T03:23:04.204 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-12T03:23:04.205 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-12T03:23:04.205 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-12T03:23:04.205 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-12T03:23:04.205 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-12T03:23:04.205 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-12T03:23:04.205 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-12T03:23:04.205 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-12T03:23:04.206 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-12T03:23:04.207 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-12T03:23:04.207 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-12T03:23:04.207 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:23:04.208 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:23:04.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-12T03:23:04.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-12T03:23:04.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-12T03:23:04.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-12T03:23:04.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-12T03:23:04.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-12T03:23:04.210 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-12T03:23:04.526 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-12T03:23:04.526 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:23:04.526 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:23:04.527 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:23:04.527 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:04.527 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:04.527 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:04.528 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-12T03:23:04.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-12T03:23:04.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-12T03:23:04.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-12T03:23:04.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-12T03:23:04.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-12T03:23:04.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-12T03:23:04.670 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-12T03:23:04.670 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-12T03:23:04.671 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-12T03:23:04.671 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:23:04.671 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:04.672 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:04.673 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-mon.a.asok 2020-03-12T03:23:04.673 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-12T03:23:04.673 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.16628/ceph-mon.a.asok config get fsid 2020-03-12T03:23:04.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-12T03:23:04.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-12T03:23:04.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-12T03:23:04.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-12T03:23:04.759 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-12T03:23:04.759 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-12T03:23:04.759 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-12T03:23:04.759 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-12T03:23:04.759 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:23:04.759 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:04.759 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:04.760 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-mon.a.asok 2020-03-12T03:23:04.760 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-12T03:23:04.760 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.16628/ceph-mon.a.asok config get mon_host 2020-03-12T03:23:04.844 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-12T03:23:04.844 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-12T03:23:04.844 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-12T03:23:04.844 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-12T03:23:04.845 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-12T03:23:04.845 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-12T03:23:04.845 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-12T03:23:05.710 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-12T03:23:05.710 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:23:05.710 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:23:05.710 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:23:05.710 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:05.710 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:05.711 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:05.711 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-12T03:23:05.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-12T03:23:05.739 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-12T03:23:05.739 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-12T03:23:05.739 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:23:05.739 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:23:05.739 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-12T03:23:05.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:23:05.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-12T03:23:05.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:23:05.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:23:05.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:23:05.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:23:05.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-12T03:23:05.741 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-12T03:23:05.741 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:23:05.741 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:23:05.741 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:23:05.741 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:23:05.741 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:23:05.741 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:23:05.742 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:23:05.742 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:05.742 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:05.742 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:05.742 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:05.742 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:23:05.742 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:23:05.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:23:05.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:23:05.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:23:05.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:23:05.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:23:05.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:23:05.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:23:05.744 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:23:05.744 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-12T03:23:05.744 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:23:05.744 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=59ecfd79-e15b-4216-a9fb-a337e29afe1e 2020-03-12T03:23:05.745 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 59ecfd79-e15b-4216-a9fb-a337e29afe1e' 2020-03-12T03:23:05.745 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:23:05.745 INFO:tasks.workunit.client.0.smithi180.stdout:add osd0 59ecfd79-e15b-4216-a9fb-a337e29afe1e 2020-03-12T03:23:05.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAZq2lenyIdLRAATGxKasQsMPVsRrj47JI6RQ== 2020-03-12T03:23:05.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAZq2lenyIdLRAATGxKasQsMPVsRrj47JI6RQ=="}' 2020-03-12T03:23:05.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 59ecfd79-e15b-4216-a9fb-a337e29afe1e -i td/divergent-priors/0/new.json 2020-03-12T03:23:06.100 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:23:06.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-12T03:23:06.110 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAZq2lenyIdLRAATGxKasQsMPVsRrj47JI6RQ== --osd-uuid 59ecfd79-e15b-4216-a9fb-a337e29afe1e 2020-03-12T03:23:06.139 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:06.137+0000 7f3a0d670ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:06.140 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:06.139+0000 7f3a0d670ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:06.143 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:06.142+0000 7f3a0d670ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:06.144 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:06.142+0000 7f3a0d670ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:23:06.144 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:06.142+0000 7f3a0d670ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:23:06.144 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:06.142+0000 7f3a0d670ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:23:06.184 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:06.183+0000 7f3a0d670ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-12T03:23:09.182 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-12T03:23:09.183 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:23:09.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-12T03:23:09.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:23:09.184 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd0 key to auth repository 2020-03-12T03:23:09.637 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-12T03:23:09.637 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:23:09.638 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:23:09.638 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:23:09.638 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:23:09.638 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.0 2020-03-12T03:23:09.663 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:09.662+0000 7f4ae3071ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:09.665 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:09.663+0000 7f4ae3071ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:09.666 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:09.665+0000 7f4ae3071ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:10.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-12T03:23:10.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:23:10.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:23:10.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:23:10.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:23:10.001 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:23:10.001 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:23:10.001 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:23:10.001 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:23:10.001 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:23:10.182 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:10.181+0000 7f4ae3071ec0 -1 Falling back to public interface 2020-03-12T03:23:10.344 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:23:10.934 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:10.933+0000 7f4ae3071ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-12T03:23:11.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:23:11.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:23:11.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:23:11.346 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:23:11.346 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:23:11.346 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:23:11.696 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/24719,v1:127.0.0.1:6803/24719] [v2:127.0.0.1:6804/24719,v1:127.0.0.1:6805/24719] exists,up 59ecfd79-e15b-4216-a9fb-a337e29afe1e 2020-03-12T03:23:11.697 INFO:tasks.workunit.client.0.smithi180.stdout:add osd1 eb5be901-51a3-4f54-bb09-dd4860cf0f9c 2020-03-12T03:23:11.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:23:11.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:23:11.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:23:11.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-12T03:23:11.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-12T03:23:11.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:23:11.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:23:11.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-12T03:23:11.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:23:11.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-12T03:23:11.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:23:11.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:23:11.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:23:11.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:23:11.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T03:23:11.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T03:23:11.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:23:11.700 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:23:11.700 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:23:11.700 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:23:11.700 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:23:11.700 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:23:11.700 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:23:11.700 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:11.701 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:11.701 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:11.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:11.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:23:11.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:23:11.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:23:11.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:23:11.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:23:11.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:23:11.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:23:11.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:23:11.703 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:23:11.703 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:23:11.703 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-12T03:23:11.703 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:23:11.703 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=eb5be901-51a3-4f54-bb09-dd4860cf0f9c 2020-03-12T03:23:11.704 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 eb5be901-51a3-4f54-bb09-dd4860cf0f9c' 2020-03-12T03:23:11.704 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:23:11.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAfq2let7yEKhAA84nYw0z2zgA9K44bNJzEXQ== 2020-03-12T03:23:11.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAfq2let7yEKhAA84nYw0z2zgA9K44bNJzEXQ=="}' 2020-03-12T03:23:11.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new eb5be901-51a3-4f54-bb09-dd4860cf0f9c -i td/divergent-priors/1/new.json 2020-03-12T03:23:12.197 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:23:12.207 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-12T03:23:12.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAfq2let7yEKhAA84nYw0z2zgA9K44bNJzEXQ== --osd-uuid eb5be901-51a3-4f54-bb09-dd4860cf0f9c 2020-03-12T03:23:12.237 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:12.236+0000 7f224cf90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:12.258 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:12.256+0000 7f224cf90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:12.258 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:12.257+0000 7f224cf90ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:12.259 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:12.258+0000 7f224cf90ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:23:12.259 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:12.258+0000 7f224cf90ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:23:12.259 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:12.258+0000 7f224cf90ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:23:12.382 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:12.381+0000 7f224cf90ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-12T03:23:15.435 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-12T03:23:15.435 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:23:15.436 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd1 key to auth repository 2020-03-12T03:23:15.436 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-12T03:23:15.437 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:23:15.886 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-12T03:23:15.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:23:15.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:23:15.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:23:15.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:23:15.887 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T03:23:15.913 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:15.912+0000 7f0ae2531ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:15.915 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:15.914+0000 7f0ae2531ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:15.917 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:15.916+0000 7f0ae2531ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:16.255 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-12T03:23:16.255 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:23:16.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:23:16.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:23:16.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:23:16.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:23:16.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:23:16.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:23:16.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:23:16.257 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:23:16.438 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:16.436+0000 7f0ae2531ec0 -1 Falling back to public interface 2020-03-12T03:23:16.601 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:23:17.230 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:17.228+0000 7f0ae2531ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-12T03:23:17.602 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:23:17.603 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:23:17.603 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:23:17.603 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:23:17.603 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:23:17.603 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:23:17.970 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25074,v1:127.0.0.1:6811/25074] [v2:127.0.0.1:6812/25074,v1:127.0.0.1:6813/25074] exists,up eb5be901-51a3-4f54-bb09-dd4860cf0f9c 2020-03-12T03:23:17.970 INFO:tasks.workunit.client.0.smithi180.stdout:add osd2 3f99c107-f1bf-46d1-92a5-89dec9b23f29 2020-03-12T03:23:17.970 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:23:17.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:23:17.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:23:17.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-12T03:23:17.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-12T03:23:17.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:23:17.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:23:17.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-12T03:23:17.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:23:17.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-12T03:23:17.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:23:17.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:23:17.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:23:17.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:23:17.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-12T03:23:17.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-12T03:23:17.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:23:17.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:23:17.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:23:17.973 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:23:17.974 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:23:17.974 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:23:17.974 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:23:17.974 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:17.974 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:17.974 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:17.974 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:23:17.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:23:17.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:23:17.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:23:17.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-12T03:23:17.976 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:23:17.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3f99c107-f1bf-46d1-92a5-89dec9b23f29 2020-03-12T03:23:17.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 3f99c107-f1bf-46d1-92a5-89dec9b23f29' 2020-03-12T03:23:17.977 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:23:17.986 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAlq2letbGtOhAAYvYG+6B9ok4u5PBnnhiKfw== 2020-03-12T03:23:17.986 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAlq2letbGtOhAAYvYG+6B9ok4u5PBnnhiKfw=="}' 2020-03-12T03:23:17.986 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3f99c107-f1bf-46d1-92a5-89dec9b23f29 -i td/divergent-priors/2/new.json 2020-03-12T03:23:18.518 INFO:tasks.workunit.client.0.smithi180.stdout:2 2020-03-12T03:23:18.528 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-12T03:23:18.529 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAlq2letbGtOhAAYvYG+6B9ok4u5PBnnhiKfw== --osd-uuid 3f99c107-f1bf-46d1-92a5-89dec9b23f29 2020-03-12T03:23:18.564 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:18.556+0000 7fe8273eeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:18.564 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:18.558+0000 7fe8273eeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:18.564 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:18.561+0000 7fe8273eeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:18.565 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:18.561+0000 7fe8273eeec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:23:18.565 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:18.561+0000 7fe8273eeec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:23:18.565 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:18.561+0000 7fe8273eeec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:23:18.679 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:18.678+0000 7fe8273eeec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-12T03:23:21.755 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-12T03:23:21.755 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:23:21.756 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd2 key to auth repository 2020-03-12T03:23:21.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-12T03:23:21.757 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:23:22.212 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-12T03:23:22.212 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:23:22.213 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:23:22.213 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:23:22.213 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:23:22.213 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.2 2020-03-12T03:23:22.240 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:22.238+0000 7fdf951e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:22.241 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:22.240+0000 7fdf951e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:22.242 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:22.241+0000 7fdf951e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:23:22.569 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-12T03:23:22.569 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:23:22.570 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:23:22.570 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:23:22.570 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:23:22.570 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:23:22.571 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:23:22.571 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:23:22.571 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:23:22.571 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:23:22.757 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:22.755+0000 7fdf951e0ec0 -1 Falling back to public interface 2020-03-12T03:23:22.910 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:23:23.478 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:23:23.477+0000 7fdf951e0ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-12T03:23:23.912 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:23:23.912 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:23:23.912 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:23:23.912 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:23:23.912 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:23:23.913 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:23:24.264 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/25446,v1:127.0.0.1:6819/25446] [v2:127.0.0.1:6820/25446,v1:127.0.0.1:6821/25446] exists,up 3f99c107-f1bf-46d1-92a5-89dec9b23f29 2020-03-12T03:23:24.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:23:24.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:23:24.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:23:24.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-12T03:23:24.923 INFO:tasks.workunit.client.0.smithi180.stderr:noout is set 2020-03-12T03:23:24.931 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-12T03:23:25.587 INFO:tasks.workunit.client.0.smithi180.stderr:noin is set 2020-03-12T03:23:25.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-12T03:23:26.275 INFO:tasks.workunit.client.0.smithi180.stderr:nodown is set 2020-03-12T03:23:26.283 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-12T03:23:26.283 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-12T03:23:26.760 INFO:tasks.workunit.client.0.smithi180.stderr:pool 'test' already exists 2020-03-12T03:23:26.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-12T03:23:27.769 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-12T03:23:28.334 INFO:tasks.workunit.client.0.smithi180.stderr:set pool 1 size to 3 2020-03-12T03:23:28.341 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-12T03:23:28.959 INFO:tasks.workunit.client.0.smithi180.stderr:set pool 1 min_size to 2 2020-03-12T03:23:28.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-12T03:23:28.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:23:28.968 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:23:29.315 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:23:29.362 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:23:29.362 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:23:29.363 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:23:29.363 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:29.363 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:23:29.466 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-12T03:23:29.467 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-12T03:23:29.467 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:29.467 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:23:29.595 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-03-12T03:23:29.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-38654705669' 2020-03-12T03:23:29.596 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:29.596 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:23:29.723 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542147 2020-03-12T03:23:29.724 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-38654705669 2-60129542147' 2020-03-12T03:23:29.724 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:29.724 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-12T03:23:29.725 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:29.725 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:23:29.725 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-12T03:23:29.725 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:29.726 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-12T03:23:29.727 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-12T03:23:29.727 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:23:29.727 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836486 2020-03-12T03:23:30.082 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-12T03:23:30.082 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:23:31.082 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:23:31.083 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:23:31.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-12T03:23:31.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:31.442 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705669 2020-03-12T03:23:31.442 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:31.443 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:23:31.444 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705669 2020-03-12T03:23:31.444 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:31.444 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705669 2020-03-12T03:23:31.444 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-03-12T03:23:31.445 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 38654705669 2020-03-12T03:23:31.445 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:23:31.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705669 -lt 38654705669 2020-03-12T03:23:31.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:31.804 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542147 2020-03-12T03:23:31.804 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:31.804 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:23:31.805 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542147 2020-03-12T03:23:31.805 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:31.806 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542147 2020-03-12T03:23:31.806 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2020-03-12T03:23:31.806 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:23:31.807 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 60129542147 2020-03-12T03:23:32.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542147 2020-03-12T03:23:32.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-12T03:23:32.167 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:23:32.167 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:23:32.167 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:23:32.167 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:23:32.167 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-12T03:23:32.167 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:23:32.167 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:23:32.168 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:23:32.168 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:23:32.168 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:23:32.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:23:32.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:23:32.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:23:32.256 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:23:32.256 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:23:32.598 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:23:32.598 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:23:32.599 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:23:32.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:23:32.599 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:32.599 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:23:32.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-12T03:23:32.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-12T03:23:32.743 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:32.743 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:23:32.871 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705671 2020-03-12T03:23:32.872 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-38654705671' 2020-03-12T03:23:32.872 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:32.872 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:23:33.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-12T03:23:33.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-38654705671 2-60129542150' 2020-03-12T03:23:33.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:33.000 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-12T03:23:33.000 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:33.001 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:23:33.001 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-12T03:23:33.001 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:33.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-12T03:23:33.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-12T03:23:33.003 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:23:33.003 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836489 2020-03-12T03:23:33.342 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-12T03:23:33.342 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:33.343 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705671 2020-03-12T03:23:33.343 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:33.343 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:23:33.344 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705671 2020-03-12T03:23:33.344 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:33.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705671 2020-03-12T03:23:33.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2020-03-12T03:23:33.345 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:23:33.345 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 38654705671 2020-03-12T03:23:33.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705671 -lt 38654705671 2020-03-12T03:23:33.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:33.718 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-12T03:23:33.718 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:33.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:23:33.719 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-12T03:23:33.719 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:33.720 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-12T03:23:33.720 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-12T03:23:33.720 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:23:33.720 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 60129542150 2020-03-12T03:23:34.062 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542150 -lt 60129542150 2020-03-12T03:23:34.062 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:23:34.062 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:23:34.063 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:23:34.529 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:23:34.529 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:23:34.529 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:23:34.529 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:23:34.529 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:23:34.529 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:23:34.529 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:23:34.530 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:23:34.858 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-12T03:23:34.858 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:23:34.859 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:23:34.859 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:23:35.304 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-12T03:23:35.305 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-12T03:23:35.305 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-12T03:23:35.305 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-12T03:23:35.305 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-12T03:23:35.627 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:23:35.635 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-12T03:23:35.636 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-12T03:23:35.636 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:23:35.636 INFO:tasks.workunit.client.0.smithi180.stdout:primary and soon to be divergent is 1 2020-03-12T03:23:35.956 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:23:35.957 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-12T03:23:27.426205+0000 0'0 24:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:23:35.957 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:23:35.957 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:23:35.958 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:23:35.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-12T03:23:35.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-12T03:23:35.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-12T03:23:35.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-12T03:23:35.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-12T03:23:35.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-12T03:23:35.968 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-12T03:23:35.969 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-12T03:23:35.969 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-12T03:23:35.969 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-12T03:23:35.969 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-12T03:23:35.969 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-12T03:23:35.969 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:35.969 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-12T03:23:35.970 INFO:tasks.workunit.client.0.smithi180.stdout:writing initial objects 2020-03-12T03:23:36.013 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.013 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-12T03:23:36.052 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.052 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-12T03:23:36.091 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.092 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-12T03:23:36.130 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.130 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-12T03:23:36.168 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.169 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-12T03:23:36.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-12T03:23:36.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-12T03:23:36.355 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.355 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-12T03:23:36.431 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.431 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-12T03:23:36.515 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.515 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-12T03:23:36.555 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.555 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-12T03:23:36.593 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.594 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-12T03:23:36.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-12T03:23:36.674 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.674 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-12T03:23:36.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.719 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-12T03:23:36.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-12T03:23:36.797 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.797 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-12T03:23:36.835 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.836 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-12T03:23:36.915 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.916 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-12T03:23:36.955 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:36.955 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-12T03:23:37.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-12T03:23:37.133 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.133 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-12T03:23:37.216 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.217 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-12T03:23:37.255 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.255 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-12T03:23:37.294 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.294 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-12T03:23:37.333 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.333 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-12T03:23:37.372 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.372 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-12T03:23:37.410 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.410 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-12T03:23:37.450 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.450 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-12T03:23:37.502 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.503 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-12T03:23:37.583 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.584 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-12T03:23:37.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-12T03:23:37.700 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.703 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-12T03:23:37.784 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.784 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-12T03:23:37.825 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.825 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-12T03:23:37.882 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.882 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-12T03:23:37.927 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.927 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-12T03:23:37.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:37.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-12T03:23:38.043 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.043 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-12T03:23:38.083 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.084 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-12T03:23:38.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.122 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-12T03:23:38.160 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.161 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-12T03:23:38.235 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.235 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-12T03:23:38.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-12T03:23:38.400 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-12T03:23:38.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-12T03:23:38.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-12T03:23:38.802 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-12T03:23:38.842 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.842 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-12T03:23:38.881 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.881 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-12T03:23:38.919 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.919 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-12T03:23:38.959 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.959 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-12T03:23:38.997 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:38.997 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-12T03:23:39.036 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.036 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-12T03:23:39.161 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.161 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-12T03:23:39.245 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.245 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-12T03:23:39.321 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.321 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-12T03:23:39.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-12T03:23:39.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-12T03:23:39.439 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.439 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-12T03:23:39.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-12T03:23:39.542 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.542 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-12T03:23:39.581 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.581 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-12T03:23:39.619 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.620 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-12T03:23:39.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-12T03:23:39.737 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.738 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-12T03:23:39.778 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.778 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-12T03:23:39.866 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.867 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-12T03:23:39.988 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:39.989 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-12T03:23:40.036 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.036 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-12T03:23:40.084 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.085 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-12T03:23:40.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-12T03:23:40.161 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.161 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-12T03:23:40.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.201 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-12T03:23:40.239 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.240 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-12T03:23:40.309 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.310 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-12T03:23:40.348 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.348 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-12T03:23:40.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-12T03:23:40.514 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.514 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-12T03:23:40.553 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.554 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-12T03:23:40.631 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.631 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-12T03:23:40.686 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.687 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-12T03:23:40.726 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.726 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-12T03:23:40.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-12T03:23:40.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-12T03:23:40.842 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.843 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-12T03:23:40.882 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.882 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-12T03:23:40.920 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:40.920 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-12T03:23:41.028 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.028 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-12T03:23:41.140 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-12T03:23:41.215 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.215 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-12T03:23:41.299 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.299 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-12T03:23:41.374 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.374 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-12T03:23:41.415 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-12T03:23:41.478 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.478 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-12T03:23:41.518 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.518 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-12T03:23:41.591 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.591 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-12T03:23:41.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-12T03:23:41.670 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-12T03:23:41.670 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-12T03:23:41.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T03:23:41.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-12T03:23:41.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:23:41.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:23:41.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:23:41.713 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:23:41.713 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T03:23:41.713 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:23:41.714 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:23:41.714 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:23:41.714 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:23:41.714 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:23:41.766 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:23:41.766 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:23:41.767 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:23:41.767 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:23:41.767 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:23:42.106 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:23:42.107 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:23:42.107 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:23:42.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:23:42.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:42.108 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:23:42.235 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-12T03:23:42.235 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-12T03:23:42.236 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:42.236 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:23:42.365 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705675 2020-03-12T03:23:42.365 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705675' 2020-03-12T03:23:42.366 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:23:42.366 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:23:42.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542154 2020-03-12T03:23:42.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705675 2-60129542154' 2020-03-12T03:23:42.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:42.495 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-12T03:23:42.495 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:42.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:23:42.496 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-12T03:23:42.496 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:42.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-12T03:23:42.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-12T03:23:42.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:23:42.497 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836493 2020-03-12T03:23:42.838 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836493 2020-03-12T03:23:42.838 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:23:43.839 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:23:43.840 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:23:44.196 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-12T03:23:44.196 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:44.196 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705675 2020-03-12T03:23:44.196 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:44.197 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:23:44.197 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705675 2020-03-12T03:23:44.198 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:44.199 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705675 2020-03-12T03:23:44.199 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2020-03-12T03:23:44.199 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:23:44.199 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 38654705675 2020-03-12T03:23:44.541 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705675 -lt 38654705675 2020-03-12T03:23:44.541 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:23:44.542 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542154 2020-03-12T03:23:44.542 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:23:44.542 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:23:44.543 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542154 2020-03-12T03:23:44.543 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:23:44.544 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542154 2020-03-12T03:23:44.544 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2020-03-12T03:23:44.544 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:23:44.544 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 60129542154 2020-03-12T03:23:44.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542154 -lt 60129542154 2020-03-12T03:23:44.887 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:23:44.888 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:23:44.888 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:23:45.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:23:45.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:23:45.330 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:23:45.331 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:23:45.331 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:23:45.331 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:23:45.331 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:23:45.331 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:23:45.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-12T03:23:45.659 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:23:45.660 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:23:45.660 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:23:46.115 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-12T03:23:46.115 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-12T03:23:46.116 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-12T03:23:46.116 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-12T03:23:46.116 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-12T03:23:46.116 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-12T03:23:46.116 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-12T03:23:46.116 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-12T03:23:46.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-12T03:23:46.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-12T03:23:46.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:23:46.461 INFO:tasks.workunit.client.0.smithi180.stdout:blackholing osds 0 2 2020-03-12T03:23:46.783 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:23:46.816 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-12T03:23:27.426205+0000 27'100 27:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:23:46.816 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:23:46.817 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:23:46.819 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:23:46.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:23:46.819 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-12T03:23:46.819 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-12T03:23:46.820 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-12T03:23:46.820 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:23:46.820 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:46.821 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:46.821 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.0.asok 2020-03-12T03:23:46.821 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-12T03:23:46.821 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-12T03:23:46.879 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:23:46.879 INFO:tasks.workunit.client.0.smithi180.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-12T03:23:46.880 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:23:46.887 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:23:46.888 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-12T03:23:46.888 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-12T03:23:46.888 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-12T03:23:46.889 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:23:46.889 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:23:46.889 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:23:46.890 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.2.asok 2020-03-12T03:23:46.890 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-12T03:23:46.890 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-12T03:23:46.975 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:23:46.976 INFO:tasks.workunit.client.0.smithi180.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-12T03:23:46.976 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:23:46.982 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-12T03:23:46.982 INFO:tasks.workunit.client.0.smithi180.stdout:create new divergent objects 2020-03-12T03:23:46.983 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-12T03:23:46.983 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-12T03:23:46.984 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-12T03:23:46.984 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-12T03:23:46.984 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-12T03:23:46.984 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-12T03:23:46.984 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-12T03:23:46.985 INFO:tasks.workunit.client.0.smithi180.stdout:writing divergent objects 2020-03-12T03:23:46.986 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-12T03:23:46.986 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-12T03:23:46.986 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-12T03:23:46.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-12T03:23:46.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-12T03:23:46.987 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-12T03:23:46.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-12T03:23:46.987 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-12T03:23:46.987 INFO:tasks.workunit.client.0.smithi180.stdout:remove divergent objects 2020-03-12T03:23:46.988 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-12T03:23:46.989 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-12T03:23:46.989 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-12T03:23:46.989 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-12T03:23:46.989 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-12T03:23:46.989 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-12T03:23:46.990 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-12T03:23:56.990 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-12T03:23:56.992 INFO:tasks.workunit.client.0.smithi180.stderr:rados: no process found 2020-03-12T03:23:56.993 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-12T03:23:56.993 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:23:56.993 INFO:tasks.workunit.client.0.smithi180.stdout:killing all the osds 2020-03-12T03:23:57.323 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:23:57.324 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-12T03:23:27.426205+0000 27'106 27:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:23:57.324 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:23:57.324 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:23:57.325 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:23:57.333 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-12T03:23:57.334 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:23:57.334 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:23:57.334 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:23:57.335 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:23:57.335 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:23:57.440 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:23:57.440 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-12T03:23:57.440 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-12T03:23:57.933 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already down. 2020-03-12T03:23:57.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-12T03:23:57.942 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-12T03:23:58.433 INFO:tasks.workunit.client.0.smithi180.stderr:osd.1 is already down. 2020-03-12T03:23:58.441 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-12T03:23:58.442 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-12T03:23:58.940 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already down. 2020-03-12T03:23:58.949 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:23:58.949 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-12T03:23:59.449 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already out. 2020-03-12T03:23:59.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:23:59.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-12T03:23:59.966 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already out. 2020-03-12T03:23:59.974 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-12T03:23:59.974 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:23:59.974 INFO:tasks.workunit.client.0.smithi180.stdout:bringing up non_divergent 0 2 2020-03-12T03:24:00.299 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:24:00.299 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-12T03:23:27.426205+0000 27'106 27:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:24:00.299 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:24:00.299 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:24:00.301 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:24:00.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:24:00.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-12T03:24:00.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:24:00.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:24:00.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-12T03:24:00.312 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:24:00.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-12T03:24:00.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:24:00.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:24:00.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:24:00.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:24:00.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-12T03:24:00.313 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-12T03:24:00.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:24:00.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:24:00.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:24:00.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:24:00.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:24:00.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:24:00.314 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:24:00.315 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:24:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:24:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:24:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:24:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:24:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-12T03:24:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-12T03:24:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:24:00.317 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-12T03:24:00.317 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.0 2020-03-12T03:24:00.317 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-12T03:24:00.318 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:24:00.318 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:24:00.318 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:24:00.338 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:00.337+0000 7fb1716b7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:24:00.339 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:00.338+0000 7fb1716b7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:24:00.341 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:00.340+0000 7fb1716b7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:24:00.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-12T03:24:00.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:24:00.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:24:00.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:24:00.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:24:00.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:24:00.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:24:00.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:24:00.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:24:00.659 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:24:00.855 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:00.854+0000 7fb1716b7ec0 -1 Falling back to public interface 2020-03-12T03:24:01.000 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:24:01.717 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:01.716+0000 7fb1716b7ec0 -1 osd.0 27 log_to_monitors {default=true} 2020-03-12T03:24:02.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:24:02.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:24:02.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:24:02.002 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:24:02.003 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:24:02.003 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:24:02.358 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up out weight 0 up_from 33 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/29454,v1:127.0.0.1:6803/29454] [v2:127.0.0.1:6804/29454,v1:127.0.0.1:6805/29454] exists,up 59ecfd79-e15b-4216-a9fb-a337e29afe1e 2020-03-12T03:24:02.358 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.2 2020-03-12T03:24:02.359 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:24:02.359 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:24:02.359 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:24:02.359 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:24:02.359 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-12T03:24:02.359 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:24:02.359 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:24:02.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-12T03:24:02.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:24:02.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-12T03:24:02.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:24:02.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:24:02.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:24:02.360 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:24:02.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-12T03:24:02.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-12T03:24:02.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:24:02.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:24:02.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:24:02.361 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:24:02.362 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:24:02.362 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:24:02.362 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:24:02.362 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:24:02.362 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:24:02.362 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:24:02.362 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:24:02.363 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:24:02.363 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:24:02.363 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:24:02.363 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:24:02.363 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:24:02.364 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:24:02.364 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:24:02.364 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:24:02.364 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-12T03:24:02.364 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-12T03:24:02.364 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:24:02.364 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-12T03:24:02.365 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-12T03:24:02.365 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:24:02.365 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:24:02.365 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:24:02.384 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:02.382+0000 7f871a26dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:24:02.385 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:02.384+0000 7f871a26dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:24:02.386 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:02.385+0000 7f871a26dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:24:02.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-12T03:24:02.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:24:02.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:24:02.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:24:02.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:24:02.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:24:02.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:24:02.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:24:02.718 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:24:02.718 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:24:02.903 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:02.902+0000 7f871a26dec0 -1 Falling back to public interface 2020-03-12T03:24:03.062 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:24:03.761 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:24:03.760+0000 7f871a26dec0 -1 osd.2 27 log_to_monitors {default=true} 2020-03-12T03:24:04.064 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:24:04.064 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:24:04.064 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:24:04.064 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:24:04.065 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:24:04.065 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:24:04.420 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up out weight 0 up_from 34 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/29648,v1:127.0.0.1:6811/29648] [v2:127.0.0.1:6812/29648,v1:127.0.0.1:6813/29648] exists,up 3f99c107-f1bf-46d1-92a5-89dec9b23f29 2020-03-12T03:24:04.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:24:04.421 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:24:04.421 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:24:04.421 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:24:04.421 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-12T03:24:04.990 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already in. 2020-03-12T03:24:04.998 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:24:04.998 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-12T03:24:05.499 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already in. 2020-03-12T03:24:05.507 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T03:24:05.507 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-12T03:24:05.507 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:24:05.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:24:05.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:24:05.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:24:05.508 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T03:24:05.509 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:24:05.509 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:24:05.509 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:24:05.509 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:24:05.510 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:24:05.563 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:24:05.563 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:24:05.563 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:24:05.578 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:24:05.578 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:24:05.916 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:24:05.916 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:24:05.916 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:24:05.916 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:24:05.916 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:24:05.916 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:24:06.045 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-12T03:24:06.045 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771' 2020-03-12T03:24:06.045 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:24:06.045 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:24:06.136 INFO:tasks.workunit.client.0.smithi180.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-12T03:24:06.145 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-12T03:24:06.145 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771 1-' 2020-03-12T03:24:06.145 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:24:06.146 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:24:06.274 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-12T03:24:06.274 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920771 1- 2-146028888067' 2020-03-12T03:24:06.274 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:24:06.275 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-141733920771 2020-03-12T03:24:06.275 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:24:06.275 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:24:06.276 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-141733920771 2020-03-12T03:24:06.276 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:24:06.277 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-12T03:24:06.277 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2020-03-12T03:24:06.277 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:24:06.277 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 141733920771 2020-03-12T03:24:06.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 141733920771 2020-03-12T03:24:06.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:24:07.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:24:07.623 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:24:07.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920772 -lt 141733920771 2020-03-12T03:24:07.972 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:24:07.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-12T03:24:07.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:24:07.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:24:07.975 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-12T03:24:07.975 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:24:07.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-12T03:24:07.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-12T03:24:07.976 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:24:07.976 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 2020-03-12T03:24:08.325 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705678 -lt 2020-03-12T03:24:08.325 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705678: unary operator expected 2020-03-12T03:24:08.326 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:24:08.326 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-146028888067 2020-03-12T03:24:08.326 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:24:08.327 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:24:08.327 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-146028888067 2020-03-12T03:24:08.327 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:24:08.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-12T03:24:08.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2020-03-12T03:24:08.328 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:24:08.328 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 146028888067 2020-03-12T03:24:08.671 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888067 -lt 146028888067 2020-03-12T03:24:08.671 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:24:08.672 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:08.672 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:09.117 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:24:09.118 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:09.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:09.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:09.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:09.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:09.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:09.118 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:09.449 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:09.449 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:09.450 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:09.450 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:09.895 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:09.895 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-12T03:24:09.896 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-12T03:24:09.896 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-12T03:24:09.896 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:09.896 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-12T03:24:09.996 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:09.997 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:09.997 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:09.997 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:09.997 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:09.997 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:09.997 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:09.998 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:10.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:10.329 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:10.329 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:10.329 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:10.770 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:10.770 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:10.770 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:10.770 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:10.771 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:10.771 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:10.771 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:10.771 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:10.771 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:11.250 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:11.250 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:11.251 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-12T03:24:11.251 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:11.251 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-12T03:24:11.451 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:11.452 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:11.452 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:11.452 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:11.452 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:11.452 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:11.452 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:11.453 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:11.780 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:11.780 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:11.781 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:11.781 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:12.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:12.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:12.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:12.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:12.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:12.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:12.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:12.225 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:12.225 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:12.689 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:12.689 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:12.690 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-12T03:24:12.690 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:12.690 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-12T03:24:13.090 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:13.091 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:13.091 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:13.091 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:13.091 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:13.091 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:13.091 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:13.092 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:13.425 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:13.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:13.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:13.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:13.871 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:13.871 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:13.871 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:13.871 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:13.871 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:13.872 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:13.872 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:13.872 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:13.872 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:14.310 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:14.310 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:14.310 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-12T03:24:14.310 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:14.311 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-12T03:24:15.112 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:15.112 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:15.112 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:15.113 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:15.113 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:15.113 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:15.113 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:15.114 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:15.448 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:15.448 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:15.449 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:15.449 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:15.893 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:15.893 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:15.894 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:15.894 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:15.894 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:15.894 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:15.895 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:15.895 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:15.895 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:16.332 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:16.332 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:16.332 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-12T03:24:16.332 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:16.332 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-12T03:24:17.932 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:17.933 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:17.933 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:17.934 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:17.934 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:17.934 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:17.934 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:17.934 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:18.274 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:18.274 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:18.275 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:18.275 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:18.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:18.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:18.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:18.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:18.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:18.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:18.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:18.715 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:18.715 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:19.160 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:19.160 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:19.160 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-12T03:24:19.160 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:19.160 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-12T03:24:22.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:22.361 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:22.362 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:22.362 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:22.362 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:22.363 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:22.363 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:22.363 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:22.700 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:22.700 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:22.701 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:22.701 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:23.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:23.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:23.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:23.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:23.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:23.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:23.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:23.142 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:23.142 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:23.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:23.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:23.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-12T03:24:23.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:23.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-12T03:24:29.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:29.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:29.988 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:29.988 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:29.988 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:29.988 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:29.988 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:29.989 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:30.327 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:30.327 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:30.328 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:30.328 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:30.772 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:30.772 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:30.772 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:30.772 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:30.773 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:30.773 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:30.773 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:30.773 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:30.773 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:31.264 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:31.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:31.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-12T03:24:31.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:31.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-12T03:24:38.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:38.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:38.566 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:38.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:38.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:38.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:38.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:38.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:38.908 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:38.908 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:38.909 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:38.909 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:39.352 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:39.352 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:39.352 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:39.353 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:39.353 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:39.353 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:39.353 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:39.353 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:39.353 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:39.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:39.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:39.803 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-12T03:24:39.804 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-12T03:24:40.246 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:24:40.246 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_fingerprint": "00b67cd6-f96e-4da1-bded-61a36e835a93", 2020-03-12T03:24:40.247 INFO:tasks.workunit.client.0.smithi180.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-12T03:24:40.247 INFO:tasks.workunit.client.0.smithi180.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-12T03:24:40.247 INFO:tasks.workunit.client.0.smithi180.stdout: "timestamp": "2020-03-12T03:24:40.244358+0000", 2020-03-12T03:24:40.247 INFO:tasks.workunit.client.0.smithi180.stdout: "tag": "", 2020-03-12T03:24:40.247 INFO:tasks.workunit.client.0.smithi180.stdout: "health": { 2020-03-12T03:24:40.247 INFO:tasks.workunit.client.0.smithi180.stdout: "status": "HEALTH_WARN", 2020-03-12T03:24:40.247 INFO:tasks.workunit.client.0.smithi180.stdout: "checks": { 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: "OSDMAP_FLAGS": { 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 17 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [], 2020-03-12T03:24:40.248 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:24:40.249 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.249 INFO:tasks.workunit.client.0.smithi180.stdout: "OSD_DOWN": { 2020-03-12T03:24:40.249 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:24:40.249 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:24:40.249 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 osds down", 2020-03-12T03:24:40.249 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:24:40.249 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.250 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:24:40.250 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.250 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "osd.1 (root=default,host=smithi180) is down" 2020-03-12T03:24:40.250 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.250 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.250 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:24:40.250 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.251 INFO:tasks.workunit.client.0.smithi180.stdout: "PG_DEGRADED": { 2020-03-12T03:24:40.251 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:24:40.251 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:24:40.251 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-12T03:24:40.251 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:24:40.251 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.252 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:24:40.252 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.252 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-12T03:24:40.252 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.252 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.252 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:24:40.252 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:24:40.253 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.254 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "application not enabled on pool 'test'" 2020-03-12T03:24:40.254 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.254 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.254 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-12T03:24:40.254 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.254 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.254 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:24:40.255 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.255 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.255 INFO:tasks.workunit.client.0.smithi180.stdout: "mutes": [] 2020-03-12T03:24:40.255 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.255 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_first_committed": 1, 2020-03-12T03:24:40.255 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_last_committed": 1, 2020-03-12T03:24:40.255 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap": { 2020-03-12T03:24:40.256 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:24:40.256 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:24:40.256 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:23:04.226979Z", 2020-03-12T03:24:40.256 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:23:04.226979Z", 2020-03-12T03:24:40.256 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release": 15, 2020-03-12T03:24:40.256 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release_name": "octopus", 2020-03-12T03:24:40.256 INFO:tasks.workunit.client.0.smithi180.stdout: "features": { 2020-03-12T03:24:40.257 INFO:tasks.workunit.client.0.smithi180.stdout: "persistent": [ 2020-03-12T03:24:40.257 INFO:tasks.workunit.client.0.smithi180.stdout: "kraken", 2020-03-12T03:24:40.257 INFO:tasks.workunit.client.0.smithi180.stdout: "luminous", 2020-03-12T03:24:40.257 INFO:tasks.workunit.client.0.smithi180.stdout: "mimic", 2020-03-12T03:24:40.257 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap-prune", 2020-03-12T03:24:40.257 INFO:tasks.workunit.client.0.smithi180.stdout: "nautilus", 2020-03-12T03:24:40.257 INFO:tasks.workunit.client.0.smithi180.stdout: "octopus" 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: "optional": [] 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: "mons": [ 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: "rank": 0, 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "a", 2020-03-12T03:24:40.258 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:24:40.259 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.259 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.259 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.259 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115", 2020-03-12T03:24:40.259 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 0 2020-03-12T03:24:40.259 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.259 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.260 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.260 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115/0", 2020-03-12T03:24:40.260 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-12T03:24:40.260 INFO:tasks.workunit.client.0.smithi180.stdout: "priority": 0, 2020-03-12T03:24:40.260 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 0 2020-03-12T03:24:40.261 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.261 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.261 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.261 INFO:tasks.workunit.client.0.smithi180.stdout: "quorum": [ 2020-03-12T03:24:40.262 INFO:tasks.workunit.client.0.smithi180.stdout: 0 2020-03-12T03:24:40.262 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.262 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap": { 2020-03-12T03:24:40.263 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 39, 2020-03-12T03:24:40.263 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:24:40.263 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:23:04.948358+0000", 2020-03-12T03:24:40.263 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:24:05.603910+0000", 2020-03-12T03:24:40.263 INFO:tasks.workunit.client.0.smithi180.stdout: "last_up_change": "2020-03-12T03:24:03.834098+0000", 2020-03-12T03:24:40.263 INFO:tasks.workunit.client.0.smithi180.stdout: "last_in_change": "2020-03-12T03:24:05.314095+0000", 2020-03-12T03:24:40.263 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-12T03:24:40.264 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_num": 5800384, 2020-03-12T03:24:40.264 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_set": [ 2020-03-12T03:24:40.264 INFO:tasks.workunit.client.0.smithi180.stdout: "nodown", 2020-03-12T03:24:40.264 INFO:tasks.workunit.client.0.smithi180.stdout: "noin", 2020-03-12T03:24:40.264 INFO:tasks.workunit.client.0.smithi180.stdout: "noout", 2020-03-12T03:24:40.264 INFO:tasks.workunit.client.0.smithi180.stdout: "pglog_hardlimit", 2020-03-12T03:24:40.265 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snapdirs", 2020-03-12T03:24:40.265 INFO:tasks.workunit.client.0.smithi180.stdout: "recovery_deletes", 2020-03-12T03:24:40.265 INFO:tasks.workunit.client.0.smithi180.stdout: "sortbitwise" 2020-03-12T03:24:40.265 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.265 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_version": 7, 2020-03-12T03:24:40.265 INFO:tasks.workunit.client.0.smithi180.stdout: "full_ratio": 0.99000000953674316, 2020-03-12T03:24:40.265 INFO:tasks.workunit.client.0.smithi180.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-12T03:24:40.266 INFO:tasks.workunit.client.0.smithi180.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-12T03:24:40.266 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_snapshot": "", 2020-03-12T03:24:40.266 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_max": 1, 2020-03-12T03:24:40.266 INFO:tasks.workunit.client.0.smithi180.stdout: "max_osd": 3, 2020-03-12T03:24:40.266 INFO:tasks.workunit.client.0.smithi180.stdout: "require_min_compat_client": "jewel", 2020-03-12T03:24:40.266 INFO:tasks.workunit.client.0.smithi180.stdout: "min_compat_client": "jewel", 2020-03-12T03:24:40.266 INFO:tasks.workunit.client.0.smithi180.stdout: "require_osd_release": "octopus", 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: "pools": [ 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: "pool": 1, 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_name": "test", 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: "create_time": "2020-03-12T03:23:26.536291+0000", 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": 1, 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_names": "hashpspool", 2020-03-12T03:24:40.267 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:24:40.268 INFO:tasks.workunit.client.0.smithi180.stdout: "size": 3, 2020-03-12T03:24:40.268 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 2, 2020-03-12T03:24:40.268 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_rule": 0, 2020-03-12T03:24:40.268 INFO:tasks.workunit.client.0.smithi180.stdout: "object_hash": 2, 2020-03-12T03:24:40.268 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_autoscale_mode": "off", 2020-03-12T03:24:40.268 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num": 1, 2020-03-12T03:24:40.268 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num": 1, 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num_target": 1, 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_target": 1, 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_pending": 1, 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pg_merge_meta": { 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "source_pgid": "0.0", 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "ready_epoch": 0, 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_started": 0, 2020-03-12T03:24:40.269 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_clean": 0, 2020-03-12T03:24:40.270 INFO:tasks.workunit.client.0.smithi180.stdout: "source_version": "0'0", 2020-03-12T03:24:40.270 INFO:tasks.workunit.client.0.smithi180.stdout: "target_version": "0'0" 2020-03-12T03:24:40.270 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.270 INFO:tasks.workunit.client.0.smithi180.stdout: "last_change": "27", 2020-03-12T03:24:40.270 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend": "0", 2020-03-12T03:24:40.270 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-12T03:24:40.270 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_preluminous": "0", 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "auid": 0, 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_mode": "selfmanaged", 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_seq": 0, 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_epoch": 0, 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_snaps": [], 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps": "[]", 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_bytes": 0, 2020-03-12T03:24:40.271 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_objects": 0, 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "tiers": [], 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "tier_of": -1, 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "read_tier": -1, 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "write_tier": -1, 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_mode": "none", 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_bytes": 0, 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_objects": 0, 2020-03-12T03:24:40.272 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-12T03:24:40.273 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-12T03:24:40.273 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-12T03:24:40.273 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_flush_age": 0, 2020-03-12T03:24:40.273 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_evict_age": 0, 2020-03-12T03:24:40.273 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profile": "", 2020-03-12T03:24:40.273 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_params": { 2020-03-12T03:24:40.273 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "none" 2020-03-12T03:24:40.274 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.274 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_period": 0, 2020-03-12T03:24:40.274 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_count": 0, 2020-03-12T03:24:40.274 INFO:tasks.workunit.client.0.smithi180.stdout: "use_gmt_hitset": true, 2020-03-12T03:24:40.274 INFO:tasks.workunit.client.0.smithi180.stdout: "min_read_recency_for_promote": 0, 2020-03-12T03:24:40.274 INFO:tasks.workunit.client.0.smithi180.stdout: "min_write_recency_for_promote": 0, 2020-03-12T03:24:40.274 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_grade_decay_rate": 0, 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_search_last_n": 0, 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: "grade_table": [], 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: "stripe_width": 0, 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: "expected_num_objects": 0, 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: "fast_read": false, 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: "options": {}, 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: "application_metadata": {} 2020-03-12T03:24:40.275 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: "osds": [ 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "59ecfd79-e15b-4216-a9fb-a337e29afe1e", 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:24:40.276 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:24:40.277 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:24:40.277 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 5, 2020-03-12T03:24:40.277 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 27, 2020-03-12T03:24:40.277 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 33, 2020-03-12T03:24:40.277 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 38, 2020-03-12T03:24:40.277 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 28, 2020-03-12T03:24:40.277 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:24:40.278 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:24:40.278 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.278 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.278 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.279 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6802", 2020-03-12T03:24:40.279 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.279 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.280 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.280 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.280 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6803", 2020-03-12T03:24:40.280 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.280 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.280 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.281 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.281 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:24:40.281 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.281 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.281 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.281 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6804", 2020-03-12T03:24:40.282 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.282 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.282 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.282 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.282 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6805", 2020-03-12T03:24:40.282 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.282 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6808", 2020-03-12T03:24:40.283 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.284 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.284 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.284 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.284 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6809", 2020-03-12T03:24:40.284 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.284 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.284 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.285 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.285 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:24:40.285 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.285 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.285 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.285 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6806", 2020-03-12T03:24:40.285 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6807", 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.286 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.287 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6803/29454", 2020-03-12T03:24:40.287 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6805/29454", 2020-03-12T03:24:40.287 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6809/29454", 2020-03-12T03:24:40.287 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6807/29454", 2020-03-12T03:24:40.287 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:24:40.287 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:24:40.287 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:24:40.288 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.288 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.288 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.289 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:24:40.289 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "eb5be901-51a3-4f54-bb09-dd4860cf0f9c", 2020-03-12T03:24:40.289 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 0, 2020-03-12T03:24:40.289 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:24:40.289 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:24:40.289 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:24:40.289 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 0, 2020-03-12T03:24:40.290 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 0, 2020-03-12T03:24:40.290 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 9, 2020-03-12T03:24:40.290 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 21, 2020-03-12T03:24:40.290 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 29, 2020-03-12T03:24:40.290 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:24:40.290 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:24:40.290 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.291 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.291 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.291 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:24:40.291 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.292 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.292 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.292 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.292 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:24:40.292 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.292 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.292 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.293 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.293 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:24:40.293 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.293 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.293 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.293 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:24:40.294 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.294 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.294 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.294 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.294 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:24:40.294 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.295 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.295 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.295 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.295 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:24:40.295 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.295 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.295 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.296 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:24:40.296 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.296 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.296 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.296 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.296 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:24:40.297 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.297 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.297 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.297 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.297 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:24:40.297 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.297 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.298 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.298 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:24:40.298 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.298 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.298 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.298 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.298 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:24:40.299 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:24:40.299 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.299 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.299 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.299 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/25074", 2020-03-12T03:24:40.299 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/25074", 2020-03-12T03:24:40.299 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25074", 2020-03-12T03:24:40.300 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25074", 2020-03-12T03:24:40.300 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:24:40.300 INFO:tasks.workunit.client.0.smithi180.stdout: "exists" 2020-03-12T03:24:40.300 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.300 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.300 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.300 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:24:40.301 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "3f99c107-f1bf-46d1-92a5-89dec9b23f29", 2020-03-12T03:24:40.301 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:24:40.301 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:24:40.301 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:24:40.301 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:24:40.301 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 14, 2020-03-12T03:24:40.302 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 27, 2020-03-12T03:24:40.302 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 34, 2020-03-12T03:24:40.302 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 0, 2020-03-12T03:24:40.302 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 30, 2020-03-12T03:24:40.302 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:24:40.302 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:24:40.302 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.303 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.303 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.306 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:24:40.306 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.306 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.306 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.308 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.308 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:24:40.308 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.308 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.309 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.309 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.309 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:24:40.309 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.309 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.309 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.310 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:24:40.310 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.310 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.310 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.310 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.310 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:24:40.310 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.311 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.311 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.311 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.311 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:24:40.311 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.312 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.312 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.312 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:24:40.312 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.312 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.312 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.313 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.313 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:24:40.313 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.313 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.313 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.313 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.313 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:24:40.314 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:24:40.314 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.314 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:24:40.314 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:24:40.314 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.314 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.315 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.315 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:24:40.315 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:24:40.315 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:24:40.315 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.315 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.315 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.316 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/29648", 2020-03-12T03:24:40.316 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/29648", 2020-03-12T03:24:40.316 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/29648", 2020-03-12T03:24:40.316 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/29648", 2020-03-12T03:24:40.316 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:24:40.316 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:24:40.316 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:24:40.317 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.317 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.317 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.317 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_xinfo": [ 2020-03-12T03:24:40.317 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.317 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:24:40.318 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:23:57.738718+0000", 2020-03-12T03:24:40.318 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:24:40.318 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:24:40.318 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:24:40.318 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:24:40.318 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:23:11.947656+0000", 2020-03-12T03:24:40.318 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:24:40.319 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.319 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.319 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:24:40.319 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:23:58.239159+0000", 2020-03-12T03:24:40.319 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:24:40.319 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:24:40.319 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:24:40.320 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:24:40.320 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:23:18.186236+0000", 2020-03-12T03:24:40.320 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:24:40.320 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.320 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.320 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:24:40.321 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:23:58.737011+0000", 2020-03-12T03:24:40.321 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:24:40.321 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:24:40.321 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:24:40.321 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:24:40.322 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:23:24.519963+0000", 2020-03-12T03:24:40.322 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:24:40.323 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.323 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.323 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap": [], 2020-03-12T03:24:40.323 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap_items": [], 2020-03-12T03:24:40.323 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_temp": [], 2020-03-12T03:24:40.323 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_temp": [], 2020-03-12T03:24:40.323 INFO:tasks.workunit.client.0.smithi180.stdout: "blacklist": {}, 2020-03-12T03:24:40.324 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profiles": { 2020-03-12T03:24:40.324 INFO:tasks.workunit.client.0.smithi180.stdout: "default": { 2020-03-12T03:24:40.324 INFO:tasks.workunit.client.0.smithi180.stdout: "k": "2", 2020-03-12T03:24:40.324 INFO:tasks.workunit.client.0.smithi180.stdout: "m": "2", 2020-03-12T03:24:40.324 INFO:tasks.workunit.client.0.smithi180.stdout: "plugin": "jerasure", 2020-03-12T03:24:40.324 INFO:tasks.workunit.client.0.smithi180.stdout: "technique": "reed_sol_van" 2020-03-12T03:24:40.324 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.325 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.325 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps_queue": [], 2020-03-12T03:24:40.325 INFO:tasks.workunit.client.0.smithi180.stdout: "new_removed_snaps": [], 2020-03-12T03:24:40.325 INFO:tasks.workunit.client.0.smithi180.stdout: "new_purged_snaps": [], 2020-03-12T03:24:40.325 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_node_flags": {}, 2020-03-12T03:24:40.325 INFO:tasks.workunit.client.0.smithi180.stdout: "device_class_flags": {} 2020-03-12T03:24:40.325 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.326 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_metadata": [ 2020-03-12T03:24:40.326 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.326 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:24:40.326 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:24:40.326 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6804/29454,v1:127.0.0.1:6805/29454]", 2020-03-12T03:24:40.326 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:24:40.326 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:24:40.327 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:24:40.327 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:24:40.327 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:24:40.327 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:24:40.327 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:24:40.327 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:24:40.327 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:24:40.328 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-12T03:24:40.328 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:24:40.328 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:24:40.328 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:24:40.328 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:24:40.328 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:24:40.328 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:24:40.329 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:24:40.329 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:24:40.329 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:24:40.329 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:24:40.329 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:24:40.329 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:24:40.329 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6802/29454,v1:127.0.0.1:6803/29454]", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/29454,v1:127.0.0.1:6809/29454]", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/29454,v1:127.0.0.1:6807/29454]", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:24:40.330 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:24:40.331 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:24:40.331 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:24:40.331 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:24:40.331 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:24:40.331 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:24:40.331 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:24:40.331 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:24:40.332 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/0", 2020-03-12T03:24:40.332 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:24:40.332 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:24:40.332 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.332 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.332 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:24:40.332 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/25074,v1:127.0.0.1:6813/25074]", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:24:40.333 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:24:40.334 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:24:40.334 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:24:40.334 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-12T03:24:40.334 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:24:40.334 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:24:40.334 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:24:40.334 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:24:40.335 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:24:40.335 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:24:40.335 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:24:40.335 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:24:40.335 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:24:40.335 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:24:40.335 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:24:40.336 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:24:40.336 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:24:40.336 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:24:40.336 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:24:40.336 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/25074,v1:127.0.0.1:6811/25074]", 2020-03-12T03:24:40.336 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:24:40.336 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25074,v1:127.0.0.1:6817/25074]", 2020-03-12T03:24:40.337 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25074,v1:127.0.0.1:6815/25074]", 2020-03-12T03:24:40.337 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:24:40.337 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:24:40.337 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:24:40.337 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:24:40.337 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:24:40.337 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:24:40.338 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:24:40.338 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:24:40.338 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:24:40.338 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/1", 2020-03-12T03:24:40.338 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:24:40.338 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:24:40.338 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.339 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.339 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:24:40.340 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:24:40.341 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/29648,v1:127.0.0.1:6813/29648]", 2020-03-12T03:24:40.341 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:24:40.341 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:24:40.341 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:24:40.341 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:24:40.342 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:24:40.342 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:24:40.342 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:24:40.342 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:24:40.342 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:24:40.342 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-12T03:24:40.343 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:24:40.343 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:24:40.343 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:24:40.343 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:24:40.343 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:24:40.344 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:24:40.344 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:24:40.344 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:24:40.344 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:24:40.344 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:24:40.345 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:24:40.345 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:24:40.345 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:24:40.345 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:24:40.345 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:24:40.346 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/29648,v1:127.0.0.1:6811/29648]", 2020-03-12T03:24:40.346 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:24:40.346 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/29648,v1:127.0.0.1:6817/29648]", 2020-03-12T03:24:40.346 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/29648,v1:127.0.0.1:6815/29648]", 2020-03-12T03:24:40.346 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:24:40.347 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:24:40.347 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:24:40.347 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:24:40.347 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:24:40.347 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:24:40.347 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:24:40.348 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:24:40.348 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:24:40.348 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/2", 2020-03-12T03:24:40.348 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:24:40.349 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:24:40.349 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.349 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.349 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_first_committed": 1, 2020-03-12T03:24:40.350 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_last_committed": 39, 2020-03-12T03:24:40.350 INFO:tasks.workunit.client.0.smithi180.stdout: "crushmap": { 2020-03-12T03:24:40.350 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": [ 2020-03-12T03:24:40.350 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.351 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:24:40.351 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.0", 2020-03-12T03:24:40.351 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:24:40.351 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.351 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.352 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:24:40.352 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.1", 2020-03-12T03:24:40.352 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:24:40.352 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.352 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.353 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:24:40.353 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.2", 2020-03-12T03:24:40.353 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:24:40.353 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.353 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.353 INFO:tasks.workunit.client.0.smithi180.stdout: "types": [ 2020-03-12T03:24:40.354 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.354 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 0, 2020-03-12T03:24:40.354 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd" 2020-03-12T03:24:40.354 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.354 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.354 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:24:40.355 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "host" 2020-03-12T03:24:40.355 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.355 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.355 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 2, 2020-03-12T03:24:40.355 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "chassis" 2020-03-12T03:24:40.356 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.356 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.356 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 3, 2020-03-12T03:24:40.356 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "rack" 2020-03-12T03:24:40.356 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.356 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.356 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 4, 2020-03-12T03:24:40.357 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "row" 2020-03-12T03:24:40.357 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.357 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.357 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 5, 2020-03-12T03:24:40.357 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pdu" 2020-03-12T03:24:40.357 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.358 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.358 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 6, 2020-03-12T03:24:40.358 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pod" 2020-03-12T03:24:40.358 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.358 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.358 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 7, 2020-03-12T03:24:40.358 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "room" 2020-03-12T03:24:40.359 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.359 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.359 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 8, 2020-03-12T03:24:40.359 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "datacenter" 2020-03-12T03:24:40.360 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.360 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.360 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 9, 2020-03-12T03:24:40.360 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "zone" 2020-03-12T03:24:40.360 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.361 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.361 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 10, 2020-03-12T03:24:40.361 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "region" 2020-03-12T03:24:40.361 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.362 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.362 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:24:40.362 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "root" 2020-03-12T03:24:40.362 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.362 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.362 INFO:tasks.workunit.client.0.smithi180.stdout: "buckets": [ 2020-03-12T03:24:40.362 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.363 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -1, 2020-03-12T03:24:40.363 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default", 2020-03-12T03:24:40.363 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:24:40.363 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:24:40.363 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:24:40.363 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:24:40.363 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:24:40.364 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:24:40.364 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.364 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:24:40.364 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:24:40.364 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:24:40.365 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.365 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.365 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.365 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.365 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -2, 2020-03-12T03:24:40.365 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default~hdd", 2020-03-12T03:24:40.366 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:24:40.366 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:24:40.366 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:24:40.366 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:24:40.366 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:24:40.366 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:24:40.367 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.367 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:24:40.367 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:24:40.367 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:24:40.367 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.367 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.367 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.368 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.368 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:24:40.368 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180", 2020-03-12T03:24:40.368 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:24:40.368 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:24:40.368 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:24:40.368 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:24:40.369 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:24:40.369 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:24:40.369 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.370 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:24:40.370 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:24:40.370 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:24:40.370 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.370 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.371 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:24:40.371 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:24:40.371 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:24:40.371 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.371 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.371 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:24:40.371 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:24:40.372 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:24:40.372 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.373 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.373 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.373 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.373 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:24:40.373 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180~hdd", 2020-03-12T03:24:40.374 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:24:40.374 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:24:40.374 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:24:40.374 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:24:40.374 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:24:40.374 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:24:40.375 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.375 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:24:40.375 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:24:40.375 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:24:40.375 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.375 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.376 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:24:40.376 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:24:40.376 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:24:40.376 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.376 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.376 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:24:40.377 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:24:40.377 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:24:40.377 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.377 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.377 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.377 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.378 INFO:tasks.workunit.client.0.smithi180.stdout: "rules": [ 2020-03-12T03:24:40.378 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.378 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_id": 0, 2020-03-12T03:24:40.378 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_name": "replicated_rule", 2020-03-12T03:24:40.378 INFO:tasks.workunit.client.0.smithi180.stdout: "ruleset": 0, 2020-03-12T03:24:40.378 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:24:40.378 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 1, 2020-03-12T03:24:40.379 INFO:tasks.workunit.client.0.smithi180.stdout: "max_size": 10, 2020-03-12T03:24:40.379 INFO:tasks.workunit.client.0.smithi180.stdout: "steps": [ 2020-03-12T03:24:40.379 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.379 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "take", 2020-03-12T03:24:40.379 INFO:tasks.workunit.client.0.smithi180.stdout: "item": -1, 2020-03-12T03:24:40.380 INFO:tasks.workunit.client.0.smithi180.stdout: "item_name": "default" 2020-03-12T03:24:40.380 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.380 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.380 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "choose_firstn", 2020-03-12T03:24:40.380 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 0, 2020-03-12T03:24:40.380 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "osd" 2020-03-12T03:24:40.381 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.381 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.381 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "emit" 2020-03-12T03:24:40.381 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.381 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:24:40.381 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.382 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.382 INFO:tasks.workunit.client.0.smithi180.stdout: "tunables": { 2020-03-12T03:24:40.382 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_tries": 0, 2020-03-12T03:24:40.382 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_fallback_tries": 0, 2020-03-12T03:24:40.382 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_total_tries": 50, 2020-03-12T03:24:40.382 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_descend_once": 1, 2020-03-12T03:24:40.383 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_vary_r": 1, 2020-03-12T03:24:40.383 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_stable": 1, 2020-03-12T03:24:40.383 INFO:tasks.workunit.client.0.smithi180.stdout: "straw_calc_version": 1, 2020-03-12T03:24:40.383 INFO:tasks.workunit.client.0.smithi180.stdout: "allowed_bucket_algs": 54, 2020-03-12T03:24:40.383 INFO:tasks.workunit.client.0.smithi180.stdout: "profile": "jewel", 2020-03-12T03:24:40.383 INFO:tasks.workunit.client.0.smithi180.stdout: "optimal_tunables": 1, 2020-03-12T03:24:40.384 INFO:tasks.workunit.client.0.smithi180.stdout: "legacy_tunables": 0, 2020-03-12T03:24:40.384 INFO:tasks.workunit.client.0.smithi180.stdout: "minimum_required_version": "jewel", 2020-03-12T03:24:40.384 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables": 1, 2020-03-12T03:24:40.384 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables2": 1, 2020-03-12T03:24:40.384 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v2_rules": 0, 2020-03-12T03:24:40.384 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables3": 1, 2020-03-12T03:24:40.385 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v3_rules": 0, 2020-03-12T03:24:40.385 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v4_buckets": 1, 2020-03-12T03:24:40.385 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables5": 1, 2020-03-12T03:24:40.385 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v5_rules": 0 2020-03-12T03:24:40.385 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.385 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_args": {} 2020-03-12T03:24:40.385 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.386 INFO:tasks.workunit.client.0.smithi180.stdout: "fsmap": { 2020-03-12T03:24:40.386 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:24:40.386 INFO:tasks.workunit.client.0.smithi180.stdout: "default_fscid": -1, 2020-03-12T03:24:40.386 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": { 2020-03-12T03:24:40.386 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": {}, 2020-03-12T03:24:40.387 INFO:tasks.workunit.client.0.smithi180.stdout: "ro_compat": {}, 2020-03-12T03:24:40.387 INFO:tasks.workunit.client.0.smithi180.stdout: "incompat": { 2020-03-12T03:24:40.387 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_1": "base v0.20", 2020-03-12T03:24:40.387 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_2": "client writeable ranges", 2020-03-12T03:24:40.388 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_3": "default file layouts on dirs", 2020-03-12T03:24:40.388 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_4": "dir inode in separate object", 2020-03-12T03:24:40.388 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_5": "mds uses versioned encoding", 2020-03-12T03:24:40.388 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-12T03:24:40.388 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_8": "no anchor table", 2020-03-12T03:24:40.388 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_9": "file layout v2", 2020-03-12T03:24:40.388 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_10": "snaprealm v2" 2020-03-12T03:24:40.389 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.389 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.389 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_flags": { 2020-03-12T03:24:40.389 INFO:tasks.workunit.client.0.smithi180.stdout: "enable_multiple": false, 2020-03-12T03:24:40.389 INFO:tasks.workunit.client.0.smithi180.stdout: "ever_enabled_multiple": false 2020-03-12T03:24:40.389 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.390 INFO:tasks.workunit.client.0.smithi180.stdout: "standbys": [], 2020-03-12T03:24:40.390 INFO:tasks.workunit.client.0.smithi180.stdout: "filesystems": [] 2020-03-12T03:24:40.390 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.390 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_first_committed": 1, 2020-03-12T03:24:40.390 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_last_committed": 1, 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: "auth": { 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 5, 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: "num_secrets": 3 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 1, 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_active": 1, 2020-03-12T03:24:40.391 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_unknown": 0, 2020-03-12T03:24:40.392 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osd": 3, 2020-03-12T03:24:40.392 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_sum": { 2020-03-12T03:24:40.392 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:24:40.392 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 48300, 2020-03-12T03:24:40.392 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:24:40.392 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:24:40.392 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:24:40.393 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:24:40.394 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:24:40.394 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 100, 2020-03-12T03:24:40.394 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 100, 2020-03-12T03:24:40.394 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:24:40.394 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:24:40.394 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:24:40.394 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:24:40.395 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:24:40.396 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:24:40.397 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:24:40.397 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:24:40.397 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:24:40.397 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:24:40.397 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:24:40.397 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.398 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:24:40.398 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:24:40.398 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:24:40.398 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:24:40.398 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 0, 2020-03-12T03:24:40.399 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 0, 2020-03-12T03:24:40.399 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:24:40.399 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:24:40.399 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:24:40.399 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:24:40.399 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:24:40.400 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.400 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 16, 2020-03-12T03:24:40.402 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 16, 2020-03-12T03:24:40.402 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:24:40.402 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:24:40.402 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 0 2020-03-12T03:24:40.403 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.403 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum": { 2020-03-12T03:24:40.403 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:24:40.403 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:24:40.403 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:24:40.404 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:24:40.404 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:24:40.404 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:24:40.404 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:24:40.404 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3178944, 2020-03-12T03:24:40.404 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 32896, 2020-03-12T03:24:40.404 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 12, 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145715, 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393856, 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867308544, 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33685504, 2020-03-12T03:24:40.405 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 402724, 2020-03-12T03:24:40.406 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:24:40.406 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:24:40.406 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:24:40.406 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 12914, 2020-03-12T03:24:40.406 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221212558 2020-03-12T03:24:40.406 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.406 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:24:40.407 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:24:40.407 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:24:40.407 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 0, 2020-03-12T03:24:40.407 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:24:40.407 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:24:40.407 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:24:40.407 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:24:40.408 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:24:40.409 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.409 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum_by_class": { 2020-03-12T03:24:40.409 INFO:tasks.workunit.client.0.smithi180.stdout: "hdd": { 2020-03-12T03:24:40.409 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:24:40.409 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:24:40.409 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:24:40.410 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:24:40.410 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:24:40.410 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:24:40.410 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:24:40.411 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3178944, 2020-03-12T03:24:40.411 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 32896, 2020-03-12T03:24:40.411 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 12, 2020-03-12T03:24:40.411 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145715, 2020-03-12T03:24:40.411 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393856, 2020-03-12T03:24:40.412 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:24:40.412 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:24:40.412 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867308544, 2020-03-12T03:24:40.412 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:24:40.413 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33685504, 2020-03-12T03:24:40.413 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 402724, 2020-03-12T03:24:40.413 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:24:40.413 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 12914, 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221212558 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:24:40.414 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 0, 2020-03-12T03:24:40.415 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:24:40.415 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:24:40.415 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:24:40.415 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.415 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:24:40.415 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:24:40.415 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.416 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_stats": [ 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: "poolid": 1, 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 1, 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 48300, 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:24:40.417 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:24:40.418 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:24:40.418 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:24:40.418 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:24:40.418 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:24:40.418 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:24:40.418 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:24:40.418 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:24:40.419 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:24:40.419 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:24:40.419 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 100, 2020-03-12T03:24:40.419 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 100, 2020-03-12T03:24:40.419 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:24:40.419 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:24:40.419 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:24:40.420 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:24:40.421 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:24:40.421 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:24:40.421 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:24:40.421 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:24:40.421 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:24:40.421 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:24:40.421 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:24:40.422 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.423 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:24:40.423 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:24:40.423 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:24:40.423 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:24:40.423 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 19529728, 2020-03-12T03:24:40.423 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 143228, 2020-03-12T03:24:40.423 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 16, 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 16, 2020-03-12T03:24:40.424 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:24:40.425 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:24:40.425 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 3 2020-03-12T03:24:40.425 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.425 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.425 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_stats": [ 2020-03-12T03:24:40.425 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.425 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:24:40.426 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 141733920778 2020-03-12T03:24:40.426 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.426 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.426 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:24:40.426 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 38654705678 2020-03-12T03:24:40.426 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.426 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 146028888074 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_state": [ 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: "state": "active+undersized+degraded", 2020-03-12T03:24:40.427 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 1 2020-03-12T03:24:40.428 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.428 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.428 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_osd": [ 2020-03-12T03:24:40.428 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.428 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:24:40.428 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:24:40.428 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:24:40.429 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:24:40.429 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.429 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.429 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:24:40.429 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:24:40.429 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 0, 2020-03-12T03:24:40.429 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:24:40.430 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.430 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.430 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:24:40.430 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 1, 2020-03-12T03:24:40.430 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:24:40.430 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:24:40.431 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.431 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.431 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snaps": [ 2020-03-12T03:24:40.431 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:24:40.431 INFO:tasks.workunit.client.0.smithi180.stdout: "pool": 1, 2020-03-12T03:24:40.431 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snaps": {} 2020-03-12T03:24:40.431 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.432 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:24:40.432 INFO:tasks.workunit.client.0.smithi180.stdout: "servicemap": { 2020-03-12T03:24:40.432 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:24:40.432 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:23:04.948788+0000", 2020-03-12T03:24:40.432 INFO:tasks.workunit.client.0.smithi180.stdout: "services": {} 2020-03-12T03:24:40.432 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:24:40.432 INFO:tasks.workunit.client.0.smithi180.stdout: "paxos": { 2020-03-12T03:24:40.433 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:24:40.433 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 299, 2020-03-12T03:24:40.433 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pn": 0, 2020-03-12T03:24:40.433 INFO:tasks.workunit.client.0.smithi180.stdout: "accepted_pn": 0 2020-03-12T03:24:40.433 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:24:40.433 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:24:40.434 INFO:tasks.workunit.client.0.smithi180.stderr:report 590429894 2020-03-12T03:24:40.434 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-12T03:24:40.435 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-12T03:24:40.435 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-12T03:24:40.435 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-12T03:24:40.435 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:24:40.435 INFO:tasks.workunit.client.0.smithi180.stdout:writing non-divergent object existing_4 2020-03-12T03:24:40.602 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:24:40.602 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-12T03:24:21.237258+0000 27'100 39:130 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:24:40.602 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:24:40.602 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:24:40.603 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:24:40.611 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-12T03:24:40.655 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T03:24:40.656 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-12T03:24:40.656 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:24:40.656 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:24:40.656 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:24:40.656 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:24:40.656 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T03:24:40.657 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:24:40.657 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:24:40.658 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:24:40.658 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:24:40.658 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:24:40.711 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:24:40.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:24:40.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:24:40.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:24:40.712 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:24:41.053 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:24:41.053 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:24:41.054 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:24:41.054 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:24:41.054 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:24:41.054 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:24:41.183 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920780 2020-03-12T03:24:41.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920780' 2020-03-12T03:24:41.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:24:41.184 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:24:41.274 INFO:tasks.workunit.client.0.smithi180.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-12T03:24:41.282 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-12T03:24:41.283 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920780 1-' 2020-03-12T03:24:41.283 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:24:41.283 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:24:41.412 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888076 2020-03-12T03:24:41.412 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-141733920780 1- 2-146028888076' 2020-03-12T03:24:41.412 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:24:41.413 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-141733920780 2020-03-12T03:24:41.413 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:24:41.413 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:24:41.413 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-141733920780 2020-03-12T03:24:41.414 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:24:41.414 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920780 2020-03-12T03:24:41.414 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 141733920780' 2020-03-12T03:24:41.415 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:24:41.415 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 141733920780 2020-03-12T03:24:41.757 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920778 -lt 141733920780 2020-03-12T03:24:41.757 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:24:42.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:24:42.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:24:43.110 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920778 -lt 141733920780 2020-03-12T03:24:43.110 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:24:44.111 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-12T03:24:44.111 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:24:44.474 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920781 -lt 141733920780 2020-03-12T03:24:44.475 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:24:44.475 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-12T03:24:44.475 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:24:44.475 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:24:44.476 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-12T03:24:44.476 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:24:44.477 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-12T03:24:44.477 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-12T03:24:44.477 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:24:44.477 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 2020-03-12T03:24:44.823 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705678 -lt 2020-03-12T03:24:44.823 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705678: unary operator expected 2020-03-12T03:24:44.823 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:24:44.824 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-146028888076 2020-03-12T03:24:44.824 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:24:44.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:24:44.825 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-146028888076 2020-03-12T03:24:44.825 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:24:44.826 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888076 2020-03-12T03:24:44.826 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 146028888076' 2020-03-12T03:24:44.826 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:24:44.827 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 146028888076 2020-03-12T03:24:45.173 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888077 -lt 146028888076 2020-03-12T03:24:45.173 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:24:45.173 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:45.174 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:45.627 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:24:45.627 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:45.628 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:45.628 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:45.628 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:45.628 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:45.628 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:45.628 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:45.956 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:45.956 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:45.957 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:45.957 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:46.397 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:46.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-12T03:24:46.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-12T03:24:46.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-12T03:24:46.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:46.398 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-12T03:24:46.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:46.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:46.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:46.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:46.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:46.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:46.500 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:46.500 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:46.827 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:46.828 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:46.828 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:46.828 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:47.270 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:47.270 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:47.270 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:47.270 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:47.271 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:47.271 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:47.271 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:47.271 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:47.271 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:47.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:47.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:47.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-12T03:24:47.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:47.741 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-12T03:24:47.949 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:47.949 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:47.950 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:47.950 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:47.950 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:47.950 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:47.951 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:47.951 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:48.295 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:48.296 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:48.297 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:48.297 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:48.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:48.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:48.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:48.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:48.762 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:48.762 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:48.762 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:48.762 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:48.762 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:49.207 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:49.207 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:49.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-12T03:24:49.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:49.208 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-12T03:24:49.608 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:49.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:49.609 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:49.609 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:49.609 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:49.609 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:49.609 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:49.610 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:49.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:49.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:49.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:49.942 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:50.386 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:50.386 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:50.387 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:50.387 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:50.387 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:50.387 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:50.387 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:50.387 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:50.388 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:50.850 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:50.850 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:50.850 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-12T03:24:50.850 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:50.850 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-12T03:24:51.651 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:51.652 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:51.652 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:51.652 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:51.652 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:51.652 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:51.652 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:51.653 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:51.991 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:51.991 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:51.991 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:51.992 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:52.433 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:52.433 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:52.433 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:52.433 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:52.434 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:52.434 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:52.434 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:52.434 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:52.434 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:52.876 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:52.876 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:52.876 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-12T03:24:52.877 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:52.877 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-12T03:24:54.476 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:54.476 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:54.477 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:54.477 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:54.482 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:54.482 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:54.482 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:54.483 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:54.816 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:54.816 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:54.817 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:54.817 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:55.266 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:55.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:55.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:55.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:55.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:55.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:55.267 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:55.268 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:55.268 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:24:55.708 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:24:55.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:24:55.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-12T03:24:55.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:24:55.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-12T03:24:58.909 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:24:58.909 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:24:58.910 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:24:58.910 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:24:58.910 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:24:58.911 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:24:58.911 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:24:58.911 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:24:59.252 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:24:59.252 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:24:59.253 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:24:59.253 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:24:59.692 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:24:59.692 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:24:59.693 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:24:59.693 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:24:59.693 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:24:59.693 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:24:59.693 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:24:59.693 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:24:59.694 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:25:00.147 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:25:00.147 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:25:00.148 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-12T03:25:00.148 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:25:00.148 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-12T03:25:06.548 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:25:06.549 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:25:06.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:25:06.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:25:06.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:25:06.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:25:06.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:25:06.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:25:06.891 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:25:06.891 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:25:06.892 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:25:06.892 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:25:07.338 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:25:07.339 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:25:07.339 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:25:07.339 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:25:07.339 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:25:07.339 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:25:07.339 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:25:07.340 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:25:07.340 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:25:07.779 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:25:07.779 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:25:07.779 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-12T03:25:07.780 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:25:07.780 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-12T03:25:15.080 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:25:15.080 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:25:15.081 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:25:15.081 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:25:15.081 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:25:15.082 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:25:15.082 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:25:15.082 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:25:15.425 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:25:15.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:25:15.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:25:15.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:25:15.866 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:25:15.866 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:25:15.866 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:25:15.866 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:25:15.867 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:25:15.867 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:25:15.867 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:25:15.867 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:25:15.867 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:25:16.307 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:25:16.307 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:25:16.307 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-12T03:25:16.307 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-12T03:25:16.741 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:25:16.741 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_fingerprint": "00b67cd6-f96e-4da1-bded-61a36e835a93", 2020-03-12T03:25:16.742 INFO:tasks.workunit.client.0.smithi180.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-12T03:25:16.742 INFO:tasks.workunit.client.0.smithi180.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-12T03:25:16.742 INFO:tasks.workunit.client.0.smithi180.stdout: "timestamp": "2020-03-12T03:25:16.739428+0000", 2020-03-12T03:25:16.742 INFO:tasks.workunit.client.0.smithi180.stdout: "tag": "", 2020-03-12T03:25:16.742 INFO:tasks.workunit.client.0.smithi180.stdout: "health": { 2020-03-12T03:25:16.742 INFO:tasks.workunit.client.0.smithi180.stdout: "status": "HEALTH_WARN", 2020-03-12T03:25:16.743 INFO:tasks.workunit.client.0.smithi180.stdout: "checks": { 2020-03-12T03:25:16.743 INFO:tasks.workunit.client.0.smithi180.stdout: "OSDMAP_FLAGS": { 2020-03-12T03:25:16.743 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:25:16.743 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:25:16.743 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-12T03:25:16.743 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 17 2020-03-12T03:25:16.744 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.744 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [], 2020-03-12T03:25:16.744 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:25:16.744 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.744 INFO:tasks.workunit.client.0.smithi180.stdout: "OSD_DOWN": { 2020-03-12T03:25:16.744 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:25:16.744 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:25:16.745 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 osds down", 2020-03-12T03:25:16.745 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:25:16.745 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.745 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:25:16.745 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.745 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "osd.1 (root=default,host=smithi180) is down" 2020-03-12T03:25:16.746 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.746 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.746 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:25:16.746 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.746 INFO:tasks.workunit.client.0.smithi180.stdout: "PG_DEGRADED": { 2020-03-12T03:25:16.746 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:25:16.746 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:25:16.747 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-12T03:25:16.747 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 2 2020-03-12T03:25:16.747 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.747 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:25:16.747 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.748 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-12T03:25:16.748 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.748 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.748 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:25:16.748 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.749 INFO:tasks.workunit.client.0.smithi180.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-12T03:25:16.749 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:25:16.749 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:25:16.749 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-12T03:25:16.749 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:25:16.749 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.750 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:25:16.750 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.750 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "application not enabled on pool 'test'" 2020-03-12T03:25:16.750 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.750 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.750 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-12T03:25:16.751 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.751 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.751 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:25:16.751 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.751 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.751 INFO:tasks.workunit.client.0.smithi180.stdout: "mutes": [] 2020-03-12T03:25:16.751 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.752 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_first_committed": 1, 2020-03-12T03:25:16.752 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_last_committed": 1, 2020-03-12T03:25:16.752 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap": { 2020-03-12T03:25:16.752 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:25:16.752 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:25:16.752 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:23:04.226979Z", 2020-03-12T03:25:16.753 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:23:04.226979Z", 2020-03-12T03:25:16.753 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release": 15, 2020-03-12T03:25:16.753 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release_name": "octopus", 2020-03-12T03:25:16.753 INFO:tasks.workunit.client.0.smithi180.stdout: "features": { 2020-03-12T03:25:16.753 INFO:tasks.workunit.client.0.smithi180.stdout: "persistent": [ 2020-03-12T03:25:16.753 INFO:tasks.workunit.client.0.smithi180.stdout: "kraken", 2020-03-12T03:25:16.753 INFO:tasks.workunit.client.0.smithi180.stdout: "luminous", 2020-03-12T03:25:16.754 INFO:tasks.workunit.client.0.smithi180.stdout: "mimic", 2020-03-12T03:25:16.754 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap-prune", 2020-03-12T03:25:16.754 INFO:tasks.workunit.client.0.smithi180.stdout: "nautilus", 2020-03-12T03:25:16.754 INFO:tasks.workunit.client.0.smithi180.stdout: "octopus" 2020-03-12T03:25:16.754 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.754 INFO:tasks.workunit.client.0.smithi180.stdout: "optional": [] 2020-03-12T03:25:16.754 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.755 INFO:tasks.workunit.client.0.smithi180.stdout: "mons": [ 2020-03-12T03:25:16.755 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.755 INFO:tasks.workunit.client.0.smithi180.stdout: "rank": 0, 2020-03-12T03:25:16.755 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "a", 2020-03-12T03:25:16.755 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:25:16.755 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.756 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.756 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.756 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115", 2020-03-12T03:25:16.756 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 0 2020-03-12T03:25:16.756 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.756 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.757 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.757 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115/0", 2020-03-12T03:25:16.757 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-12T03:25:16.757 INFO:tasks.workunit.client.0.smithi180.stdout: "priority": 0, 2020-03-12T03:25:16.757 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 0 2020-03-12T03:25:16.758 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.758 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.758 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.758 INFO:tasks.workunit.client.0.smithi180.stdout: "quorum": [ 2020-03-12T03:25:16.758 INFO:tasks.workunit.client.0.smithi180.stdout: 0 2020-03-12T03:25:16.758 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.758 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap": { 2020-03-12T03:25:16.759 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 39, 2020-03-12T03:25:16.759 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:25:16.759 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:23:04.948358+0000", 2020-03-12T03:25:16.759 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:24:05.603910+0000", 2020-03-12T03:25:16.759 INFO:tasks.workunit.client.0.smithi180.stdout: "last_up_change": "2020-03-12T03:24:03.834098+0000", 2020-03-12T03:25:16.759 INFO:tasks.workunit.client.0.smithi180.stdout: "last_in_change": "2020-03-12T03:24:05.314095+0000", 2020-03-12T03:25:16.760 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-12T03:25:16.760 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_num": 5800384, 2020-03-12T03:25:16.760 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_set": [ 2020-03-12T03:25:16.760 INFO:tasks.workunit.client.0.smithi180.stdout: "nodown", 2020-03-12T03:25:16.760 INFO:tasks.workunit.client.0.smithi180.stdout: "noin", 2020-03-12T03:25:16.760 INFO:tasks.workunit.client.0.smithi180.stdout: "noout", 2020-03-12T03:25:16.760 INFO:tasks.workunit.client.0.smithi180.stdout: "pglog_hardlimit", 2020-03-12T03:25:16.761 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snapdirs", 2020-03-12T03:25:16.761 INFO:tasks.workunit.client.0.smithi180.stdout: "recovery_deletes", 2020-03-12T03:25:16.761 INFO:tasks.workunit.client.0.smithi180.stdout: "sortbitwise" 2020-03-12T03:25:16.761 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.761 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_version": 7, 2020-03-12T03:25:16.761 INFO:tasks.workunit.client.0.smithi180.stdout: "full_ratio": 0.99000000953674316, 2020-03-12T03:25:16.762 INFO:tasks.workunit.client.0.smithi180.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-12T03:25:16.762 INFO:tasks.workunit.client.0.smithi180.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-12T03:25:16.762 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_snapshot": "", 2020-03-12T03:25:16.762 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_max": 1, 2020-03-12T03:25:16.762 INFO:tasks.workunit.client.0.smithi180.stdout: "max_osd": 3, 2020-03-12T03:25:16.762 INFO:tasks.workunit.client.0.smithi180.stdout: "require_min_compat_client": "jewel", 2020-03-12T03:25:16.763 INFO:tasks.workunit.client.0.smithi180.stdout: "min_compat_client": "jewel", 2020-03-12T03:25:16.763 INFO:tasks.workunit.client.0.smithi180.stdout: "require_osd_release": "octopus", 2020-03-12T03:25:16.763 INFO:tasks.workunit.client.0.smithi180.stdout: "pools": [ 2020-03-12T03:25:16.763 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.763 INFO:tasks.workunit.client.0.smithi180.stdout: "pool": 1, 2020-03-12T03:25:16.763 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_name": "test", 2020-03-12T03:25:16.764 INFO:tasks.workunit.client.0.smithi180.stdout: "create_time": "2020-03-12T03:23:26.536291+0000", 2020-03-12T03:25:16.764 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": 1, 2020-03-12T03:25:16.764 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_names": "hashpspool", 2020-03-12T03:25:16.764 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:25:16.764 INFO:tasks.workunit.client.0.smithi180.stdout: "size": 3, 2020-03-12T03:25:16.764 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 2, 2020-03-12T03:25:16.764 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_rule": 0, 2020-03-12T03:25:16.765 INFO:tasks.workunit.client.0.smithi180.stdout: "object_hash": 2, 2020-03-12T03:25:16.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_autoscale_mode": "off", 2020-03-12T03:25:16.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num": 1, 2020-03-12T03:25:16.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num": 1, 2020-03-12T03:25:16.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num_target": 1, 2020-03-12T03:25:16.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_target": 1, 2020-03-12T03:25:16.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_pending": 1, 2020-03-12T03:25:16.766 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pg_merge_meta": { 2020-03-12T03:25:16.766 INFO:tasks.workunit.client.0.smithi180.stdout: "source_pgid": "0.0", 2020-03-12T03:25:16.766 INFO:tasks.workunit.client.0.smithi180.stdout: "ready_epoch": 0, 2020-03-12T03:25:16.766 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_started": 0, 2020-03-12T03:25:16.766 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_clean": 0, 2020-03-12T03:25:16.766 INFO:tasks.workunit.client.0.smithi180.stdout: "source_version": "0'0", 2020-03-12T03:25:16.766 INFO:tasks.workunit.client.0.smithi180.stdout: "target_version": "0'0" 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: "last_change": "27", 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend": "0", 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_preluminous": "0", 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: "auid": 0, 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_mode": "selfmanaged", 2020-03-12T03:25:16.767 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_seq": 0, 2020-03-12T03:25:16.768 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_epoch": 0, 2020-03-12T03:25:16.768 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_snaps": [], 2020-03-12T03:25:16.768 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps": "[]", 2020-03-12T03:25:16.768 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_bytes": 0, 2020-03-12T03:25:16.768 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_objects": 0, 2020-03-12T03:25:16.768 INFO:tasks.workunit.client.0.smithi180.stdout: "tiers": [], 2020-03-12T03:25:16.768 INFO:tasks.workunit.client.0.smithi180.stdout: "tier_of": -1, 2020-03-12T03:25:16.769 INFO:tasks.workunit.client.0.smithi180.stdout: "read_tier": -1, 2020-03-12T03:25:16.769 INFO:tasks.workunit.client.0.smithi180.stdout: "write_tier": -1, 2020-03-12T03:25:16.769 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_mode": "none", 2020-03-12T03:25:16.769 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_bytes": 0, 2020-03-12T03:25:16.769 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_objects": 0, 2020-03-12T03:25:16.769 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-12T03:25:16.770 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-12T03:25:16.770 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-12T03:25:16.770 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_flush_age": 0, 2020-03-12T03:25:16.770 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_evict_age": 0, 2020-03-12T03:25:16.770 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profile": "", 2020-03-12T03:25:16.771 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_params": { 2020-03-12T03:25:16.771 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "none" 2020-03-12T03:25:16.771 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.771 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_period": 0, 2020-03-12T03:25:16.771 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_count": 0, 2020-03-12T03:25:16.771 INFO:tasks.workunit.client.0.smithi180.stdout: "use_gmt_hitset": true, 2020-03-12T03:25:16.772 INFO:tasks.workunit.client.0.smithi180.stdout: "min_read_recency_for_promote": 0, 2020-03-12T03:25:16.772 INFO:tasks.workunit.client.0.smithi180.stdout: "min_write_recency_for_promote": 0, 2020-03-12T03:25:16.772 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_grade_decay_rate": 0, 2020-03-12T03:25:16.772 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_search_last_n": 0, 2020-03-12T03:25:16.772 INFO:tasks.workunit.client.0.smithi180.stdout: "grade_table": [], 2020-03-12T03:25:16.772 INFO:tasks.workunit.client.0.smithi180.stdout: "stripe_width": 0, 2020-03-12T03:25:16.772 INFO:tasks.workunit.client.0.smithi180.stdout: "expected_num_objects": 0, 2020-03-12T03:25:16.773 INFO:tasks.workunit.client.0.smithi180.stdout: "fast_read": false, 2020-03-12T03:25:16.773 INFO:tasks.workunit.client.0.smithi180.stdout: "options": {}, 2020-03-12T03:25:16.773 INFO:tasks.workunit.client.0.smithi180.stdout: "application_metadata": {} 2020-03-12T03:25:16.773 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.773 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.773 INFO:tasks.workunit.client.0.smithi180.stdout: "osds": [ 2020-03-12T03:25:16.773 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.774 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:25:16.774 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "59ecfd79-e15b-4216-a9fb-a337e29afe1e", 2020-03-12T03:25:16.774 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:25:16.774 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:25:16.774 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:25:16.774 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:25:16.774 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 5, 2020-03-12T03:25:16.775 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 27, 2020-03-12T03:25:16.775 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 33, 2020-03-12T03:25:16.775 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 38, 2020-03-12T03:25:16.775 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 28, 2020-03-12T03:25:16.775 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:25:16.775 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:25:16.775 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.776 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.776 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.776 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6802", 2020-03-12T03:25:16.776 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.776 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.776 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.776 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.777 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6803", 2020-03-12T03:25:16.777 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.777 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.777 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.777 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.777 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:25:16.777 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.778 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.778 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.778 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6804", 2020-03-12T03:25:16.778 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.778 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.778 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.779 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.779 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6805", 2020-03-12T03:25:16.779 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.779 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.779 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.779 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.779 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:25:16.780 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.780 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.780 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.780 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6808", 2020-03-12T03:25:16.780 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.780 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.780 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.781 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.781 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6809", 2020-03-12T03:25:16.781 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.781 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.781 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.781 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.782 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:25:16.782 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.782 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.782 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.782 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6806", 2020-03-12T03:25:16.782 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.783 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.783 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.783 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.783 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6807", 2020-03-12T03:25:16.783 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29454 2020-03-12T03:25:16.783 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.783 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.784 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.784 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6803/29454", 2020-03-12T03:25:16.784 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6805/29454", 2020-03-12T03:25:16.784 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6809/29454", 2020-03-12T03:25:16.784 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6807/29454", 2020-03-12T03:25:16.784 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:25:16.785 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:25:16.785 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:25:16.785 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.785 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.785 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.785 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:25:16.785 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "eb5be901-51a3-4f54-bb09-dd4860cf0f9c", 2020-03-12T03:25:16.786 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 0, 2020-03-12T03:25:16.786 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:25:16.786 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:25:16.786 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:25:16.786 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 0, 2020-03-12T03:25:16.786 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 0, 2020-03-12T03:25:16.787 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 9, 2020-03-12T03:25:16.787 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 21, 2020-03-12T03:25:16.787 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 29, 2020-03-12T03:25:16.787 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:25:16.787 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:25:16.787 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.787 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.788 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.788 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:25:16.788 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.788 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.788 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.788 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.788 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:25:16.789 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.789 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.789 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.789 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.789 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:25:16.789 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.790 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.790 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.790 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:25:16.790 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.790 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.790 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.791 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.791 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:25:16.791 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.791 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.791 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.791 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.791 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:25:16.792 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.792 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.792 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.792 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:25:16.792 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.792 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.792 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.793 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.793 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:25:16.793 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.793 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.793 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.793 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.794 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:25:16.794 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.794 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.794 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.794 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:25:16.794 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.794 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.795 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.795 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.795 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:25:16.795 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 25074 2020-03-12T03:25:16.795 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.795 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.796 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.796 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/25074", 2020-03-12T03:25:16.796 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/25074", 2020-03-12T03:25:16.796 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25074", 2020-03-12T03:25:16.796 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25074", 2020-03-12T03:25:16.796 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:25:16.797 INFO:tasks.workunit.client.0.smithi180.stdout: "exists" 2020-03-12T03:25:16.797 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.797 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.797 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.797 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:25:16.797 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "3f99c107-f1bf-46d1-92a5-89dec9b23f29", 2020-03-12T03:25:16.797 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:25:16.798 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:25:16.798 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:25:16.798 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:25:16.798 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 14, 2020-03-12T03:25:16.798 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 27, 2020-03-12T03:25:16.798 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 34, 2020-03-12T03:25:16.798 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 0, 2020-03-12T03:25:16.799 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 30, 2020-03-12T03:25:16.799 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:25:16.799 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:25:16.799 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.799 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.799 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.799 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:25:16.800 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.800 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.800 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.800 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.800 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:25:16.800 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.800 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.801 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.801 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.801 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:25:16.801 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.801 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.801 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.801 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.802 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.803 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.803 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:25:16.803 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.803 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.803 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.803 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:25:16.804 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.804 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.804 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.804 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.804 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:25:16.804 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.805 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.805 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.805 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.805 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:25:16.805 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:25:16.805 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.806 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:25:16.806 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:25:16.806 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.806 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.806 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.806 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:25:16.807 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:25:16.807 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 29648 2020-03-12T03:25:16.807 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.807 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.807 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.807 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/29648", 2020-03-12T03:25:16.807 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/29648", 2020-03-12T03:25:16.808 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/29648", 2020-03-12T03:25:16.808 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/29648", 2020-03-12T03:25:16.808 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:25:16.808 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:25:16.808 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:25:16.808 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.808 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.809 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.809 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_xinfo": [ 2020-03-12T03:25:16.809 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.809 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:25:16.809 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:23:57.738718+0000", 2020-03-12T03:25:16.809 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:25:16.809 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:25:16.810 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:25:16.810 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:25:16.810 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:23:11.947656+0000", 2020-03-12T03:25:16.810 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:25:16.810 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.810 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.811 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:25:16.811 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:23:58.239159+0000", 2020-03-12T03:25:16.811 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:25:16.811 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:25:16.811 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:25:16.811 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:25:16.811 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:23:18.186236+0000", 2020-03-12T03:25:16.812 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:25:16.812 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.812 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.812 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:25:16.812 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:23:58.737011+0000", 2020-03-12T03:25:16.812 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:25:16.813 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:25:16.813 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:25:16.813 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:25:16.813 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:23:24.519963+0000", 2020-03-12T03:25:16.813 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:25:16.813 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.813 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.814 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap": [], 2020-03-12T03:25:16.814 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap_items": [], 2020-03-12T03:25:16.814 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_temp": [], 2020-03-12T03:25:16.814 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_temp": [], 2020-03-12T03:25:16.814 INFO:tasks.workunit.client.0.smithi180.stdout: "blacklist": {}, 2020-03-12T03:25:16.814 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profiles": { 2020-03-12T03:25:16.815 INFO:tasks.workunit.client.0.smithi180.stdout: "default": { 2020-03-12T03:25:16.815 INFO:tasks.workunit.client.0.smithi180.stdout: "k": "2", 2020-03-12T03:25:16.815 INFO:tasks.workunit.client.0.smithi180.stdout: "m": "2", 2020-03-12T03:25:16.815 INFO:tasks.workunit.client.0.smithi180.stdout: "plugin": "jerasure", 2020-03-12T03:25:16.815 INFO:tasks.workunit.client.0.smithi180.stdout: "technique": "reed_sol_van" 2020-03-12T03:25:16.815 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.816 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.816 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps_queue": [], 2020-03-12T03:25:16.816 INFO:tasks.workunit.client.0.smithi180.stdout: "new_removed_snaps": [], 2020-03-12T03:25:16.816 INFO:tasks.workunit.client.0.smithi180.stdout: "new_purged_snaps": [], 2020-03-12T03:25:16.816 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_node_flags": {}, 2020-03-12T03:25:16.816 INFO:tasks.workunit.client.0.smithi180.stdout: "device_class_flags": {} 2020-03-12T03:25:16.816 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.817 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_metadata": [ 2020-03-12T03:25:16.817 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.817 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:25:16.817 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:25:16.817 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6804/29454,v1:127.0.0.1:6805/29454]", 2020-03-12T03:25:16.817 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:25:16.817 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:25:16.818 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:25:16.818 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:25:16.818 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:25:16.818 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:25:16.818 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:25:16.818 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:25:16.818 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:25:16.819 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-12T03:25:16.819 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:25:16.819 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:25:16.819 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:25:16.819 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:25:16.819 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:25:16.819 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:25:16.820 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:25:16.821 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:25:16.821 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6802/29454,v1:127.0.0.1:6803/29454]", 2020-03-12T03:25:16.821 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:25:16.821 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/29454,v1:127.0.0.1:6809/29454]", 2020-03-12T03:25:16.821 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/29454,v1:127.0.0.1:6807/29454]", 2020-03-12T03:25:16.821 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:25:16.821 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:25:16.822 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:25:16.822 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:25:16.822 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:25:16.822 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:25:16.822 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:25:16.822 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:25:16.822 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:25:16.823 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/0", 2020-03-12T03:25:16.823 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:25:16.823 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:25:16.823 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.823 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.823 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:25:16.823 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:25:16.824 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/25074,v1:127.0.0.1:6813/25074]", 2020-03-12T03:25:16.824 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:25:16.824 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:25:16.824 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:25:16.824 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:25:16.824 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:25:16.825 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:25:16.825 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:25:16.825 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:25:16.825 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:25:16.825 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-12T03:25:16.825 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:25:16.826 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:25:16.826 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:25:16.826 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:25:16.826 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:25:16.826 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:25:16.826 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:25:16.827 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:25:16.827 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:25:16.827 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:25:16.827 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:25:16.827 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:25:16.827 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:25:16.827 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:25:16.828 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:25:16.828 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/25074,v1:127.0.0.1:6811/25074]", 2020-03-12T03:25:16.828 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:25:16.828 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25074,v1:127.0.0.1:6817/25074]", 2020-03-12T03:25:16.828 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25074,v1:127.0.0.1:6815/25074]", 2020-03-12T03:25:16.828 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:25:16.829 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:25:16.829 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:25:16.829 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:25:16.829 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:25:16.829 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:25:16.829 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:25:16.830 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:25:16.830 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:25:16.830 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/1", 2020-03-12T03:25:16.830 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:25:16.830 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:25:16.830 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.830 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.831 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:25:16.831 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:25:16.831 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/29648,v1:127.0.0.1:6813/29648]", 2020-03-12T03:25:16.831 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:25:16.831 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:25:16.831 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:25:16.831 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:25:16.832 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:25:16.832 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:25:16.832 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:25:16.832 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:25:16.832 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:25:16.832 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-12T03:25:16.832 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:25:16.833 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:25:16.833 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:25:16.833 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:25:16.833 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:25:16.833 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:25:16.833 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:25:16.834 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:25:16.835 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/29648,v1:127.0.0.1:6811/29648]", 2020-03-12T03:25:16.835 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:25:16.835 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/29648,v1:127.0.0.1:6817/29648]", 2020-03-12T03:25:16.835 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/29648,v1:127.0.0.1:6815/29648]", 2020-03-12T03:25:16.835 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:25:16.835 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:25:16.835 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:25:16.836 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:25:16.836 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:25:16.836 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:25:16.836 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:25:16.836 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:25:16.836 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:25:16.836 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/2", 2020-03-12T03:25:16.837 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:25:16.837 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:25:16.837 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.837 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.837 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_first_committed": 1, 2020-03-12T03:25:16.837 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_last_committed": 39, 2020-03-12T03:25:16.838 INFO:tasks.workunit.client.0.smithi180.stdout: "crushmap": { 2020-03-12T03:25:16.838 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": [ 2020-03-12T03:25:16.838 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.838 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:25:16.838 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.0", 2020-03-12T03:25:16.838 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:25:16.839 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.839 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.839 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:25:16.839 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.1", 2020-03-12T03:25:16.839 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:25:16.839 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.840 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.840 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:25:16.840 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.2", 2020-03-12T03:25:16.840 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:25:16.840 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.840 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.840 INFO:tasks.workunit.client.0.smithi180.stdout: "types": [ 2020-03-12T03:25:16.841 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.841 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 0, 2020-03-12T03:25:16.841 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd" 2020-03-12T03:25:16.841 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.841 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.841 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:25:16.841 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "host" 2020-03-12T03:25:16.842 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.842 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.842 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 2, 2020-03-12T03:25:16.842 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "chassis" 2020-03-12T03:25:16.842 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.842 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.842 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 3, 2020-03-12T03:25:16.843 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "rack" 2020-03-12T03:25:16.843 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.843 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.843 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 4, 2020-03-12T03:25:16.843 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "row" 2020-03-12T03:25:16.843 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.843 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.844 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 5, 2020-03-12T03:25:16.844 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pdu" 2020-03-12T03:25:16.844 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.844 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.844 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 6, 2020-03-12T03:25:16.844 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pod" 2020-03-12T03:25:16.844 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 7, 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "room" 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 8, 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "datacenter" 2020-03-12T03:25:16.845 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.846 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.846 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 9, 2020-03-12T03:25:16.846 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "zone" 2020-03-12T03:25:16.846 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.846 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.846 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 10, 2020-03-12T03:25:16.846 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "region" 2020-03-12T03:25:16.847 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.847 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.847 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:25:16.847 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "root" 2020-03-12T03:25:16.847 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.847 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.847 INFO:tasks.workunit.client.0.smithi180.stdout: "buckets": [ 2020-03-12T03:25:16.848 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.848 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -1, 2020-03-12T03:25:16.848 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default", 2020-03-12T03:25:16.848 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:25:16.848 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:25:16.849 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:25:16.849 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:25:16.849 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:25:16.849 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:25:16.849 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.850 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:25:16.850 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:25:16.850 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:25:16.850 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.850 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.850 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -2, 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default~hdd", 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:25:16.851 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:25:16.852 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:25:16.852 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.852 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:25:16.852 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:25:16.852 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:25:16.852 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.852 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.853 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.853 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.853 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:25:16.853 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180", 2020-03-12T03:25:16.853 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:25:16.853 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:25:16.853 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:25:16.854 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:25:16.854 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:25:16.854 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:25:16.854 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.854 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:25:16.854 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:25:16.854 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:25:16.855 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.855 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.855 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:25:16.855 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:25:16.855 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:25:16.855 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.855 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.856 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:25:16.856 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:25:16.856 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:25:16.856 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.856 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.856 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.856 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.857 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:25:16.857 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180~hdd", 2020-03-12T03:25:16.857 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:25:16.857 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:25:16.857 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:25:16.857 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:25:16.858 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:25:16.858 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:25:16.858 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.858 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:25:16.859 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:25:16.859 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:25:16.859 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.859 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.859 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:25:16.859 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:25:16.859 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:25:16.860 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.860 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.860 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:25:16.860 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:25:16.860 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:25:16.860 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.861 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.861 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.861 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.861 INFO:tasks.workunit.client.0.smithi180.stdout: "rules": [ 2020-03-12T03:25:16.861 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.861 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_id": 0, 2020-03-12T03:25:16.861 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_name": "replicated_rule", 2020-03-12T03:25:16.862 INFO:tasks.workunit.client.0.smithi180.stdout: "ruleset": 0, 2020-03-12T03:25:16.862 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:25:16.862 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 1, 2020-03-12T03:25:16.862 INFO:tasks.workunit.client.0.smithi180.stdout: "max_size": 10, 2020-03-12T03:25:16.862 INFO:tasks.workunit.client.0.smithi180.stdout: "steps": [ 2020-03-12T03:25:16.862 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.862 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "take", 2020-03-12T03:25:16.863 INFO:tasks.workunit.client.0.smithi180.stdout: "item": -1, 2020-03-12T03:25:16.863 INFO:tasks.workunit.client.0.smithi180.stdout: "item_name": "default" 2020-03-12T03:25:16.863 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.863 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.863 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "choose_firstn", 2020-03-12T03:25:16.863 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 0, 2020-03-12T03:25:16.864 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "osd" 2020-03-12T03:25:16.864 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.864 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.864 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "emit" 2020-03-12T03:25:16.864 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.864 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:25:16.865 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.865 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.865 INFO:tasks.workunit.client.0.smithi180.stdout: "tunables": { 2020-03-12T03:25:16.865 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_tries": 0, 2020-03-12T03:25:16.865 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_fallback_tries": 0, 2020-03-12T03:25:16.865 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_total_tries": 50, 2020-03-12T03:25:16.866 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_descend_once": 1, 2020-03-12T03:25:16.866 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_vary_r": 1, 2020-03-12T03:25:16.866 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_stable": 1, 2020-03-12T03:25:16.866 INFO:tasks.workunit.client.0.smithi180.stdout: "straw_calc_version": 1, 2020-03-12T03:25:16.866 INFO:tasks.workunit.client.0.smithi180.stdout: "allowed_bucket_algs": 54, 2020-03-12T03:25:16.866 INFO:tasks.workunit.client.0.smithi180.stdout: "profile": "jewel", 2020-03-12T03:25:16.866 INFO:tasks.workunit.client.0.smithi180.stdout: "optimal_tunables": 1, 2020-03-12T03:25:16.867 INFO:tasks.workunit.client.0.smithi180.stdout: "legacy_tunables": 0, 2020-03-12T03:25:16.867 INFO:tasks.workunit.client.0.smithi180.stdout: "minimum_required_version": "jewel", 2020-03-12T03:25:16.867 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables": 1, 2020-03-12T03:25:16.867 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables2": 1, 2020-03-12T03:25:16.867 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v2_rules": 0, 2020-03-12T03:25:16.867 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables3": 1, 2020-03-12T03:25:16.867 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v3_rules": 0, 2020-03-12T03:25:16.868 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v4_buckets": 1, 2020-03-12T03:25:16.868 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables5": 1, 2020-03-12T03:25:16.868 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v5_rules": 0 2020-03-12T03:25:16.868 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.868 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_args": {} 2020-03-12T03:25:16.868 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.868 INFO:tasks.workunit.client.0.smithi180.stdout: "fsmap": { 2020-03-12T03:25:16.869 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:25:16.869 INFO:tasks.workunit.client.0.smithi180.stdout: "default_fscid": -1, 2020-03-12T03:25:16.869 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": { 2020-03-12T03:25:16.869 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": {}, 2020-03-12T03:25:16.869 INFO:tasks.workunit.client.0.smithi180.stdout: "ro_compat": {}, 2020-03-12T03:25:16.869 INFO:tasks.workunit.client.0.smithi180.stdout: "incompat": { 2020-03-12T03:25:16.869 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_1": "base v0.20", 2020-03-12T03:25:16.870 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_2": "client writeable ranges", 2020-03-12T03:25:16.870 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_3": "default file layouts on dirs", 2020-03-12T03:25:16.870 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_4": "dir inode in separate object", 2020-03-12T03:25:16.870 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_5": "mds uses versioned encoding", 2020-03-12T03:25:16.870 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-12T03:25:16.871 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_8": "no anchor table", 2020-03-12T03:25:16.871 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_9": "file layout v2", 2020-03-12T03:25:16.871 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_10": "snaprealm v2" 2020-03-12T03:25:16.871 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.871 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.871 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_flags": { 2020-03-12T03:25:16.871 INFO:tasks.workunit.client.0.smithi180.stdout: "enable_multiple": false, 2020-03-12T03:25:16.872 INFO:tasks.workunit.client.0.smithi180.stdout: "ever_enabled_multiple": false 2020-03-12T03:25:16.872 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.872 INFO:tasks.workunit.client.0.smithi180.stdout: "standbys": [], 2020-03-12T03:25:16.872 INFO:tasks.workunit.client.0.smithi180.stdout: "filesystems": [] 2020-03-12T03:25:16.872 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.872 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_first_committed": 1, 2020-03-12T03:25:16.872 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_last_committed": 1, 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: "auth": { 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 5, 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: "num_secrets": 3 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 1, 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_active": 1, 2020-03-12T03:25:16.873 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_unknown": 0, 2020-03-12T03:25:16.874 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osd": 3, 2020-03-12T03:25:16.874 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_sum": { 2020-03-12T03:25:16.874 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:25:16.874 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 47947, 2020-03-12T03:25:16.874 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:25:16.874 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:25:16.874 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:25:16.875 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:25:16.875 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:25:16.875 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:25:16.875 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:25:16.875 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:25:16.875 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:25:16.875 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:25:16.876 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:25:16.876 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:25:16.876 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 101, 2020-03-12T03:25:16.876 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 101, 2020-03-12T03:25:16.876 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:25:16.876 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:25:16.877 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:25:16.877 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:25:16.877 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:25:16.877 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:25:16.877 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:25:16.877 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:25:16.878 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:25:16.878 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:25:16.878 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:25:16.878 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:25:16.878 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:25:16.878 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:25:16.878 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:25:16.879 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:25:16.879 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:25:16.879 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:25:16.879 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:25:16.879 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:25:16.879 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:25:16.880 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:25:16.880 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:25:16.880 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:25:16.880 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:25:16.880 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.880 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:25:16.880 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:25:16.881 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:25:16.881 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:25:16.881 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 0, 2020-03-12T03:25:16.881 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 0, 2020-03-12T03:25:16.881 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:25:16.881 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:25:16.882 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:25:16.882 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:25:16.882 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:25:16.882 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.882 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 17, 2020-03-12T03:25:16.882 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 17, 2020-03-12T03:25:16.882 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:25:16.883 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:25:16.883 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 0 2020-03-12T03:25:16.883 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.883 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum": { 2020-03-12T03:25:16.883 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:25:16.883 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:25:16.883 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:25:16.884 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:25:16.884 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:25:16.884 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:25:16.884 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:25:16.884 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3178944, 2020-03-12T03:25:16.884 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 32896, 2020-03-12T03:25:16.884 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 12, 2020-03-12T03:25:16.885 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145715, 2020-03-12T03:25:16.885 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393856, 2020-03-12T03:25:16.885 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:25:16.885 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:25:16.885 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867308544, 2020-03-12T03:25:16.885 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:25:16.886 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33685504, 2020-03-12T03:25:16.886 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 402018, 2020-03-12T03:25:16.886 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:25:16.886 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:25:16.886 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:25:16.886 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 12914, 2020-03-12T03:25:16.887 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221212558 2020-03-12T03:25:16.887 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.887 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:25:16.887 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:25:16.887 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:25:16.887 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 0, 2020-03-12T03:25:16.888 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:25:16.888 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:25:16.888 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:25:16.888 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.888 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:25:16.888 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:25:16.888 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:25:16.889 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:25:16.889 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:25:16.889 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.889 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:25:16.889 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:25:16.889 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.890 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum_by_class": { 2020-03-12T03:25:16.890 INFO:tasks.workunit.client.0.smithi180.stdout: "hdd": { 2020-03-12T03:25:16.890 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:25:16.890 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:25:16.890 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:25:16.890 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:25:16.891 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:25:16.891 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:25:16.891 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:25:16.891 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3178944, 2020-03-12T03:25:16.891 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 32896, 2020-03-12T03:25:16.891 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 12, 2020-03-12T03:25:16.891 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145715, 2020-03-12T03:25:16.892 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393856, 2020-03-12T03:25:16.892 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:25:16.892 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:25:16.892 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867308544, 2020-03-12T03:25:16.892 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:25:16.892 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33685504, 2020-03-12T03:25:16.893 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 402018, 2020-03-12T03:25:16.893 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:25:16.893 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:25:16.893 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:25:16.893 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 12914, 2020-03-12T03:25:16.893 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221212558 2020-03-12T03:25:16.893 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.894 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:25:16.894 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:25:16.894 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:25:16.894 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 0, 2020-03-12T03:25:16.894 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:25:16.894 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:25:16.895 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:25:16.895 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.895 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:25:16.895 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:25:16.895 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:25:16.895 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:25:16.895 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:25:16.896 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.896 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:25:16.896 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:25:16.896 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.896 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.896 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_stats": [ 2020-03-12T03:25:16.897 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.897 INFO:tasks.workunit.client.0.smithi180.stdout: "poolid": 1, 2020-03-12T03:25:16.897 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 1, 2020-03-12T03:25:16.897 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:25:16.897 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 47947, 2020-03-12T03:25:16.897 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:25:16.898 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:25:16.898 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:25:16.898 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:25:16.898 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:25:16.898 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:25:16.898 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:25:16.898 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:25:16.899 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:25:16.899 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:25:16.899 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:25:16.899 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:25:16.899 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 101, 2020-03-12T03:25:16.899 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 101, 2020-03-12T03:25:16.899 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:25:16.900 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:25:16.900 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:25:16.900 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:25:16.900 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:25:16.900 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:25:16.900 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:25:16.900 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:25:16.901 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:25:16.901 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:25:16.901 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:25:16.901 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:25:16.901 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:25:16.901 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:25:16.901 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:25:16.902 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:25:16.902 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:25:16.902 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:25:16.902 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:25:16.902 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:25:16.902 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:25:16.902 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:25:16.903 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:25:16.903 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:25:16.903 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:25:16.903 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.903 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:25:16.903 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:25:16.903 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:25:16.904 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:25:16.904 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 19529728, 2020-03-12T03:25:16.904 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 142522, 2020-03-12T03:25:16.904 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:25:16.904 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:25:16.904 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:25:16.904 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:25:16.905 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:25:16.905 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.905 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 17, 2020-03-12T03:25:16.905 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 17, 2020-03-12T03:25:16.905 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:25:16.905 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:25:16.905 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 3 2020-03-12T03:25:16.906 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.906 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.906 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_stats": [ 2020-03-12T03:25:16.906 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.906 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:25:16.907 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 141733920787 2020-03-12T03:25:16.907 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.907 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.907 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:25:16.907 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 38654705678 2020-03-12T03:25:16.907 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.907 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.908 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:25:16.908 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 146028888083 2020-03-12T03:25:16.908 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.908 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.908 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_state": [ 2020-03-12T03:25:16.908 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.909 INFO:tasks.workunit.client.0.smithi180.stdout: "state": "active+undersized+degraded", 2020-03-12T03:25:16.909 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 1 2020-03-12T03:25:16.909 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.909 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.909 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_osd": [ 2020-03-12T03:25:16.909 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.909 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:25:16.910 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:25:16.910 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:25:16.910 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:25:16.910 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.910 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.910 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:25:16.911 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:25:16.911 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 0, 2020-03-12T03:25:16.911 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:25:16.911 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.911 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.911 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:25:16.911 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 1, 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snaps": [ 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: "pool": 1, 2020-03-12T03:25:16.912 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snaps": {} 2020-03-12T03:25:16.913 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.913 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:25:16.913 INFO:tasks.workunit.client.0.smithi180.stdout: "servicemap": { 2020-03-12T03:25:16.913 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:25:16.913 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:23:04.948788+0000", 2020-03-12T03:25:16.913 INFO:tasks.workunit.client.0.smithi180.stdout: "services": {} 2020-03-12T03:25:16.913 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:25:16.914 INFO:tasks.workunit.client.0.smithi180.stdout: "paxos": { 2020-03-12T03:25:16.914 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:25:16.914 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 378, 2020-03-12T03:25:16.914 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pn": 0, 2020-03-12T03:25:16.914 INFO:tasks.workunit.client.0.smithi180.stdout: "accepted_pn": 0 2020-03-12T03:25:16.914 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:25:16.914 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:25:16.915 INFO:tasks.workunit.client.0.smithi180.stderr:report 4102013629 2020-03-12T03:25:16.916 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-12T03:25:16.916 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-12T03:25:16.916 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:16.916 INFO:tasks.workunit.client.0.smithi180.stdout:delay recovery 2020-03-12T03:25:17.071 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:17.072 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:24:21.237258+0000 39'101 39:131 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:17.072 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:17.072 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:17.073 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:17.081 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:25:17.082 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-12T03:25:17.083 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-12T03:25:17.083 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-12T03:25:17.083 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:25:17.083 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:25:17.083 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:25:17.083 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.0.asok 2020-03-12T03:25:17.083 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-12T03:25:17.084 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.0.asok set_recovery_delay 100000 2020-03-12T03:25:17.174 INFO:tasks.workunit.client.0.smithi180.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-12T03:25:17.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-12T03:25:17.183 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-12T03:25:17.183 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-12T03:25:17.183 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-12T03:25:17.183 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:25:17.184 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:25:17.184 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:25:17.184 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.2.asok 2020-03-12T03:25:17.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-12T03:25:17.184 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.2.asok set_recovery_delay 100000 2020-03-12T03:25:17.273 INFO:tasks.workunit.client.0.smithi180.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-12T03:25:17.280 INFO:tasks.workunit.client.0.smithi180.stdout:revive divergent 1 2020-03-12T03:25:17.280 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-12T03:25:17.280 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:17.600 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:17.600 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:24:21.237258+0000 39'101 39:131 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:17.600 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:17.601 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:17.602 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:17.610 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-12T03:25:18.176 INFO:tasks.workunit.client.0.smithi180.stderr:noup is set 2020-03-12T03:25:18.187 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-12T03:25:18.187 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:25:18.187 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:25:18.187 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-12T03:25:18.187 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:25:18.188 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-12T03:25:18.188 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:25:18.188 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:25:18.188 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:25:18.188 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:25:18.188 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T03:25:18.188 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T03:25:18.189 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:25:18.189 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:25:18.189 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:25:18.189 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:25:18.189 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:25:18.189 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:25:18.189 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:25:18.190 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:25:18.190 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:25:18.190 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:25:18.190 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:25:18.190 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:25:18.190 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:25:18.190 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:25:18.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:25:18.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:25:18.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:25:18.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:25:18.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:25:18.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-12T03:25:18.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-12T03:25:18.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:25:18.192 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-12T03:25:18.192 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T03:25:18.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-12T03:25:18.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:25:18.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:25:18.193 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:25:18.213 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:18.211+0000 7fb6d1285ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:25:18.215 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:18.213+0000 7fb6d1285ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:25:18.216 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:18.214+0000 7fb6d1285ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:25:18.537 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-12T03:25:18.537 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-12T03:25:18.736 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:18.734+0000 7fb6d1285ec0 -1 Falling back to public interface 2020-03-12T03:25:19.600 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:19.598+0000 7fb6d1285ec0 -1 osd.1 27 log_to_monitors {default=true} 2020-03-12T03:25:23.538 INFO:tasks.workunit.client.0.smithi180.stdout:delay recovery divergent 2020-03-12T03:25:23.539 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-12T03:25:23.539 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:23.868 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:23.868 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:24:21.237258+0000 39'101 39:132 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:23.868 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:23.868 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:23.870 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:23.879 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-12T03:25:23.879 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-12T03:25:23.879 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-12T03:25:23.879 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:25:23.879 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:25:23.879 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:25:23.880 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.1.asok 2020-03-12T03:25:23.880 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-12T03:25:23.880 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.1.asok set_recovery_delay 100000 2020-03-12T03:25:23.970 INFO:tasks.workunit.client.0.smithi180.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-12T03:25:23.977 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-12T03:25:24.599 INFO:tasks.workunit.client.0.smithi180.stderr:noup is unset 2020-03-12T03:25:24.608 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-12T03:25:24.608 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:25:24.608 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:25:24.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:25:24.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:25:24.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:25:24.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:25:24.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:25:24.609 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:25:24.610 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:25:24.962 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up in weight 1 up_from 33 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/29454,v1:127.0.0.1:6803/29454] [v2:127.0.0.1:6804/29454,v1:127.0.0.1:6805/29454] exists,up 59ecfd79-e15b-4216-a9fb-a337e29afe1e 2020-03-12T03:25:24.963 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:25:24.963 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:25:24.963 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:25:24.964 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:25:24.964 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-12T03:25:24.964 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:25:24.964 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:25:24.965 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:25:24.965 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:25:24.965 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:25:24.965 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:25:24.965 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:25:24.966 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:25:25.329 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 29 last_clean_interval [9,27) [v2:127.0.0.1:6818/32236,v1:127.0.0.1:6819/32236] [v2:127.0.0.1:6820/32236,v1:127.0.0.1:6821/32236] exists,up eb5be901-51a3-4f54-bb09-dd4860cf0f9c 2020-03-12T03:25:25.329 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:25:25.329 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:25:25.329 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:25:25.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:25:25.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-12T03:25:25.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:25:25.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:25:25.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:25:25.330 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:25:25.331 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:25:25.331 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:25:25.331 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:25:25.331 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:25:25.681 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/29648,v1:127.0.0.1:6811/29648] [v2:127.0.0.1:6812/29648,v1:127.0.0.1:6813/29648] exists,up 3f99c107-f1bf-46d1-92a5-89dec9b23f29 2020-03-12T03:25:25.682 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:25:25.682 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:25:25.682 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:25:25.682 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:26.001 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:26.002 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:24:21.237258+0000 39'101 39:132 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:26.002 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:26.002 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:26.003 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:26.011 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-12T03:25:26.012 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:26.058 INFO:tasks.workunit.client.0.smithi180.stdout:wait for peering 2020-03-12T03:25:26.330 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:26.330 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-12T03:24:21.237258+0000 39'101 39:132 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:26.330 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:26.330 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:26.331 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:26.339 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-12T03:25:33.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-12T03:25:33.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:33.814 INFO:tasks.workunit.client.0.smithi180.stdout:killing divergent 1 2020-03-12T03:25:34.142 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:34.142 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-12T03:25:24.828472+0000 39'101 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:34.143 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:34.143 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:34.144 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:34.152 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-12T03:25:34.152 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:25:34.152 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:25:34.153 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:25:34.153 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:25:34.153 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:25:34.258 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:25:34.259 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.16628.out 2020-03-12T03:25:34.259 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.16628.out 2020-03-12T03:25:34.259 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-12T03:25:34.259 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-12T03:25:34.259 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-12T03:25:34.259 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-12T03:25:34.259 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-12T03:25:34.260 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.16628.out 2020-03-12T03:25:34.820 INFO:tasks.workunit.client.0.smithi180.stderr:Exporting 1.0 info 1.0( v 44'102 lc 0'0 (27'90,44'102] local-lis/les=43/44 n=101 ec=21/21 lis/c=43/21 les/c/f=44/22/0 sis=43 pruub=8.925691086s) 2020-03-12T03:25:34.829 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-12T03:25:34.845 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-12T03:25:34.856 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-12T03:25:34.863 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-12T03:25:34.864 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-12T03:25:34.869 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-12T03:25:34.870 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-12T03:25:34.872 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:169157fa:::existing_81:head# 2020-03-12T03:25:34.877 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:17731382:::existing_58:head# 2020-03-12T03:25:34.879 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-12T03:25:34.879 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-12T03:25:34.882 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-12T03:25:34.882 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-12T03:25:34.883 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-12T03:25:34.883 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:28566edb:::existing_78:head# 2020-03-12T03:25:34.893 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:2a101182:::existing_20:head# 2020-03-12T03:25:34.899 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-12T03:25:34.900 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-12T03:25:34.901 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-12T03:25:34.905 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-12T03:25:34.905 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-12T03:25:34.905 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-12T03:25:34.909 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-12T03:25:34.910 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-12T03:25:34.910 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-12T03:25:34.912 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-12T03:25:34.913 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:53588912:::existing_9:head# 2020-03-12T03:25:34.913 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-12T03:25:34.914 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-12T03:25:34.914 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-12T03:25:34.915 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-12T03:25:34.920 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:602f83fe:::foo:head# 2020-03-12T03:25:34.920 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-12T03:25:34.921 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:628918ca:::existing_79:head# 2020-03-12T03:25:34.921 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-12T03:25:34.922 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-12T03:25:34.923 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-12T03:25:34.924 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-12T03:25:34.924 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-12T03:25:34.924 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-12T03:25:34.925 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-12T03:25:34.925 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-12T03:25:34.926 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:750ace21:::existing_19:head# 2020-03-12T03:25:34.926 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-12T03:25:34.927 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:79881152:::existing_63:head# 2020-03-12T03:25:34.927 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-12T03:25:34.928 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:81490677:::existing_76:head# 2020-03-12T03:25:34.928 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-12T03:25:34.929 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-12T03:25:34.930 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-12T03:25:34.931 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-12T03:25:34.931 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:92731e71:::existing_89:head# 2020-03-12T03:25:34.931 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:928205cf:::existing_82:head# 2020-03-12T03:25:34.932 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:93be2267:::existing_24:head# 2020-03-12T03:25:34.934 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-12T03:25:34.935 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-12T03:25:34.936 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-12T03:25:34.936 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-12T03:25:34.937 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-12T03:25:34.937 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:a504a533:::existing_90:head# 2020-03-12T03:25:34.938 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:a691337e:::existing_61:head# 2020-03-12T03:25:34.939 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-12T03:25:34.939 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-12T03:25:34.949 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-12T03:25:34.949 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-12T03:25:34.950 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-12T03:25:34.951 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:b343e277:::existing_88:head# 2020-03-12T03:25:34.952 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-12T03:25:34.953 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-12T03:25:34.953 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-12T03:25:34.954 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-12T03:25:34.954 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:bebec877:::existing_74:head# 2020-03-12T03:25:34.955 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-12T03:25:34.956 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:c156033e:::existing_100:head# 2020-03-12T03:25:34.956 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-12T03:25:34.956 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-12T03:25:34.965 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-12T03:25:34.966 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-12T03:25:34.966 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-12T03:25:34.967 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-12T03:25:34.968 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-12T03:25:34.969 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-12T03:25:34.970 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-12T03:25:34.970 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-12T03:25:34.971 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-12T03:25:34.971 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-12T03:25:34.971 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-12T03:25:34.972 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-12T03:25:34.973 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:e255b710:::existing_55:head# 2020-03-12T03:25:34.973 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-12T03:25:34.974 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-12T03:25:34.974 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-12T03:25:34.975 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:f6809456:::existing_11:head# 2020-03-12T03:25:34.975 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-12T03:25:34.976 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-12T03:25:34.976 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-12T03:25:34.978 INFO:tasks.workunit.client.0.smithi180.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-12T03:25:34.978 INFO:tasks.workunit.client.0.smithi180.stderr:Export successful 2020-03-12T03:25:34.978 INFO:tasks.workunit.client.0.smithi180.stdout: marking collection for removal 2020-03-12T03:25:34.978 INFO:tasks.workunit.client.0.smithi180.stdout:setting '_remove' omap key 2020-03-12T03:25:34.979 INFO:tasks.workunit.client.0.smithi180.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-12T03:25:35.034 INFO:tasks.workunit.client.0.smithi180.stderr:Remove successful 2020-03-12T03:25:35.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.16628.out 2020-03-12T03:25:35.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-12T03:25:35.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-12T03:25:35.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-12T03:25:35.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-12T03:25:35.494 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-12T03:25:35.495 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.16628.out 2020-03-12T03:25:36.006 INFO:tasks.workunit.client.0.smithi180.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={21=1}} deleted_pools ) 2020-03-12T03:25:36.007 INFO:tasks.workunit.client.0.smithi180.stdout:Importing pgid 1.0 2020-03-12T03:25:36.007 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-12T03:25:36.007 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.059 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-12T03:25:36.059 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.067 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-12T03:25:36.067 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.075 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-12T03:25:36.075 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.084 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-12T03:25:36.084 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.092 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-12T03:25:36.092 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.100 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-12T03:25:36.101 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.109 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:169157fa:::existing_81:head# 2020-03-12T03:25:36.109 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.117 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:17731382:::existing_58:head# 2020-03-12T03:25:36.117 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.125 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-12T03:25:36.125 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.134 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-12T03:25:36.134 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.142 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-12T03:25:36.142 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.150 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-12T03:25:36.150 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.159 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-12T03:25:36.159 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.188 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:28566edb:::existing_78:head# 2020-03-12T03:25:36.188 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.197 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:2a101182:::existing_20:head# 2020-03-12T03:25:36.198 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.206 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-12T03:25:36.206 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.214 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-12T03:25:36.214 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.222 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-12T03:25:36.223 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.231 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-12T03:25:36.231 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.239 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-12T03:25:36.239 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.248 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-12T03:25:36.248 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.256 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-12T03:25:36.256 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.273 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-12T03:25:36.273 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.290 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-12T03:25:36.290 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.321 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-12T03:25:36.321 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.331 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:53588912:::existing_9:head# 2020-03-12T03:25:36.331 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.340 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-12T03:25:36.340 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.372 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-12T03:25:36.372 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.384 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-12T03:25:36.384 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.392 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-12T03:25:36.392 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.401 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:602f83fe:::foo:head# 2020-03-12T03:25:36.401 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.409 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-12T03:25:36.409 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.417 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:628918ca:::existing_79:head# 2020-03-12T03:25:36.417 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.426 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-12T03:25:36.426 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.434 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-12T03:25:36.434 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.448 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-12T03:25:36.449 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.451 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-12T03:25:36.451 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.459 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-12T03:25:36.459 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.467 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-12T03:25:36.467 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.476 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-12T03:25:36.476 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.505 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-12T03:25:36.505 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.517 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:750ace21:::existing_19:head# 2020-03-12T03:25:36.517 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.526 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-12T03:25:36.526 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.534 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:79881152:::existing_63:head# 2020-03-12T03:25:36.534 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.542 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-12T03:25:36.543 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.551 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:81490677:::existing_76:head# 2020-03-12T03:25:36.551 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.559 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-12T03:25:36.559 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.567 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-12T03:25:36.568 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.576 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-12T03:25:36.576 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.584 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-12T03:25:36.584 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.593 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:92731e71:::existing_89:head# 2020-03-12T03:25:36.593 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.601 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:928205cf:::existing_82:head# 2020-03-12T03:25:36.601 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.609 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:93be2267:::existing_24:head# 2020-03-12T03:25:36.609 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.618 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-12T03:25:36.618 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.647 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-12T03:25:36.647 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.659 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-12T03:25:36.660 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.668 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-12T03:25:36.668 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.676 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-12T03:25:36.676 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.684 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:a504a533:::existing_90:head# 2020-03-12T03:25:36.684 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.726 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:a691337e:::existing_61:head# 2020-03-12T03:25:36.726 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.734 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-12T03:25:36.735 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.743 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-12T03:25:36.743 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.751 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-12T03:25:36.751 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.760 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-12T03:25:36.760 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.808 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-12T03:25:36.808 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.840 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:b343e277:::existing_88:head# 2020-03-12T03:25:36.840 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.843 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-12T03:25:36.843 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.872 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-12T03:25:36.873 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.885 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-12T03:25:36.885 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.893 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-12T03:25:36.894 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.901 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:bebec877:::existing_74:head# 2020-03-12T03:25:36.902 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.910 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-12T03:25:36.910 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.918 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:c156033e:::existing_100:head# 2020-03-12T03:25:36.918 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.927 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-12T03:25:36.927 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.935 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-12T03:25:36.935 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.943 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-12T03:25:36.943 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.952 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-12T03:25:36.952 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.960 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-12T03:25:36.960 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.968 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-12T03:25:36.968 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.977 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-12T03:25:36.977 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:36.985 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-12T03:25:36.985 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.014 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-12T03:25:37.014 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.018 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-12T03:25:37.018 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.027 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-12T03:25:37.027 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.035 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-12T03:25:37.036 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.044 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-12T03:25:37.044 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.052 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-12T03:25:37.052 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.060 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:e255b710:::existing_55:head# 2020-03-12T03:25:37.060 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.068 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-12T03:25:37.069 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.077 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-12T03:25:37.077 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.131 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-12T03:25:37.131 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.143 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:f6809456:::existing_11:head# 2020-03-12T03:25:37.143 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.156 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-12T03:25:37.156 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.161 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-12T03:25:37.161 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.168 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-12T03:25:37.169 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.198 INFO:tasks.workunit.client.0.smithi180.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-12T03:25:37.198 INFO:tasks.workunit.client.0.smithi180.stdout:snapset 0=[]:{} 2020-03-12T03:25:37.210 INFO:tasks.workunit.client.0.smithi180.stdout:write_pg epoch 44 info 1.0( v 44'102 lc 0'0 (27'90,44'102] local-lis/les=43/44 n=101 ec=21/21 lis/c=43/21 les/c/f=44/22/0 sis=43 pruub=8.925691086s) 2020-03-12T03:25:37.219 INFO:tasks.workunit.client.0.smithi180.stdout:Import successful 2020-03-12T03:25:37.745 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-12T03:25:37.745 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:37.745 INFO:tasks.workunit.client.0.smithi180.stdout:reviving divergent 1 2020-03-12T03:25:38.073 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:38.073 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-12T03:25:24.828472+0000 39'101 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:38.074 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:38.074 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:38.075 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:38.088 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-12T03:25:38.089 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:25:38.089 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:25:38.089 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-12T03:25:38.089 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:25:38.089 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-12T03:25:38.090 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:25:38.090 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:25:38.090 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:25:38.090 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:25:38.091 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T03:25:38.091 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T03:25:38.091 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:25:38.091 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:25:38.091 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:25:38.092 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:25:38.092 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:25:38.092 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:25:38.092 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:25:38.092 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:25:38.093 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:25:38.093 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:25:38.093 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:25:38.093 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:25:38.093 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:25:38.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:25:38.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:25:38.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:25:38.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:25:38.095 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:25:38.095 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:25:38.095 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-12T03:25:38.095 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-12T03:25:38.095 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:25:38.096 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-12T03:25:38.096 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-12T03:25:38.096 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:25:38.096 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:25:38.096 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:25:38.097 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T03:25:38.112 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:38.110+0000 7fc8fa9c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:25:38.114 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:38.112+0000 7fc8fa9c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:25:38.115 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:38.113+0000 7fc8fa9c7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:25:38.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-12T03:25:38.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:25:38.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:25:38.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:25:38.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:25:38.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:25:38.431 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:25:38.431 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:25:38.431 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:25:38.431 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:25:38.630 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:38.628+0000 7fc8fa9c7ec0 -1 Falling back to public interface 2020-03-12T03:25:38.775 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 29 last_clean_interval [9,27) [v2:127.0.0.1:6818/32236,v1:127.0.0.1:6819/32236] [v2:127.0.0.1:6820/32236,v1:127.0.0.1:6821/32236] exists,up eb5be901-51a3-4f54-bb09-dd4860cf0f9c 2020-03-12T03:25:38.775 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:25:38.775 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:25:38.776 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:25:38.776 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:25:38.776 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-12T03:25:38.776 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:25:38.776 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:25:38.776 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:25:38.776 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:25:38.777 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:25:38.777 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:25:38.777 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:25:38.777 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:25:39.161 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 29 last_clean_interval [9,27) [v2:127.0.0.1:6818/32236,v1:127.0.0.1:6819/32236] [v2:127.0.0.1:6820/32236,v1:127.0.0.1:6821/32236] exists,up eb5be901-51a3-4f54-bb09-dd4860cf0f9c 2020-03-12T03:25:39.161 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:25:39.161 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:25:39.162 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:25:39.162 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-12T03:25:39.439 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:25:39.436+0000 7fc8fa9c7ec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-12T03:25:59.164 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-12T03:25:59.164 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-12T03:25:59.164 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-12T03:25:59.165 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:25:59.165 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:25:59.165 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:25:59.165 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.1.asok 2020-03-12T03:25:59.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-12T03:25:59.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.1.asok dump_ops_in_flight 2020-03-12T03:25:59.260 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:25:59.260 INFO:tasks.workunit.client.0.smithi180.stdout: "ops": [], 2020-03-12T03:25:59.260 INFO:tasks.workunit.client.0.smithi180.stdout: "num_ops": 0 2020-03-12T03:25:59.261 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:25:59.269 INFO:tasks.workunit.client.0.smithi180.stdout:allowing recovery 2020-03-12T03:25:59.269 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-12T03:25:59.270 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:59.588 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:25:59.588 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-12T03:25:40.637772+0000 44'102 47:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:25:59.588 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:25:59.589 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:25:59.590 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:25:59.598 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-12T03:25:59.598 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-12T03:25:59.718 INFO:tasks.workunit.client.0.smithi180.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-12T03:25:59.726 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-12T03:25:59.726 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-12T03:25:59.847 INFO:tasks.workunit.client.0.smithi180.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-12T03:25:59.855 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-12T03:25:59.855 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-12T03:25:59.977 INFO:tasks.workunit.client.0.smithi180.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-12T03:25:59.984 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-12T03:25:59.985 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-12T03:25:59.985 INFO:tasks.workunit.client.0.smithi180.stdout:reading divergent objects 2020-03-12T03:26:00.304 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:26:00.304 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-12T03:25:40.637772+0000 44'102 47:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:23:26.586426+0000 0'0 2020-03-12T03:23:26.586426+0000 0 2020-03-12T03:26:00.304 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:26:00.304 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:26:00.305 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:26:00.314 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-12T03:26:00.315 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-12T03:26:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:26:00.316 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-12T03:26:00.344 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:26:00.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-12T03:26:00.372 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:26:00.372 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-12T03:26:00.400 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-12T03:26:00.401 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-12T03:26:00.429 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-12T03:26:00.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-12T03:26:00.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-12T03:26:00.456 INFO:tasks.workunit.client.0.smithi180.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-12T03:26:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-12T03:26:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-12T03:26:00.484 INFO:tasks.workunit.client.0.smithi180.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-12T03:26:00.485 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-12T03:26:00.486 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-12T03:26:00.487 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-12T03:26:00.490 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 27'106 (27'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-12T03:23:47.030594+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-12T03:26:00.490 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: keeping 27'106 (27'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-12T03:23:47.030594+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-12T03:26:00.490 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-12T03:26:00.490 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 27'4 first_divergent_update: 27'106 last_divergent_update: 27'106 2020-03-12T03:26:00.490 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (27'4) modify 1:70b3bf12:::existing_4:head by client.4650.0:1 2020-03-12T03:24:40.636940+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-12T03:26:00.491 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 27'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2020-03-12T03:23:47.023683+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.491 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: keeping 27'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2020-03-12T03:23:47.023683+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.491 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-12T03:26:00.491 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 27'103 last_divergent_update: 27'103 2020-03-12T03:26:00.491 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-12T03:26:00.491 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-12T03:26:00.492 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 27'105 (27'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-12T03:23:47.028948+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-12T03:26:00.492 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: keeping 27'105 (27'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-12T03:23:47.028948+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-12T03:26:00.492 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-12T03:26:00.492 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 27'3 first_divergent_update: 27'105 last_divergent_update: 27'105 2020-03-12T03:26:00.492 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-12T03:26:00.492 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-12T03:26:00.492 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 27'106 2020-03-12T03:26:00.493 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 27'105 (27'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-12T03:23:47.028948+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-12T03:26:00.493 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-12T03:26:00.493 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 27'3 <= info.log_tail 27'90 2020-03-12T03:26:00.493 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 27'104 (27'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2020-03-12T03:23:47.027768+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.493 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: keeping 27'104 (27'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2020-03-12T03:23:47.027768+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.493 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-12T03:26:00.493 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 27'2 first_divergent_update: 27'104 last_divergent_update: 27'104 2020-03-12T03:26:00.494 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-12T03:26:00.495 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-12T03:26:00.495 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 27'106 2020-03-12T03:26:00.495 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 27'104 (27'2) modify 1:97db2a0f:::existing_2:head by client.4536.0:1 2020-03-12T03:23:47.027768+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.495 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-12T03:26:00.495 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 27'2 <= info.log_tail 27'90 2020-03-12T03:26:00.495 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 27'102 (27'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2020-03-12T03:23:47.014040+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.496 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: keeping 27'102 (27'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2020-03-12T03:23:47.014040+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.496 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-12T03:26:00.497 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 27'1 first_divergent_update: 27'102 last_divergent_update: 27'102 2020-03-12T03:26:00.497 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-12T03:26:00.497 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-12T03:26:00.497 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 27'106 2020-03-12T03:26:00.497 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 27'102 (27'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2020-03-12T03:23:47.014040+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.497 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-12T03:26:00.497 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 27'1 <= info.log_tail 27'90 2020-03-12T03:26:00.498 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 27'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4526.0:1 2020-03-12T03:23:47.011567+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.498 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 20 _merge_object_divergent_entries: keeping 27'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4526.0:1 2020-03-12T03:23:47.011567+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-12T03:26:00.498 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-12T03:26:00.498 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 27'101 last_divergent_update: 27'101 2020-03-12T03:26:00.498 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-12T03:26:00.498 INFO:tasks.workunit.client.0.smithi180.stdout:td/divergent-priors/osd.1.log:2020-03-12T03:25:24.797+0000 7fb6af440700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-12T03:26:00.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-12T03:26:00.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-12T03:26:00.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-12T03:26:00.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-12T03:26:00.500 INFO:tasks.workunit.client.0.smithi180.stdout:success 2020-03-12T03:26:00.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-12T03:26:00.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.16628.out 2020-03-12T03:26:00.501 INFO:tasks.workunit.client.0.smithi180.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.16628.out': No such file or directory 2020-03-12T03:26:00.501 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-12T03:26:00.501 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-12T03:26:00.501 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-12T03:26:01.024 INFO:tasks.workunit.client.0.smithi180.stderr:pool 'test' does not exist 2020-03-12T03:26:01.032 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-12T03:26:01.033 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:26:01.033 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:26:01.033 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:26:01.033 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:26:01.034 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:26:01.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:26:01.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-12T03:26:01.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-12T03:26:01.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-12T03:26:01.141 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-12T03:26:01.141 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:26:01.142 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:26:01.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:26:01.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:26:01.142 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:26:01.145 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:26:01.146 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-12T03:26:01.146 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-12T03:26:01.146 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-12T03:26:01.147 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-12T03:26:01.147 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-12T03:26:01.148 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-12T03:26:01.149 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:26:01.150 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-12T03:26:01.150 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-12T03:26:01.150 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:26:01.150 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-12T03:26:01.151 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-12T03:26:01.151 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-12T03:26:01.163 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-12T03:26:01.163 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:01.163 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:01.164 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.16628 2020-03-12T03:26:01.165 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-12T03:26:01.165 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-12T03:26:01.165 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-12T03:26:01.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-12T03:26:01.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-12T03:26:01.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-12T03:26:01.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-12T03:26:01.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-12T03:26:01.166 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-12T03:26:01.166 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:26:01.167 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:26:01.167 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:26:01.167 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:26:01.167 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:26:01.168 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:26:01.168 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-12T03:26:01.168 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-12T03:26:01.168 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-12T03:26:01.169 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-12T03:26:01.169 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-12T03:26:01.169 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-12T03:26:01.170 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:26:01.170 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-12T03:26:01.171 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-12T03:26:01.171 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T03:26:01.172 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-12T03:26:01.172 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-12T03:26:01.172 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-12T03:26:01.173 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-12T03:26:01.174 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:01.174 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:01.174 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.16628 2020-03-12T03:26:01.174 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-12T03:26:01.175 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-12T03:26:01.175 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-12T03:26:01.175 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-12T03:26:01.176 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:01.176 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:01.176 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.16628 2020-03-12T03:26:01.177 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-12T03:26:01.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-12T03:26:01.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-12T03:26:01.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-12T03:26:01.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-12T03:26:01.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-12T03:26:01.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-12T03:26:01.179 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-12T03:26:01.179 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-12T03:26:01.179 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-12T03:26:01.180 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-12T03:26:01.180 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-12T03:26:01.180 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:26:01.180 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:26:01.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-12T03:26:01.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-12T03:26:01.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-12T03:26:01.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-12T03:26:01.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-12T03:26:01.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-12T03:26:01.181 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-12T03:26:01.478 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-12T03:26:01.479 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:26:01.479 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:26:01.479 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:26:01.479 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:01.479 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:01.479 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:01.480 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-12T03:26:01.647 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-12T03:26:01.648 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-12T03:26:01.648 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-12T03:26:01.648 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-12T03:26:01.648 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-12T03:26:01.648 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-12T03:26:01.649 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-12T03:26:01.649 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-12T03:26:01.649 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-12T03:26:01.650 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:26:01.650 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:01.650 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:01.650 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-mon.a.asok 2020-03-12T03:26:01.650 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-12T03:26:01.650 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.16628/ceph-mon.a.asok config get fsid 2020-03-12T03:26:01.736 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-12T03:26:01.736 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-12T03:26:01.736 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-12T03:26:01.736 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-12T03:26:01.737 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-12T03:26:01.737 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-12T03:26:01.737 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-12T03:26:01.737 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-12T03:26:01.737 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:26:01.737 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:01.737 INFO:tasks.workunit.client.0.smithi180.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:01.738 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-mon.a.asok 2020-03-12T03:26:01.738 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-12T03:26:01.738 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.16628/ceph-mon.a.asok config get mon_host 2020-03-12T03:26:01.823 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-12T03:26:01.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-12T03:26:01.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-12T03:26:01.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-12T03:26:01.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-12T03:26:01.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-12T03:26:01.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-12T03:26:02.729 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-12T03:26:02.730 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:26:02.730 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:26:02.730 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:26:02.730 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:02.731 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:02.731 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:02.731 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-12T03:26:02.732 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-12T03:26:02.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-12T03:26:02.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-12T03:26:02.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:26:02.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:26:02.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-12T03:26:02.760 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:26:02.760 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-12T03:26:02.760 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:26:02.760 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:26:02.760 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:26:02.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:26:02.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-12T03:26:02.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-12T03:26:02.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:26:02.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:26:02.761 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:26:02.762 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:26:02.762 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:26:02.763 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:26:02.763 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:26:02.763 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:02.764 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:02.764 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:02.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:02.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:26:02.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:26:02.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:26:02.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:26:02.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:26:02.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:26:02.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:26:02.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:02.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:26:02.765 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:26:02.766 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-12T03:26:02.766 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:26:02.767 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c7b93388-ae83-429e-aa88-c49797480a0a 2020-03-12T03:26:02.767 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 c7b93388-ae83-429e-aa88-c49797480a0a' 2020-03-12T03:26:02.767 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:26:02.767 INFO:tasks.workunit.client.0.smithi180.stdout:add osd0 c7b93388-ae83-429e-aa88-c49797480a0a 2020-03-12T03:26:02.787 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDKq2leTCPcLhAA46cklKigsw+FB3kVrMlQJQ== 2020-03-12T03:26:02.788 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDKq2leTCPcLhAA46cklKigsw+FB3kVrMlQJQ=="}' 2020-03-12T03:26:02.788 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c7b93388-ae83-429e-aa88-c49797480a0a -i td/divergent-priors/0/new.json 2020-03-12T03:26:03.120 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:26:03.130 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-12T03:26:03.131 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKq2leTCPcLhAA46cklKigsw+FB3kVrMlQJQ== --osd-uuid c7b93388-ae83-429e-aa88-c49797480a0a 2020-03-12T03:26:03.159 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:03.157+0000 7fd64f535ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:03.161 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:03.159+0000 7fd64f535ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:03.163 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:03.161+0000 7fd64f535ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:03.163 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:03.161+0000 7fd64f535ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:26:03.163 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:03.161+0000 7fd64f535ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:26:03.164 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:03.161+0000 7fd64f535ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-12T03:26:03.203 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:03.201+0000 7fd64f535ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-12T03:26:06.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-12T03:26:06.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:26:06.179 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd0 key to auth repository 2020-03-12T03:26:06.179 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-12T03:26:06.179 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:26:06.630 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.0 2020-03-12T03:26:06.631 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-12T03:26:06.631 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:26:06.631 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:06.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:26:06.632 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:26:06.658 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:06.655+0000 7f871e408ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:06.680 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:06.678+0000 7f871e408ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:06.682 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:06.680+0000 7f871e408ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:06.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-12T03:26:06.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:26:06.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:26:06.987 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:26:06.988 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:26:06.988 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:26:06.988 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:26:06.988 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:26:06.988 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:26:06.988 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:26:07.200 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:07.198+0000 7f871e408ec0 -1 Falling back to public interface 2020-03-12T03:26:07.335 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:26:07.953 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:07.951+0000 7f871e408ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-12T03:26:08.337 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:26:08.337 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:26:08.337 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:26:08.338 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:26:08.338 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:26:08.338 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:26:08.694 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/33731,v1:127.0.0.1:6803/33731] [v2:127.0.0.1:6804/33731,v1:127.0.0.1:6805/33731] exists,up c7b93388-ae83-429e-aa88-c49797480a0a 2020-03-12T03:26:08.694 INFO:tasks.workunit.client.0.smithi180.stdout:add osd1 37f3150e-8872-4e83-8f22-2e6d16cd0dd6 2020-03-12T03:26:08.695 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:26:08.695 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:26:08.695 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:26:08.695 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-12T03:26:08.695 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-12T03:26:08.695 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:26:08.695 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:26:08.696 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-12T03:26:08.696 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:26:08.696 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-12T03:26:08.696 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:26:08.696 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:26:08.696 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:26:08.696 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:26:08.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T03:26:08.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T03:26:08.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:26:08.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:26:08.697 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:26:08.697 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:26:08.697 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:26:08.698 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:26:08.698 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:26:08.698 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:08.698 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:08.698 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:08.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:08.698 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:26:08.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:26:08.699 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:26:08.700 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:26:08.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:26:08.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:26:08.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:26:08.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:08.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:26:08.701 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:26:08.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-12T03:26:08.702 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:26:08.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=37f3150e-8872-4e83-8f22-2e6d16cd0dd6 2020-03-12T03:26:08.702 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 37f3150e-8872-4e83-8f22-2e6d16cd0dd6' 2020-03-12T03:26:08.702 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:26:08.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDQq2lexBFaKhAAnnCjJMIhfFOXq/mlRqcJhw== 2020-03-12T03:26:08.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDQq2lexBFaKhAAnnCjJMIhfFOXq/mlRqcJhw=="}' 2020-03-12T03:26:08.712 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 37f3150e-8872-4e83-8f22-2e6d16cd0dd6 -i td/divergent-priors/1/new.json 2020-03-12T03:26:09.188 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:26:09.198 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-12T03:26:09.199 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQq2lexBFaKhAAnnCjJMIhfFOXq/mlRqcJhw== --osd-uuid 37f3150e-8872-4e83-8f22-2e6d16cd0dd6 2020-03-12T03:26:09.229 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:09.225+0000 7f6dde60dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:09.242 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:09.240+0000 7f6dde60dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:09.244 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:09.242+0000 7f6dde60dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:09.245 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:09.243+0000 7f6dde60dec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:26:09.245 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:09.243+0000 7f6dde60dec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:26:09.245 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:09.243+0000 7f6dde60dec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-12T03:26:09.301 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:09.299+0000 7f6dde60dec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-12T03:26:12.293 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-12T03:26:12.293 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:26:12.294 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-12T03:26:12.294 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:26:12.295 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd1 key to auth repository 2020-03-12T03:26:12.750 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T03:26:12.751 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-12T03:26:12.751 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:12.751 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:26:12.752 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:26:12.752 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:26:12.777 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:12.775+0000 7f8efb2ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:12.779 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:12.777+0000 7f8efb2ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:12.782 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:12.779+0000 7f8efb2ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:13.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-12T03:26:13.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:26:13.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-12T03:26:13.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:26:13.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:26:13.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:26:13.104 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:26:13.104 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:26:13.104 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:26:13.104 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:26:13.297 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:13.295+0000 7f8efb2ffec0 -1 Falling back to public interface 2020-03-12T03:26:13.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:26:13.982 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:13.980+0000 7f8efb2ffec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-12T03:26:14.464 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:26:14.464 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:26:14.464 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:26:14.464 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:26:14.464 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-12T03:26:14.465 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:26:14.813 INFO:tasks.workunit.client.0.smithi180.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34086,v1:127.0.0.1:6811/34086] [v2:127.0.0.1:6812/34086,v1:127.0.0.1:6813/34086] exists,up 37f3150e-8872-4e83-8f22-2e6d16cd0dd6 2020-03-12T03:26:14.813 INFO:tasks.workunit.client.0.smithi180.stdout:add osd2 f41cf40c-d822-4c35-b108-1c2fbc92d04c 2020-03-12T03:26:14.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:26:14.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:26:14.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:26:14.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-12T03:26:14.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-12T03:26:14.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-12T03:26:14.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-12T03:26:14.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-12T03:26:14.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-12T03:26:14.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-12T03:26:14.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:26:14.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:26:14.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:26:14.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:26:14.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-12T03:26:14.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-12T03:26:14.816 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-12T03:26:14.816 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-12T03:26:14.816 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:26:14.816 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-12T03:26:14.816 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:26:14.816 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:26:14.816 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:26:14.817 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:14.817 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:14.817 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:14.817 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:14.817 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:26:14.817 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-12T03:26:14.817 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-12T03:26:14.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:26:14.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:26:14.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:26:14.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:26:14.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:14.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-12T03:26:14.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-12T03:26:14.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-12T03:26:14.819 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-12T03:26:14.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f41cf40c-d822-4c35-b108-1c2fbc92d04c 2020-03-12T03:26:14.819 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 f41cf40c-d822-4c35-b108-1c2fbc92d04c' 2020-03-12T03:26:14.820 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-12T03:26:14.831 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDWq2le8lN4MRAAdhvoui0gEyjbYuttRBeaVg== 2020-03-12T03:26:14.831 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDWq2le8lN4MRAAdhvoui0gEyjbYuttRBeaVg=="}' 2020-03-12T03:26:14.832 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f41cf40c-d822-4c35-b108-1c2fbc92d04c -i td/divergent-priors/2/new.json 2020-03-12T03:26:15.321 INFO:tasks.workunit.client.0.smithi180.stdout:2 2020-03-12T03:26:15.331 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-12T03:26:15.332 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDWq2le8lN4MRAAdhvoui0gEyjbYuttRBeaVg== --osd-uuid f41cf40c-d822-4c35-b108-1c2fbc92d04c 2020-03-12T03:26:15.360 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:15.357+0000 7f94e968aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:15.361 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:15.359+0000 7f94e968aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:15.363 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:15.361+0000 7f94e968aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:15.364 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:15.362+0000 7f94e968aec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:26:15.364 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:15.362+0000 7f94e968aec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:26:15.364 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:15.362+0000 7f94e968aec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-12T03:26:15.398 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:15.396+0000 7f94e968aec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-12T03:26:18.378 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-12T03:26:18.378 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-12T03:26:18.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-12T03:26:18.379 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-12T03:26:18.379 INFO:tasks.workunit.client.0.smithi180.stdout:adding osd2 key to auth repository 2020-03-12T03:26:18.837 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.2 2020-03-12T03:26:18.837 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-12T03:26:18.837 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-12T03:26:18.837 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:18.837 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-12T03:26:18.838 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-12T03:26:18.863 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:18.861+0000 7ff4580caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:18.865 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:18.863+0000 7ff4580caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:18.866 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:18.864+0000 7ff4580caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:19.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-12T03:26:19.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:26:19.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:26:19.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:26:19.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:26:19.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:26:19.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:26:19.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:26:19.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:26:19.192 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:26:19.385 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:19.383+0000 7ff4580caec0 -1 Falling back to public interface 2020-03-12T03:26:19.538 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:26:20.190 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:20.187+0000 7ff4580caec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-12T03:26:20.539 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:26:20.539 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:26:20.540 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:26:20.540 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:26:20.540 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:26:20.540 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:26:20.925 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/34441,v1:127.0.0.1:6819/34441] [v2:127.0.0.1:6820/34441,v1:127.0.0.1:6821/34441] exists,up f41cf40c-d822-4c35-b108-1c2fbc92d04c 2020-03-12T03:26:20.925 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:26:20.926 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:26:20.926 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:26:20.926 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-12T03:26:21.650 INFO:tasks.workunit.client.0.smithi180.stderr:noout is set 2020-03-12T03:26:21.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-12T03:26:22.335 INFO:tasks.workunit.client.0.smithi180.stderr:noin is set 2020-03-12T03:26:22.343 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-12T03:26:23.170 INFO:tasks.workunit.client.0.smithi180.stderr:nodown is set 2020-03-12T03:26:23.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-12T03:26:23.178 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-12T03:26:23.675 INFO:tasks.workunit.client.0.smithi180.stderr:pool 'test' already exists 2020-03-12T03:26:23.683 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-12T03:26:24.684 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-12T03:26:25.364 INFO:tasks.workunit.client.0.smithi180.stderr:set pool 1 size to 3 2020-03-12T03:26:25.372 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-12T03:26:26.057 INFO:tasks.workunit.client.0.smithi180.stderr:set pool 1 min_size to 2 2020-03-12T03:26:26.066 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-12T03:26:26.709 INFO:tasks.workunit.client.0.smithi180.stderr:set pool 1 pg_autoscale_mode to on 2020-03-12T03:26:26.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-12T03:26:26.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:26:26.718 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:26:27.063 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:26:27.063 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:26:27.063 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:26:27.064 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:26:27.064 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:27.064 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:26:27.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-12T03:26:27.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-12T03:26:27.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:27.192 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:26:27.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-03-12T03:26:27.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2020-03-12T03:26:27.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:27.324 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:26:27.451 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542149 2020-03-12T03:26:27.451 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-60129542149' 2020-03-12T03:26:27.452 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:27.452 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-12T03:26:27.452 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:27.453 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:26:27.453 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-12T03:26:27.453 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:27.454 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-12T03:26:27.454 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-12T03:26:27.455 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:26:27.455 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836487 2020-03-12T03:26:27.799 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-12T03:26:27.800 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:26:28.801 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:26:28.801 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:26:29.145 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-12T03:26:29.145 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:29.145 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705669 2020-03-12T03:26:29.145 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:29.146 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:26:29.146 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705669 2020-03-12T03:26:29.147 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:29.147 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705669 2020-03-12T03:26:29.147 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-03-12T03:26:29.148 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 38654705669 2020-03-12T03:26:29.148 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:26:29.514 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705669 -lt 38654705669 2020-03-12T03:26:29.515 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:29.515 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542149 2020-03-12T03:26:29.515 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:29.516 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:26:29.516 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542149 2020-03-12T03:26:29.516 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:29.517 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542149 2020-03-12T03:26:29.517 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542149' 2020-03-12T03:26:29.518 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:26:29.518 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 60129542149 2020-03-12T03:26:29.889 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542149 -lt 60129542149 2020-03-12T03:26:29.889 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-12T03:26:29.889 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:26:29.889 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:26:29.889 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:26:29.890 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:26:29.890 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-12T03:26:29.890 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:26:29.890 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:26:29.890 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:26:29.890 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:26:29.890 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:26:29.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:26:29.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:26:29.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:26:29.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:26:29.979 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:26:30.343 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:26:30.343 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:26:30.343 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:26:30.344 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:26:30.344 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:30.344 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:26:30.472 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-12T03:26:30.473 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-12T03:26:30.473 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:30.473 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:26:30.602 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705672 2020-03-12T03:26:30.602 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705672' 2020-03-12T03:26:30.602 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:30.603 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:26:30.731 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542151 2020-03-12T03:26:30.731 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-60129542151' 2020-03-12T03:26:30.732 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:30.732 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-12T03:26:30.732 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:30.732 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:26:30.733 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-12T03:26:30.733 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:30.734 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-12T03:26:30.734 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-12T03:26:30.734 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:26:30.734 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836490 2020-03-12T03:26:31.076 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-12T03:26:31.076 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:26:32.077 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:26:32.078 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:26:32.427 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-12T03:26:32.427 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:32.427 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705672 2020-03-12T03:26:32.428 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:32.428 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:26:32.429 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705672 2020-03-12T03:26:32.429 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:32.429 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705672 2020-03-12T03:26:32.430 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2020-03-12T03:26:32.430 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 38654705672 2020-03-12T03:26:32.430 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:26:32.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705672 -lt 38654705672 2020-03-12T03:26:32.815 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:32.816 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542151 2020-03-12T03:26:32.816 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:32.816 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:26:32.817 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542151 2020-03-12T03:26:32.817 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:32.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542151 2020-03-12T03:26:32.818 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2020-03-12T03:26:32.818 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 60129542151 2020-03-12T03:26:32.819 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:26:33.175 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542152 -lt 60129542151 2020-03-12T03:26:33.176 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:26:33.176 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:26:33.176 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:26:33.621 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:26:33.621 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:26:33.621 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:26:33.622 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:26:33.622 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:26:33.622 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:26:33.622 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:26:33.622 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:26:33.956 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-12T03:26:33.956 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:26:33.956 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:26:33.957 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:26:34.414 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-12T03:26:34.414 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-12T03:26:34.414 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-12T03:26:34.415 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-12T03:26:34.415 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-12T03:26:34.737 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:26:34.745 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-12T03:26:34.746 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-12T03:26:34.746 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-12T03:26:34.746 INFO:tasks.workunit.client.0.smithi180.stdout:primary and soon to be divergent is 1 2020-03-12T03:26:35.066 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:26:35.066 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-12T03:26:25.198448+0000 0'0 26:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:26:23.482158+0000 0'0 2020-03-12T03:26:23.482158+0000 0 2020-03-12T03:26:35.066 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:26:35.066 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:26:35.067 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:26:35.077 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-12T03:26:35.077 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-12T03:26:35.077 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-12T03:26:35.077 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-12T03:26:35.077 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-12T03:26:35.078 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-12T03:26:35.078 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-12T03:26:35.078 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-12T03:26:35.078 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-12T03:26:35.078 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-12T03:26:35.078 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-12T03:26:35.079 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-12T03:26:35.079 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.079 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-12T03:26:35.079 INFO:tasks.workunit.client.0.smithi180.stdout:writing initial objects 2020-03-12T03:26:35.123 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.124 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-12T03:26:35.206 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.206 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-12T03:26:35.264 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.264 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-12T03:26:35.372 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.373 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-12T03:26:35.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-12T03:26:35.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-12T03:26:35.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.499 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-12T03:26:35.540 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.541 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-12T03:26:35.582 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.582 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-12T03:26:35.624 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.624 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-12T03:26:35.739 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.740 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-12T03:26:35.782 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.782 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-12T03:26:35.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.824 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-12T03:26:35.865 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.866 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-12T03:26:35.948 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:35.948 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-12T03:26:36.023 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-12T03:26:36.065 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.066 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-12T03:26:36.157 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.158 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-12T03:26:36.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-12T03:26:36.282 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.282 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-12T03:26:36.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-12T03:26:36.366 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.366 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-12T03:26:36.449 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.450 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-12T03:26:36.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-12T03:26:36.549 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.549 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-12T03:26:36.634 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.634 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-12T03:26:36.682 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.683 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-12T03:26:36.767 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.767 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-12T03:26:36.849 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.849 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-12T03:26:36.899 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.899 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-12T03:26:36.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.941 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-12T03:26:36.982 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:36.983 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-12T03:26:37.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-12T03:26:37.101 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.101 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-12T03:26:37.149 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.150 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-12T03:26:37.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.191 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-12T03:26:37.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.233 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-12T03:26:37.326 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.326 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-12T03:26:37.374 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.375 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-12T03:26:37.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-12T03:26:37.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.508 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-12T03:26:37.549 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.550 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-12T03:26:37.591 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.591 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-12T03:26:37.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-12T03:26:37.683 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.683 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-12T03:26:37.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-12T03:26:37.816 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.816 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-12T03:26:37.858 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.858 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-12T03:26:37.899 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.900 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-12T03:26:37.985 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:37.986 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-12T03:26:38.211 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.211 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-12T03:26:38.320 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.320 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-12T03:26:38.366 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.367 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-12T03:26:38.408 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.408 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-12T03:26:38.450 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.450 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-12T03:26:38.491 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.492 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-12T03:26:38.587 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.587 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-12T03:26:38.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-12T03:26:38.692 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.692 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-12T03:26:38.742 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.742 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-12T03:26:38.829 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.829 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-12T03:26:38.875 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.875 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-12T03:26:38.962 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:38.963 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-12T03:26:39.008 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.008 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-12T03:26:39.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-12T03:26:39.092 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.092 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-12T03:26:39.133 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.133 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-12T03:26:39.175 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.175 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-12T03:26:39.263 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.263 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-12T03:26:39.325 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.325 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-12T03:26:39.367 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.367 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-12T03:26:39.408 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.409 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-12T03:26:39.489 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.489 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-12T03:26:39.572 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.572 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-12T03:26:39.617 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.617 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-12T03:26:39.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-12T03:26:39.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.717 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-12T03:26:39.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-12T03:26:39.800 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.800 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-12T03:26:39.842 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.842 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-12T03:26:39.923 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.923 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-12T03:26:39.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:39.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-12T03:26:40.008 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.009 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-12T03:26:40.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.050 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-12T03:26:40.190 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.190 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-12T03:26:40.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-12T03:26:40.309 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.317 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-12T03:26:40.350 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.350 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-12T03:26:40.392 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.392 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-12T03:26:40.434 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.434 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-12T03:26:40.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.501 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-12T03:26:40.582 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.582 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-12T03:26:40.625 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.626 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-12T03:26:40.667 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.672 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-12T03:26:40.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.709 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-12T03:26:40.750 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.754 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-12T03:26:40.832 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.833 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-12T03:26:40.924 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.925 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-12T03:26:40.967 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-12T03:26:40.970 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-12T03:26:41.009 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T03:26:41.009 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-12T03:26:41.010 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:26:41.010 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:26:41.010 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:26:41.010 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:26:41.010 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T03:26:41.011 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:26:41.011 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:26:41.011 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:26:41.011 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:26:41.012 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:26:41.066 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:26:41.086 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:26:41.086 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:26:41.086 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:26:41.086 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:26:41.410 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:26:41.410 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:26:41.410 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:26:41.411 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:26:41.411 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:41.411 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:26:41.538 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-12T03:26:41.539 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-12T03:26:41.539 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:41.539 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:26:41.684 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-12T03:26:41.684 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-38654705676' 2020-03-12T03:26:41.684 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:26:41.685 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:26:41.806 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542156 2020-03-12T03:26:41.806 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-38654705676 2-60129542156' 2020-03-12T03:26:41.806 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:41.807 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-12T03:26:41.807 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:41.807 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:26:41.808 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-12T03:26:41.808 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:41.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-12T03:26:41.808 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-12T03:26:41.809 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 21474836494 2020-03-12T03:26:41.825 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:26:42.151 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-12T03:26:42.151 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:42.151 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-12T03:26:42.152 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:42.152 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:26:42.152 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-12T03:26:42.153 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:42.153 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-12T03:26:42.154 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-12T03:26:42.154 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 38654705676 2020-03-12T03:26:42.154 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:26:42.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705676 -lt 38654705676 2020-03-12T03:26:42.497 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:26:42.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542156 2020-03-12T03:26:42.498 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:26:42.498 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:26:42.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542156 2020-03-12T03:26:42.499 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:26:42.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542156 2020-03-12T03:26:42.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2020-03-12T03:26:42.501 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:26:42.501 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 60129542156 2020-03-12T03:26:42.867 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542156 -lt 60129542156 2020-03-12T03:26:42.868 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:26:42.868 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:26:42.868 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:26:43.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:26:43.314 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:26:43.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:26:43.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:26:43.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:26:43.314 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:26:43.315 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:26:43.315 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:26:43.641 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-12T03:26:43.641 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:26:43.642 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:26:43.642 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:26:44.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-12T03:26:44.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-12T03:26:44.103 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-12T03:26:44.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-12T03:26:44.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-12T03:26:44.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-12T03:26:44.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-12T03:26:44.105 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-12T03:26:44.453 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-12T03:26:44.453 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-12T03:26:44.454 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-12T03:26:44.454 INFO:tasks.workunit.client.0.smithi180.stdout:blackholing osds 0 2 2020-03-12T03:26:44.771 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:26:44.771 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-12T03:26:25.198448+0000 28'100 28:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:26:23.482158+0000 0'0 2020-03-12T03:26:23.482158+0000 0 2020-03-12T03:26:44.772 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:26:44.772 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:26:44.773 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:26:44.782 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:26:44.782 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-12T03:26:44.783 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-12T03:26:44.783 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-12T03:26:44.783 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:26:44.783 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:44.783 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:44.783 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.0.asok 2020-03-12T03:26:44.784 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-12T03:26:44.784 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-12T03:26:44.869 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:26:44.870 INFO:tasks.workunit.client.0.smithi180.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-12T03:26:44.870 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:26:44.877 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:26:44.878 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-12T03:26:44.878 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-12T03:26:44.878 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-12T03:26:44.878 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T03:26:44.878 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:44.879 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:44.879 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.2.asok 2020-03-12T03:26:44.879 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-12T03:26:44.879 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-12T03:26:44.964 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:26:44.964 INFO:tasks.workunit.client.0.smithi180.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-12T03:26:44.965 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:26:44.971 INFO:tasks.workunit.client.0.smithi180.stdout:create new divergent objects 2020-03-12T03:26:44.971 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-12T03:26:44.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-12T03:26:44.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-12T03:26:44.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-12T03:26:44.973 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-12T03:26:44.974 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-12T03:26:44.974 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-12T03:26:44.974 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-12T03:26:44.974 INFO:tasks.workunit.client.0.smithi180.stdout:writing divergent objects 2020-03-12T03:26:44.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-12T03:26:44.975 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-12T03:26:44.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-12T03:26:44.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-12T03:26:44.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-12T03:26:44.976 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-12T03:26:44.976 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-12T03:26:44.977 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-12T03:26:44.977 INFO:tasks.workunit.client.0.smithi180.stdout:remove divergent objects 2020-03-12T03:26:44.978 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-12T03:26:44.978 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-12T03:26:44.978 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-12T03:26:44.978 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-12T03:26:44.978 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-12T03:26:44.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-12T03:26:44.979 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-12T03:26:54.980 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-12T03:26:54.985 INFO:tasks.workunit.client.0.smithi180.stderr:rados: no process found 2020-03-12T03:26:54.985 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-12T03:26:54.985 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-12T03:26:54.985 INFO:tasks.workunit.client.0.smithi180.stdout:killing all the osds 2020-03-12T03:26:55.312 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:26:55.312 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-12T03:26:25.198448+0000 28'106 28:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:26:23.482158+0000 0'0 2020-03-12T03:26:23.482158+0000 0 2020-03-12T03:26:55.312 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:26:55.313 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:26:55.314 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:26:55.322 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-12T03:26:55.323 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T03:26:55.323 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T03:26:55.323 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T03:26:55.323 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T03:26:55.323 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T03:26:55.429 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T03:26:55.429 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-12T03:26:55.429 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-12T03:26:55.948 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already down. 2020-03-12T03:26:55.956 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-12T03:26:55.956 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-12T03:26:56.454 INFO:tasks.workunit.client.0.smithi180.stderr:osd.1 is already down. 2020-03-12T03:26:56.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-12T03:26:56.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-12T03:26:57.014 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already down. 2020-03-12T03:26:57.022 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:26:57.022 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-12T03:26:57.555 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already out. 2020-03-12T03:26:57.563 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:26:57.564 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-12T03:26:58.073 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already out. 2020-03-12T03:26:58.081 INFO:tasks.workunit.client.0.smithi180.stdout:bringing up non_divergent 0 2 2020-03-12T03:26:58.082 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-12T03:26:58.082 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-12T03:26:58.403 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:26:58.403 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-12T03:26:25.198448+0000 28'106 28:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-12T03:26:23.482158+0000 0'0 2020-03-12T03:26:23.482158+0000 0 2020-03-12T03:26:58.403 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:26:58.403 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:26:58.405 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:26:58.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:26:58.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-12T03:26:58.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:26:58.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:26:58.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-12T03:26:58.416 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:26:58.417 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-12T03:26:58.417 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:26:58.417 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:26:58.417 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:26:58.417 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:26:58.417 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-12T03:26:58.417 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-12T03:26:58.418 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:26:58.418 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:26:58.418 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:26:58.418 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:26:58.418 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:26:58.418 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:26:58.418 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:26:58.419 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:26:58.419 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:26:58.419 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:58.419 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:26:58.419 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:26:58.419 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:26:58.419 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:26:58.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:26:58.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:26:58.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:58.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:26:58.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:26:58.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-12T03:26:58.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-12T03:26:58.421 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:26:58.421 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-12T03:26:58.421 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.0 2020-03-12T03:26:58.422 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-12T03:26:58.422 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:26:58.422 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:26:58.422 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:26:58.442 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:58.440+0000 7fe60604eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:58.443 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:58.441+0000 7fe60604eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:58.446 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:58.444+0000 7fe60604eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:26:58.763 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-12T03:26:58.763 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:26:58.763 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T03:26:58.763 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:26:58.763 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:26:58.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:26:58.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:26:58.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:26:58.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:26:58.764 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:26:58.962 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:58.959+0000 7fe60604eec0 -1 Falling back to public interface 2020-03-12T03:26:59.107 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:26:59.879 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:26:59.877+0000 7fe60604eec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-12T03:27:00.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:27:00.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:27:00.109 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:27:00.110 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:27:00.110 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T03:27:00.110 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:27:00.456 INFO:tasks.workunit.client.0.smithi180.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/38475,v1:127.0.0.1:6803/38475] [v2:127.0.0.1:6804/38475,v1:127.0.0.1:6805/38475] exists,up c7b93388-ae83-429e-aa88-c49797480a0a 2020-03-12T03:27:00.456 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.2 2020-03-12T03:27:00.456 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:27:00.456 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:27:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:27:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:27:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-12T03:27:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T03:27:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T03:27:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-12T03:27:00.457 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T03:27:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-12T03:27:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T03:27:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T03:27:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T03:27:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T03:27:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-12T03:27:00.458 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-12T03:27:00.459 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T03:27:00.459 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T03:27:00.459 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T03:27:00.459 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T03:27:00.459 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T03:27:00.459 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T03:27:00.459 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T03:27:00.460 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T03:27:00.460 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T03:27:00.460 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:27:00.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T03:27:00.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T03:27:00.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T03:27:00.460 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-12T03:27:00.461 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T03:27:00.462 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-12T03:27:00.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-12T03:27:00.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-12T03:27:00.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-12T03:27:00.462 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-12T03:27:00.481 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:27:00.479+0000 7f08f2b1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:27:00.483 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:27:00.481+0000 7f08f2b1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:27:00.485 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:27:00.482+0000 7f08f2b1cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T03:27:00.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-12T03:27:00.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T03:27:00.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-12T03:27:00.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T03:27:00.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T03:27:00.813 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:27:00.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T03:27:00.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:27:00.814 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:27:00.814 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T03:27:01.005 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:27:01.003+0000 7f08f2b1cec0 -1 Falling back to public interface 2020-03-12T03:27:01.219 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T03:27:01.928 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T03:27:01.926+0000 7f08f2b1cec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-12T03:27:02.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T03:27:02.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T03:27:02.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T03:27:02.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T03:27:02.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-12T03:27:02.221 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T03:27:02.565 INFO:tasks.workunit.client.0.smithi180.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,28) [v2:127.0.0.1:6810/38669,v1:127.0.0.1:6811/38669] [v2:127.0.0.1:6812/38669,v1:127.0.0.1:6813/38669] exists,up f41cf40c-d822-4c35-b108-1c2fbc92d04c 2020-03-12T03:27:02.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-12T03:27:02.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-12T03:27:02.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-12T03:27:02.566 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:27:02.566 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-12T03:27:03.057 INFO:tasks.workunit.client.0.smithi180.stderr:osd.0 is already in. 2020-03-12T03:27:03.065 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-12T03:27:03.065 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-12T03:27:03.563 INFO:tasks.workunit.client.0.smithi180.stderr:osd.2 is already in. 2020-03-12T03:27:03.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T03:27:03.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-12T03:27:03.565 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T03:27:03.566 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T03:27:03.566 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T03:27:03.566 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T03:27:03.566 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T03:27:03.567 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T03:27:03.567 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T03:27:03.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T03:27:03.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T03:27:03.567 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T03:27:03.621 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T03:27:03.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T03:27:03.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T03:27:03.622 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T03:27:03.622 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T03:27:03.965 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-12T03:27:03.965 INFO:tasks.workunit.client.0.smithi180.stderr:1 2020-03-12T03:27:03.965 INFO:tasks.workunit.client.0.smithi180.stderr:2' 2020-03-12T03:27:03.966 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T03:27:03.966 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:27:03.966 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-12T03:27:04.093 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-12T03:27:04.093 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-12T03:27:04.093 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:27:04.093 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-12T03:27:04.183 INFO:tasks.workunit.client.0.smithi180.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-12T03:27:04.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-12T03:27:04.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-12T03:27:04.192 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-12T03:27:04.193 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-12T03:27:04.321 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-12T03:27:04.321 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-12T03:27:04.321 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:27:04.322 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-12T03:27:04.322 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:27:04.322 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-12T03:27:04.323 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-12T03:27:04.323 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:27:04.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-12T03:27:04.324 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-12T03:27:04.324 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:27:04.325 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.0 seq 146028888067 2020-03-12T03:27:04.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-12T03:27:04.668 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:27:05.669 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-12T03:27:05.670 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:27:06.054 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-12T03:27:06.055 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-12T03:27:07.056 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-12T03:27:07.056 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-12T03:27:07.410 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-12T03:27:07.411 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:27:07.411 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-12T03:27:07.411 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:27:07.412 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-12T03:27:07.412 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-12T03:27:07.412 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:27:07.413 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-12T03:27:07.413 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-12T03:27:07.413 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.1 seq 2020-03-12T03:27:07.414 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-12T03:27:07.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 2020-03-12T03:27:07.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705679: unary operator expected 2020-03-12T03:27:07.756 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-12T03:27:07.757 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-12T03:27:07.757 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-12T03:27:07.757 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-12T03:27:07.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-12T03:27:07.758 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-12T03:27:07.758 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-12T03:27:07.759 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-12T03:27:07.759 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-12T03:27:07.759 INFO:tasks.workunit.client.0.smithi180.stdout:waiting osd.2 seq 150323855363 2020-03-12T03:27:08.102 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-12T03:27:08.103 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T03:27:08.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:08.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:08.549 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-12T03:27:08.549 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:08.549 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:08.549 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:08.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:08.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:08.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:08.550 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:08.886 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:08.887 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:08.887 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:08.887 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:09.327 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-12T03:27:09.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-12T03:27:09.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-12T03:27:09.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-12T03:27:09.328 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:09.329 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-12T03:27:09.428 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:09.428 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:09.429 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:09.429 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:09.429 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:09.430 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:09.430 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:09.430 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:09.764 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:09.764 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:09.764 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:09.765 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:10.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:10.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:10.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:10.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:10.209 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:10.210 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:10.210 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:10.210 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:10.210 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:10.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:10.658 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:10.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-12T03:27:10.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:10.659 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-12T03:27:10.859 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:10.859 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:10.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:10.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:10.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:10.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:10.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:10.861 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:11.194 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:11.194 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:11.194 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:11.195 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:11.636 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:11.636 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:11.637 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:11.637 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:11.637 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:11.637 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:11.637 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:11.637 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:11.637 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:12.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:12.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:12.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-12T03:27:12.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:12.094 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-12T03:27:12.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:12.496 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:12.496 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:12.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:12.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:12.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:12.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:12.497 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:12.829 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:12.830 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:12.830 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:12.830 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:13.272 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:13.272 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:13.273 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:13.273 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:13.273 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:13.273 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:13.273 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:13.273 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:13.274 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:13.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:13.714 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:13.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-12T03:27:13.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:13.715 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-12T03:27:14.516 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:14.517 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:14.517 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:14.517 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:14.517 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:14.517 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:14.518 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:14.518 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:14.859 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:14.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:14.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:14.860 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:15.303 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:15.303 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:15.303 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:15.303 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:15.303 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:15.304 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:15.304 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:15.304 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:15.304 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:15.744 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:15.744 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:15.745 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-12T03:27:15.745 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:15.745 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-12T03:27:17.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:17.345 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:17.346 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:17.346 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:17.347 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:17.347 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:17.347 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:17.347 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:17.690 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:17.693 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:17.693 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:17.693 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:18.143 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:18.143 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:18.143 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:18.144 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:18.144 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:18.144 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:18.144 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:18.145 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:18.145 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:18.585 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:18.585 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:18.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-12T03:27:18.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:18.586 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-12T03:27:21.785 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:21.785 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:21.786 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:21.786 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:21.786 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:21.787 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:21.787 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:21.787 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:22.130 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:22.131 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:22.131 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:22.131 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:22.576 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:22.577 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:22.577 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:22.577 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:22.577 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:22.577 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:22.578 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:22.578 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:22.578 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:23.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:23.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:23.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-12T03:27:23.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:23.024 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-12T03:27:29.424 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:29.425 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:29.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:29.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:29.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:29.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:29.426 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:29.427 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:29.768 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:29.769 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:29.769 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:29.769 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:30.211 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:30.211 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:30.211 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:30.212 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:30.212 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:30.212 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:30.212 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:30.212 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:30.212 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:30.669 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:30.670 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:30.670 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-12T03:27:30.670 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-12T03:27:30.670 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-12T03:27:37.970 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-12T03:27:37.970 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T03:27:37.971 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T03:27:37.971 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T03:27:37.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T03:27:37.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T03:27:37.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T03:27:37.972 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T03:27:38.354 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-12T03:27:38.355 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T03:27:38.355 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T03:27:38.355 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T03:27:38.798 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-12T03:27:38.798 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-12T03:27:38.798 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-12T03:27:38.798 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-12T03:27:38.798 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-12T03:27:38.799 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-12T03:27:38.799 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-12T03:27:38.799 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-12T03:27:38.799 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-12T03:27:39.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-12T03:27:39.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-12T03:27:39.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-12T03:27:39.265 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-12T03:27:39.699 INFO:tasks.workunit.client.0.smithi180.stdout:{ 2020-03-12T03:27:39.699 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_fingerprint": "666ac6fe-765d-477b-b801-a1904c8be39c", 2020-03-12T03:27:39.699 INFO:tasks.workunit.client.0.smithi180.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-12T03:27:39.700 INFO:tasks.workunit.client.0.smithi180.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-12T03:27:39.700 INFO:tasks.workunit.client.0.smithi180.stdout: "timestamp": "2020-03-12T03:27:39.696025+0000", 2020-03-12T03:27:39.700 INFO:tasks.workunit.client.0.smithi180.stdout: "tag": "", 2020-03-12T03:27:39.700 INFO:tasks.workunit.client.0.smithi180.stdout: "health": { 2020-03-12T03:27:39.700 INFO:tasks.workunit.client.0.smithi180.stdout: "status": "HEALTH_WARN", 2020-03-12T03:27:39.700 INFO:tasks.workunit.client.0.smithi180.stdout: "checks": { 2020-03-12T03:27:39.700 INFO:tasks.workunit.client.0.smithi180.stdout: "OSDMAP_FLAGS": { 2020-03-12T03:27:39.701 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:27:39.701 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:27:39.701 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-12T03:27:39.701 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 17 2020-03-12T03:27:39.701 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.701 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [], 2020-03-12T03:27:39.701 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: "OSD_DOWN": { 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 osds down", 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.702 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:27:39.703 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.703 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "osd.1 (root=default,host=smithi180) is down" 2020-03-12T03:27:39.703 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.703 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.703 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:27:39.703 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.703 INFO:tasks.workunit.client.0.smithi180.stdout: "PG_DEGRADED": { 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.704 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-12T03:27:39.705 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.705 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.705 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:27:39.705 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.705 INFO:tasks.workunit.client.0.smithi180.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-12T03:27:39.705 INFO:tasks.workunit.client.0.smithi180.stdout: "severity": "HEALTH_WARN", 2020-03-12T03:27:39.705 INFO:tasks.workunit.client.0.smithi180.stdout: "summary": { 2020-03-12T03:27:39.706 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-12T03:27:39.706 INFO:tasks.workunit.client.0.smithi180.stdout: "count": 1 2020-03-12T03:27:39.706 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.706 INFO:tasks.workunit.client.0.smithi180.stdout: "detail": [ 2020-03-12T03:27:39.706 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.706 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "application not enabled on pool 'test'" 2020-03-12T03:27:39.706 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.707 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.707 INFO:tasks.workunit.client.0.smithi180.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-12T03:27:39.707 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.707 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.707 INFO:tasks.workunit.client.0.smithi180.stdout: "muted": false 2020-03-12T03:27:39.707 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.707 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: "mutes": [] 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_first_committed": 1, 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap_last_committed": 1, 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: "monmap": { 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:27:39.708 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:26:01.200232Z", 2020-03-12T03:27:39.709 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:26:01.200232Z", 2020-03-12T03:27:39.709 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release": 15, 2020-03-12T03:27:39.709 INFO:tasks.workunit.client.0.smithi180.stdout: "min_mon_release_name": "octopus", 2020-03-12T03:27:39.709 INFO:tasks.workunit.client.0.smithi180.stdout: "features": { 2020-03-12T03:27:39.709 INFO:tasks.workunit.client.0.smithi180.stdout: "persistent": [ 2020-03-12T03:27:39.709 INFO:tasks.workunit.client.0.smithi180.stdout: "kraken", 2020-03-12T03:27:39.709 INFO:tasks.workunit.client.0.smithi180.stdout: "luminous", 2020-03-12T03:27:39.710 INFO:tasks.workunit.client.0.smithi180.stdout: "mimic", 2020-03-12T03:27:39.710 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap-prune", 2020-03-12T03:27:39.710 INFO:tasks.workunit.client.0.smithi180.stdout: "nautilus", 2020-03-12T03:27:39.710 INFO:tasks.workunit.client.0.smithi180.stdout: "octopus" 2020-03-12T03:27:39.710 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.710 INFO:tasks.workunit.client.0.smithi180.stdout: "optional": [] 2020-03-12T03:27:39.710 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.711 INFO:tasks.workunit.client.0.smithi180.stdout: "mons": [ 2020-03-12T03:27:39.711 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.711 INFO:tasks.workunit.client.0.smithi180.stdout: "rank": 0, 2020-03-12T03:27:39.711 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "a", 2020-03-12T03:27:39.711 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:27:39.711 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.712 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.712 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.712 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115", 2020-03-12T03:27:39.712 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 0 2020-03-12T03:27:39.712 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.712 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.712 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:7115/0", 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: "priority": 0, 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 0 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.713 INFO:tasks.workunit.client.0.smithi180.stdout: "quorum": [ 2020-03-12T03:27:39.714 INFO:tasks.workunit.client.0.smithi180.stdout: 0 2020-03-12T03:27:39.714 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.714 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap": { 2020-03-12T03:27:39.714 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 40, 2020-03-12T03:27:39.714 INFO:tasks.workunit.client.0.smithi180.stdout: "fsid": "3443d04a-69d5-4d26-8bbb-2314c07d4bbe", 2020-03-12T03:27:39.714 INFO:tasks.workunit.client.0.smithi180.stdout: "created": "2020-03-12T03:26:01.942416+0000", 2020-03-12T03:27:39.714 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:27:08.072632+0000", 2020-03-12T03:27:39.715 INFO:tasks.workunit.client.0.smithi180.stdout: "last_up_change": "2020-03-12T03:27:02.026203+0000", 2020-03-12T03:27:39.715 INFO:tasks.workunit.client.0.smithi180.stdout: "last_in_change": "2020-03-12T03:27:03.361745+0000", 2020-03-12T03:27:39.715 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-12T03:27:39.715 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_num": 5800384, 2020-03-12T03:27:39.715 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_set": [ 2020-03-12T03:27:39.715 INFO:tasks.workunit.client.0.smithi180.stdout: "nodown", 2020-03-12T03:27:39.715 INFO:tasks.workunit.client.0.smithi180.stdout: "noin", 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: "noout", 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: "pglog_hardlimit", 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snapdirs", 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: "recovery_deletes", 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: "sortbitwise" 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_version": 7, 2020-03-12T03:27:39.716 INFO:tasks.workunit.client.0.smithi180.stdout: "full_ratio": 0.99000000953674316, 2020-03-12T03:27:39.717 INFO:tasks.workunit.client.0.smithi180.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-12T03:27:39.717 INFO:tasks.workunit.client.0.smithi180.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-12T03:27:39.717 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_snapshot": "", 2020-03-12T03:27:39.717 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_max": 1, 2020-03-12T03:27:39.717 INFO:tasks.workunit.client.0.smithi180.stdout: "max_osd": 3, 2020-03-12T03:27:39.717 INFO:tasks.workunit.client.0.smithi180.stdout: "require_min_compat_client": "jewel", 2020-03-12T03:27:39.718 INFO:tasks.workunit.client.0.smithi180.stdout: "min_compat_client": "jewel", 2020-03-12T03:27:39.719 INFO:tasks.workunit.client.0.smithi180.stdout: "require_osd_release": "octopus", 2020-03-12T03:27:39.719 INFO:tasks.workunit.client.0.smithi180.stdout: "pools": [ 2020-03-12T03:27:39.719 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.719 INFO:tasks.workunit.client.0.smithi180.stdout: "pool": 1, 2020-03-12T03:27:39.719 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_name": "test", 2020-03-12T03:27:39.719 INFO:tasks.workunit.client.0.smithi180.stdout: "create_time": "2020-03-12T03:26:23.432012+0000", 2020-03-12T03:27:39.719 INFO:tasks.workunit.client.0.smithi180.stdout: "flags": 1, 2020-03-12T03:27:39.720 INFO:tasks.workunit.client.0.smithi180.stdout: "flags_names": "hashpspool", 2020-03-12T03:27:39.720 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:27:39.720 INFO:tasks.workunit.client.0.smithi180.stdout: "size": 3, 2020-03-12T03:27:39.720 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 2, 2020-03-12T03:27:39.720 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_rule": 0, 2020-03-12T03:27:39.720 INFO:tasks.workunit.client.0.smithi180.stdout: "object_hash": 2, 2020-03-12T03:27:39.720 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_autoscale_mode": "on", 2020-03-12T03:27:39.721 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num": 32, 2020-03-12T03:27:39.721 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num": 1, 2020-03-12T03:27:39.721 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_placement_num_target": 32, 2020-03-12T03:27:39.721 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_target": 32, 2020-03-12T03:27:39.721 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_num_pending": 32, 2020-03-12T03:27:39.721 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pg_merge_meta": { 2020-03-12T03:27:39.721 INFO:tasks.workunit.client.0.smithi180.stdout: "source_pgid": "0.0", 2020-03-12T03:27:39.722 INFO:tasks.workunit.client.0.smithi180.stdout: "ready_epoch": 0, 2020-03-12T03:27:39.722 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_started": 0, 2020-03-12T03:27:39.722 INFO:tasks.workunit.client.0.smithi180.stdout: "last_epoch_clean": 0, 2020-03-12T03:27:39.722 INFO:tasks.workunit.client.0.smithi180.stdout: "source_version": "0'0", 2020-03-12T03:27:39.722 INFO:tasks.workunit.client.0.smithi180.stdout: "target_version": "0'0" 2020-03-12T03:27:39.722 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.722 INFO:tasks.workunit.client.0.smithi180.stdout: "last_change": "40", 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend": "0", 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "last_force_op_resend_preluminous": "40", 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "auid": 0, 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_mode": "selfmanaged", 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_seq": 0, 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_epoch": 0, 2020-03-12T03:27:39.723 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_snaps": [], 2020-03-12T03:27:39.724 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps": "[]", 2020-03-12T03:27:39.724 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_bytes": 0, 2020-03-12T03:27:39.724 INFO:tasks.workunit.client.0.smithi180.stdout: "quota_max_objects": 0, 2020-03-12T03:27:39.724 INFO:tasks.workunit.client.0.smithi180.stdout: "tiers": [], 2020-03-12T03:27:39.724 INFO:tasks.workunit.client.0.smithi180.stdout: "tier_of": -1, 2020-03-12T03:27:39.724 INFO:tasks.workunit.client.0.smithi180.stdout: "read_tier": -1, 2020-03-12T03:27:39.724 INFO:tasks.workunit.client.0.smithi180.stdout: "write_tier": -1, 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_mode": "none", 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_bytes": 0, 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "target_max_objects": 0, 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_flush_age": 0, 2020-03-12T03:27:39.725 INFO:tasks.workunit.client.0.smithi180.stdout: "cache_min_evict_age": 0, 2020-03-12T03:27:39.726 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profile": "", 2020-03-12T03:27:39.726 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_params": { 2020-03-12T03:27:39.726 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "none" 2020-03-12T03:27:39.726 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.726 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_period": 0, 2020-03-12T03:27:39.726 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_count": 0, 2020-03-12T03:27:39.726 INFO:tasks.workunit.client.0.smithi180.stdout: "use_gmt_hitset": true, 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "min_read_recency_for_promote": 0, 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "min_write_recency_for_promote": 0, 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_grade_decay_rate": 0, 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "hit_set_search_last_n": 0, 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "grade_table": [], 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "stripe_width": 0, 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "expected_num_objects": 0, 2020-03-12T03:27:39.727 INFO:tasks.workunit.client.0.smithi180.stdout: "fast_read": false, 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: "options": {}, 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: "application_metadata": {} 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: "osds": [ 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:27:39.728 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "c7b93388-ae83-429e-aa88-c49797480a0a", 2020-03-12T03:27:39.729 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:27:39.729 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:27:39.729 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:27:39.729 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:27:39.729 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 5, 2020-03-12T03:27:39.729 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 28, 2020-03-12T03:27:39.729 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 34, 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 37, 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 29, 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.730 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6802", 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6803", 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.731 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.732 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.732 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:27:39.732 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.732 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.732 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.732 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6804", 2020-03-12T03:27:39.732 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.733 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.733 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.733 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.733 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6805", 2020-03-12T03:27:39.733 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.733 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.733 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6808", 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.734 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6809", 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.735 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:27:39.736 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.736 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.736 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.736 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6806", 2020-03-12T03:27:39.736 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.736 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.736 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6807", 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38475 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6803/38475", 2020-03-12T03:27:39.737 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6805/38475", 2020-03-12T03:27:39.738 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6809/38475", 2020-03-12T03:27:39.738 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6807/38475", 2020-03-12T03:27:39.738 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:27:39.738 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:27:39.738 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:27:39.738 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.738 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "37f3150e-8872-4e83-8f22-2e6d16cd0dd6", 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 0, 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:27:39.739 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 0, 2020-03-12T03:27:39.740 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 0, 2020-03-12T03:27:39.740 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 9, 2020-03-12T03:27:39.740 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 21, 2020-03-12T03:27:39.740 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 30, 2020-03-12T03:27:39.740 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:27:39.740 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:27:39.740 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.741 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.742 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.743 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:27:39.743 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.743 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.743 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.743 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.743 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:27:39.743 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.744 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.744 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.744 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.744 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:27:39.744 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.744 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.744 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.745 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:27:39.745 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.745 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.745 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.745 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.745 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.746 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.747 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:27:39.747 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.747 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.747 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.747 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.747 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:27:39.747 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 34086 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/34086", 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/34086", 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34086", 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34086", 2020-03-12T03:27:39.748 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:27:39.749 INFO:tasks.workunit.client.0.smithi180.stdout: "exists" 2020-03-12T03:27:39.749 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.749 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.749 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.749 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:27:39.749 INFO:tasks.workunit.client.0.smithi180.stdout: "uuid": "f41cf40c-d822-4c35-b108-1c2fbc92d04c", 2020-03-12T03:27:39.749 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 1, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "in": 1, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 1, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_affinity": 1, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_begin": 14, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "last_clean_end": 28, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 35, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "up_thru": 0, 2020-03-12T03:27:39.750 INFO:tasks.workunit.client.0.smithi180.stdout: "down_at": 31, 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: "lost_at": 0, 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addrs": { 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6810", 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.751 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.752 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.752 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.752 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6811", 2020-03-12T03:27:39.752 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.752 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.752 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.752 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addrs": { 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6812", 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.753 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.754 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.754 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6813", 2020-03-12T03:27:39.754 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.754 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.754 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.754 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.754 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addrs": { 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6816", 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.755 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6817", 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addrs": { 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: "addrvec": [ 2020-03-12T03:27:39.756 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.757 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v2", 2020-03-12T03:27:39.757 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6814", 2020-03-12T03:27:39.757 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.757 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.757 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.757 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "v1", 2020-03-12T03:27:39.757 INFO:tasks.workunit.client.0.smithi180.stdout: "addr": "127.0.0.1:6815", 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: "nonce": 38669 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: "public_addr": "127.0.0.1:6811/38669", 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: "cluster_addr": "127.0.0.1:6813/38669", 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_back_addr": "127.0.0.1:6817/38669", 2020-03-12T03:27:39.758 INFO:tasks.workunit.client.0.smithi180.stdout: "heartbeat_front_addr": "127.0.0.1:6815/38669", 2020-03-12T03:27:39.759 INFO:tasks.workunit.client.0.smithi180.stdout: "state": [ 2020-03-12T03:27:39.759 INFO:tasks.workunit.client.0.smithi180.stdout: "exists", 2020-03-12T03:27:39.759 INFO:tasks.workunit.client.0.smithi180.stdout: "up" 2020-03-12T03:27:39.759 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.759 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.759 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.759 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_xinfo": [ 2020-03-12T03:27:39.760 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.760 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:27:39.760 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:26:55.725398+0000", 2020-03-12T03:27:39.760 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:27:39.761 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:27:39.761 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:27:39.761 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:27:39.761 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:26:08.939690+0000", 2020-03-12T03:27:39.761 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:27:39.761 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.762 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.762 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:27:39.762 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:26:56.242592+0000", 2020-03-12T03:27:39.762 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:27:39.762 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:27:39.762 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:27:39.762 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:26:14.994187+0000", 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: "down_stamp": "2020-03-12T03:26:56.793282+0000", 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_probability": 0, 2020-03-12T03:27:39.763 INFO:tasks.workunit.client.0.smithi180.stdout: "laggy_interval": 0, 2020-03-12T03:27:39.764 INFO:tasks.workunit.client.0.smithi180.stdout: "features": 4540138292836696063, 2020-03-12T03:27:39.764 INFO:tasks.workunit.client.0.smithi180.stdout: "old_weight": 0, 2020-03-12T03:27:39.764 INFO:tasks.workunit.client.0.smithi180.stdout: "last_purged_snaps_scrub": "2020-03-12T03:26:21.146366+0000", 2020-03-12T03:27:39.764 INFO:tasks.workunit.client.0.smithi180.stdout: "dead_epoch": 0 2020-03-12T03:27:39.764 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.765 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap": [], 2020-03-12T03:27:39.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_upmap_items": [], 2020-03-12T03:27:39.765 INFO:tasks.workunit.client.0.smithi180.stdout: "pg_temp": [], 2020-03-12T03:27:39.765 INFO:tasks.workunit.client.0.smithi180.stdout: "primary_temp": [], 2020-03-12T03:27:39.765 INFO:tasks.workunit.client.0.smithi180.stdout: "blacklist": {}, 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: "erasure_code_profiles": { 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: "default": { 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: "k": "2", 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: "m": "2", 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: "plugin": "jerasure", 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: "technique": "reed_sol_van" 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.766 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.767 INFO:tasks.workunit.client.0.smithi180.stdout: "removed_snaps_queue": [], 2020-03-12T03:27:39.767 INFO:tasks.workunit.client.0.smithi180.stdout: "new_removed_snaps": [], 2020-03-12T03:27:39.767 INFO:tasks.workunit.client.0.smithi180.stdout: "new_purged_snaps": [], 2020-03-12T03:27:39.767 INFO:tasks.workunit.client.0.smithi180.stdout: "crush_node_flags": {}, 2020-03-12T03:27:39.767 INFO:tasks.workunit.client.0.smithi180.stdout: "device_class_flags": {} 2020-03-12T03:27:39.767 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.767 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_metadata": [ 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6804/38475,v1:127.0.0.1:6805/38475]", 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:27:39.768 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:27:39.769 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:27:39.769 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:27:39.769 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:27:39.769 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:27:39.769 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:27:39.769 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-12T03:27:39.769 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:27:39.770 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:27:39.771 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:27:39.771 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:27:39.771 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:27:39.771 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:27:39.771 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:27:39.771 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:27:39.771 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6802/38475,v1:127.0.0.1:6803/38475]", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/38475,v1:127.0.0.1:6809/38475]", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/38475,v1:127.0.0.1:6807/38475]", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:27:39.772 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:27:39.773 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:27:39.773 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:27:39.773 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:27:39.773 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:27:39.773 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/0", 2020-03-12T03:27:39.773 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:27:39.773 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/34086,v1:127.0.0.1:6813/34086]", 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:27:39.774 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:27:39.775 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:27:39.775 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:27:39.775 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:27:39.775 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:27:39.775 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:27:39.775 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:27:39.775 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:27:39.776 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:27:39.777 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:27:39.777 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:27:39.777 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:27:39.777 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:27:39.777 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:27:39.777 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:27:39.777 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/34086,v1:127.0.0.1:6811/34086]", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34086,v1:127.0.0.1:6817/34086]", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34086,v1:127.0.0.1:6815/34086]", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:27:39.778 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:27:39.779 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:27:39.779 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:27:39.779 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:27:39.779 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:27:39.779 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:27:39.779 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/1", 2020-03-12T03:27:39.779 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: "arch": "x86_64", 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: "back_addr": "[v2:127.0.0.1:6812/38669,v1:127.0.0.1:6813/38669]", 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: "back_iface": "", 2020-03-12T03:27:39.780 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs": "1", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_db": "0", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_dedicated_wal": "0", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluefs_single_shared_device": "1", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_access_mode": "file", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_block_size": "4096", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_devices": "sda", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-12T03:27:39.781 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-12T03:27:39.782 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_rotational": "1", 2020-03-12T03:27:39.782 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_size": "107374182400", 2020-03-12T03:27:39.782 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_support_discard": "0", 2020-03-12T03:27:39.782 INFO:tasks.workunit.client.0.smithi180.stdout: "bluestore_bdev_type": "hdd", 2020-03-12T03:27:39.782 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_release": "octopus", 2020-03-12T03:27:39.782 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-12T03:27:39.782 INFO:tasks.workunit.client.0.smithi180.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "default_device_class": "hdd", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XGGP", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": "sda", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "distro": "centos", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-12T03:27:39.783 INFO:tasks.workunit.client.0.smithi180.stdout: "distro_version": "8", 2020-03-12T03:27:39.784 INFO:tasks.workunit.client.0.smithi180.stdout: "front_addr": "[v2:127.0.0.1:6810/38669,v1:127.0.0.1:6811/38669]", 2020-03-12T03:27:39.784 INFO:tasks.workunit.client.0.smithi180.stdout: "front_iface": "", 2020-03-12T03:27:39.784 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/38669,v1:127.0.0.1:6817/38669]", 2020-03-12T03:27:39.784 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/38669,v1:127.0.0.1:6815/38669]", 2020-03-12T03:27:39.784 INFO:tasks.workunit.client.0.smithi180.stdout: "hostname": "smithi180", 2020-03-12T03:27:39.784 INFO:tasks.workunit.client.0.smithi180.stdout: "journal_rotational": "1", 2020-03-12T03:27:39.784 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_swap_kb": "0", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "mem_total_kb": "16140884", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "os": "Linux", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_data": "td/divergent-priors/2", 2020-03-12T03:27:39.785 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_objectstore": "bluestore", 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: "rotational": "1" 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_first_committed": 1, 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: "osdmap_last_committed": 40, 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: "crushmap": { 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: "devices": [ 2020-03-12T03:27:39.786 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.787 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:27:39.787 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.0", 2020-03-12T03:27:39.787 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:27:39.787 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.787 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.787 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:27:39.787 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.1", 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd.2", 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: "class": "hdd" 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.788 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: "types": [ 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 0, 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "osd" 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:27:39.789 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "host" 2020-03-12T03:27:39.790 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.790 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.790 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 2, 2020-03-12T03:27:39.790 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "chassis" 2020-03-12T03:27:39.790 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.790 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.790 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 3, 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "rack" 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 4, 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "row" 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.791 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 5, 2020-03-12T03:27:39.792 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pdu" 2020-03-12T03:27:39.792 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.792 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.792 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 6, 2020-03-12T03:27:39.792 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "pod" 2020-03-12T03:27:39.792 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.792 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 7, 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "room" 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 8, 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "datacenter" 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.793 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.794 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 9, 2020-03-12T03:27:39.794 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "zone" 2020-03-12T03:27:39.794 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.794 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.794 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 10, 2020-03-12T03:27:39.794 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "region" 2020-03-12T03:27:39.794 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "root" 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: "buckets": [ 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.795 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -1, 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default", 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:27:39.796 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.797 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:27:39.797 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:27:39.797 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:27:39.797 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.797 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.797 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.797 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.798 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -2, 2020-03-12T03:27:39.798 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "default~hdd", 2020-03-12T03:27:39.798 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 11, 2020-03-12T03:27:39.798 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "root", 2020-03-12T03:27:39.798 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:27:39.798 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:27:39.798 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:27:39.799 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:27:39.799 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.799 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:27:39.799 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:27:39.799 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:27:39.799 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.800 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.800 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.800 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.800 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -3, 2020-03-12T03:27:39.800 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180", 2020-03-12T03:27:39.800 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:27:39.800 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:27:39.801 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:27:39.802 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.802 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.802 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:27:39.802 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:27:39.802 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:27:39.802 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.802 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.803 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:27:39.803 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:27:39.803 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:27:39.803 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.803 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.803 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.803 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "id": -4, 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "name": "smithi180~hdd", 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "type_id": 1, 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "type_name": "host", 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 19206, 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "alg": "straw2", 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "hash": "rjenkins1", 2020-03-12T03:27:39.804 INFO:tasks.workunit.client.0.smithi180.stdout: "items": [ 2020-03-12T03:27:39.805 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.805 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 0, 2020-03-12T03:27:39.805 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:27:39.805 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 0 2020-03-12T03:27:39.805 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.805 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.805 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 1, 2020-03-12T03:27:39.806 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:27:39.806 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 1 2020-03-12T03:27:39.806 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.806 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.806 INFO:tasks.workunit.client.0.smithi180.stdout: "id": 2, 2020-03-12T03:27:39.806 INFO:tasks.workunit.client.0.smithi180.stdout: "weight": 6402, 2020-03-12T03:27:39.806 INFO:tasks.workunit.client.0.smithi180.stdout: "pos": 2 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: "rules": [ 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_id": 0, 2020-03-12T03:27:39.807 INFO:tasks.workunit.client.0.smithi180.stdout: "rule_name": "replicated_rule", 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: "ruleset": 0, 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: "type": 1, 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: "min_size": 1, 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: "max_size": 10, 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: "steps": [ 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "take", 2020-03-12T03:27:39.808 INFO:tasks.workunit.client.0.smithi180.stdout: "item": -1, 2020-03-12T03:27:39.809 INFO:tasks.workunit.client.0.smithi180.stdout: "item_name": "default" 2020-03-12T03:27:39.809 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.809 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.809 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "choose_firstn", 2020-03-12T03:27:39.809 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 0, 2020-03-12T03:27:39.809 INFO:tasks.workunit.client.0.smithi180.stdout: "type": "osd" 2020-03-12T03:27:39.809 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: "op": "emit" 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: ] 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: "tunables": { 2020-03-12T03:27:39.810 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_tries": 0, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_local_fallback_tries": 0, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_total_tries": 50, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_descend_once": 1, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_vary_r": 1, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "chooseleaf_stable": 1, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "straw_calc_version": 1, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "allowed_bucket_algs": 54, 2020-03-12T03:27:39.811 INFO:tasks.workunit.client.0.smithi180.stdout: "profile": "jewel", 2020-03-12T03:27:39.812 INFO:tasks.workunit.client.0.smithi180.stdout: "optimal_tunables": 1, 2020-03-12T03:27:39.812 INFO:tasks.workunit.client.0.smithi180.stdout: "legacy_tunables": 0, 2020-03-12T03:27:39.812 INFO:tasks.workunit.client.0.smithi180.stdout: "minimum_required_version": "jewel", 2020-03-12T03:27:39.812 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables": 1, 2020-03-12T03:27:39.812 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables2": 1, 2020-03-12T03:27:39.812 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v2_rules": 0, 2020-03-12T03:27:39.812 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables3": 1, 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v3_rules": 0, 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v4_buckets": 1, 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: "require_feature_tunables5": 1, 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: "has_v5_rules": 0 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: "choose_args": {} 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.813 INFO:tasks.workunit.client.0.smithi180.stdout: "fsmap": { 2020-03-12T03:27:39.814 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:27:39.814 INFO:tasks.workunit.client.0.smithi180.stdout: "default_fscid": -1, 2020-03-12T03:27:39.814 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": { 2020-03-12T03:27:39.814 INFO:tasks.workunit.client.0.smithi180.stdout: "compat": {}, 2020-03-12T03:27:39.814 INFO:tasks.workunit.client.0.smithi180.stdout: "ro_compat": {}, 2020-03-12T03:27:39.814 INFO:tasks.workunit.client.0.smithi180.stdout: "incompat": { 2020-03-12T03:27:39.814 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_1": "base v0.20", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_2": "client writeable ranges", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_3": "default file layouts on dirs", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_4": "dir inode in separate object", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_5": "mds uses versioned encoding", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_8": "no anchor table", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_9": "file layout v2", 2020-03-12T03:27:39.815 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_10": "snaprealm v2" 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: "feature_flags": { 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: "enable_multiple": false, 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: "ever_enabled_multiple": false 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: "standbys": [], 2020-03-12T03:27:39.816 INFO:tasks.workunit.client.0.smithi180.stdout: "filesystems": [] 2020-03-12T03:27:39.817 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.817 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_first_committed": 1, 2020-03-12T03:27:39.817 INFO:tasks.workunit.client.0.smithi180.stdout: "mdsmap_last_committed": 1, 2020-03-12T03:27:39.817 INFO:tasks.workunit.client.0.smithi180.stdout: "auth": { 2020-03-12T03:27:39.817 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:27:39.817 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 5, 2020-03-12T03:27:39.817 INFO:tasks.workunit.client.0.smithi180.stdout: "num_secrets": 3 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 32, 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_active": 1, 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_unknown": 31, 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osd": 3, 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_sum": { 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:27:39.818 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 48300, 2020-03-12T03:27:39.819 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:27:39.819 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:27:39.819 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:27:39.819 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:27:39.819 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:27:39.819 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:27:39.819 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 100, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 100, 2020-03-12T03:27:39.820 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:27:39.821 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:27:39.822 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:27:39.822 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:27:39.822 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:27:39.822 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:27:39.822 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:27:39.822 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:27:39.822 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:27:39.823 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 0, 2020-03-12T03:27:39.824 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 0, 2020-03-12T03:27:39.825 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:27:39.825 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:27:39.825 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:27:39.825 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:27:39.825 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:27:39.825 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.825 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 16, 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 16, 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 0 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum": { 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:27:39.826 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:27:39.827 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:27:39.827 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:27:39.827 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:27:39.827 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:27:39.827 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:27:39.827 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3178816, 2020-03-12T03:27:39.827 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 32768, 2020-03-12T03:27:39.828 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 15, 2020-03-12T03:27:39.828 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145712, 2020-03-12T03:27:39.828 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393984, 2020-03-12T03:27:39.828 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:27:39.828 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:27:39.828 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867439616, 2020-03-12T03:27:39.828 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:27:39.829 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33554432, 2020-03-12T03:27:39.829 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 400756, 2020-03-12T03:27:39.829 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:27:39.829 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:27:39.829 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:27:39.829 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 15658, 2020-03-12T03:27:39.829 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221209814 2020-03-12T03:27:39.830 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.830 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:27:39.830 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:27:39.830 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:27:39.830 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 0, 2020-03-12T03:27:39.830 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:27:39.830 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:27:39.831 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.832 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:27:39.832 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:27:39.832 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.832 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_sum_by_class": { 2020-03-12T03:27:39.832 INFO:tasks.workunit.client.0.smithi180.stdout: "hdd": { 2020-03-12T03:27:39.832 INFO:tasks.workunit.client.0.smithi180.stdout: "up_from": 0, 2020-03-12T03:27:39.832 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 0, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pgs": 3, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "num_osds": 3, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_osds": 3, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "num_per_pool_omap_osds": 3, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "kb": 314572800, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used": 3178816, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_data": 32768, 2020-03-12T03:27:39.833 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_omap": 15, 2020-03-12T03:27:39.834 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_used_meta": 3145712, 2020-03-12T03:27:39.834 INFO:tasks.workunit.client.0.smithi180.stdout: "kb_avail": 311393984, 2020-03-12T03:27:39.834 INFO:tasks.workunit.client.0.smithi180.stdout: "statfs": { 2020-03-12T03:27:39.834 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 322122547200, 2020-03-12T03:27:39.834 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 318867439616, 2020-03-12T03:27:39.834 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 3221225472, 2020-03-12T03:27:39.834 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 33554432, 2020-03-12T03:27:39.835 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 400756, 2020-03-12T03:27:39.835 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:27:39.835 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:27:39.835 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:27:39.835 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 15658, 2020-03-12T03:27:39.835 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 3221209814 2020-03-12T03:27:39.835 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: "hb_peers": [], 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: "snap_trim_queue_len": 0, 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: "num_snap_trimming": 0, 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shards_repaired": 0, 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: "op_queue_age_hist": { 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: "histogram": [], 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: "upper_bound": 1 2020-03-12T03:27:39.836 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.837 INFO:tasks.workunit.client.0.smithi180.stdout: "perf_stat": { 2020-03-12T03:27:39.837 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ms": 0, 2020-03-12T03:27:39.837 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ms": 0, 2020-03-12T03:27:39.837 INFO:tasks.workunit.client.0.smithi180.stdout: "commit_latency_ns": 0, 2020-03-12T03:27:39.837 INFO:tasks.workunit.client.0.smithi180.stdout: "apply_latency_ns": 0 2020-03-12T03:27:39.837 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.837 INFO:tasks.workunit.client.0.smithi180.stdout: "alerts": [], 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: "network_ping_times": [] 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: "pool_stats": [ 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: "poolid": 1, 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg": 32, 2020-03-12T03:27:39.838 INFO:tasks.workunit.client.0.smithi180.stdout: "stat_sum": { 2020-03-12T03:27:39.839 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes": 48300, 2020-03-12T03:27:39.839 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects": 100, 2020-03-12T03:27:39.839 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_clones": 0, 2020-03-12T03:27:39.839 INFO:tasks.workunit.client.0.smithi180.stdout: "num_object_copies": 300, 2020-03-12T03:27:39.839 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing_on_primary": 0, 2020-03-12T03:27:39.839 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_missing": 0, 2020-03-12T03:27:39.839 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_degraded": 100, 2020-03-12T03:27:39.840 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_misplaced": 0, 2020-03-12T03:27:39.840 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_unfound": 0, 2020-03-12T03:27:39.840 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_dirty": 100, 2020-03-12T03:27:39.840 INFO:tasks.workunit.client.0.smithi180.stdout: "num_whiteouts": 0, 2020-03-12T03:27:39.840 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read": 0, 2020-03-12T03:27:39.840 INFO:tasks.workunit.client.0.smithi180.stdout: "num_read_kb": 0, 2020-03-12T03:27:39.840 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write": 100, 2020-03-12T03:27:39.841 INFO:tasks.workunit.client.0.smithi180.stdout: "num_write_kb": 100, 2020-03-12T03:27:39.841 INFO:tasks.workunit.client.0.smithi180.stdout: "num_scrub_errors": 0, 2020-03-12T03:27:39.841 INFO:tasks.workunit.client.0.smithi180.stdout: "num_shallow_scrub_errors": 0, 2020-03-12T03:27:39.841 INFO:tasks.workunit.client.0.smithi180.stdout: "num_deep_scrub_errors": 0, 2020-03-12T03:27:39.841 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_recovered": 0, 2020-03-12T03:27:39.841 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_recovered": 0, 2020-03-12T03:27:39.841 INFO:tasks.workunit.client.0.smithi180.stdout: "num_keys_recovered": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_omap": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_hit_set_archive": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_bytes_hit_set_archive": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_kb": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_kb": 0, 2020-03-12T03:27:39.842 INFO:tasks.workunit.client.0.smithi180.stdout: "num_promote": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_high": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_flush_mode_low": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_some": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_evict_mode_full": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_pinned": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_legacy_snapsets": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_large_omap_objects": 0, 2020-03-12T03:27:39.843 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_manifest": 0, 2020-03-12T03:27:39.844 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_bytes": 0, 2020-03-12T03:27:39.844 INFO:tasks.workunit.client.0.smithi180.stdout: "num_omap_keys": 0, 2020-03-12T03:27:39.844 INFO:tasks.workunit.client.0.smithi180.stdout: "num_objects_repaired": 0 2020-03-12T03:27:39.844 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.844 INFO:tasks.workunit.client.0.smithi180.stdout: "store_stats": { 2020-03-12T03:27:39.844 INFO:tasks.workunit.client.0.smithi180.stdout: "total": 0, 2020-03-12T03:27:39.844 INFO:tasks.workunit.client.0.smithi180.stdout: "available": 0, 2020-03-12T03:27:39.845 INFO:tasks.workunit.client.0.smithi180.stdout: "internally_reserved": 0, 2020-03-12T03:27:39.845 INFO:tasks.workunit.client.0.smithi180.stdout: "allocated": 19529728, 2020-03-12T03:27:39.845 INFO:tasks.workunit.client.0.smithi180.stdout: "data_stored": 143228, 2020-03-12T03:27:39.845 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed": 0, 2020-03-12T03:27:39.845 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_allocated": 0, 2020-03-12T03:27:39.845 INFO:tasks.workunit.client.0.smithi180.stdout: "data_compressed_original": 0, 2020-03-12T03:27:39.845 INFO:tasks.workunit.client.0.smithi180.stdout: "omap_allocated": 0, 2020-03-12T03:27:39.846 INFO:tasks.workunit.client.0.smithi180.stdout: "internal_metadata": 0 2020-03-12T03:27:39.846 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.846 INFO:tasks.workunit.client.0.smithi180.stdout: "log_size": 16, 2020-03-12T03:27:39.846 INFO:tasks.workunit.client.0.smithi180.stdout: "ondisk_log_size": 16, 2020-03-12T03:27:39.846 INFO:tasks.workunit.client.0.smithi180.stdout: "up": 2, 2020-03-12T03:27:39.846 INFO:tasks.workunit.client.0.smithi180.stdout: "acting": 2, 2020-03-12T03:27:39.846 INFO:tasks.workunit.client.0.smithi180.stdout: "num_store_stats": 3 2020-03-12T03:27:39.847 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.847 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.847 INFO:tasks.workunit.client.0.smithi180.stdout: "osd_stats": [ 2020-03-12T03:27:39.847 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.847 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:27:39.847 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 146028888074 2020-03-12T03:27:39.847 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 38654705679 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: "seq": 150323855370 2020-03-12T03:27:39.848 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_state": [ 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: "state": "active+undersized+degraded", 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 1 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.849 INFO:tasks.workunit.client.0.smithi180.stdout: "state": "unknown", 2020-03-12T03:27:39.850 INFO:tasks.workunit.client.0.smithi180.stdout: "num": 31 2020-03-12T03:27:39.850 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.850 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.850 INFO:tasks.workunit.client.0.smithi180.stdout: "num_pg_by_osd": [ 2020-03-12T03:27:39.850 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.850 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 2, 2020-03-12T03:27:39.850 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 1, 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 0, 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 0, 2020-03-12T03:27:39.851 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: { 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: "osd": 0, 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: "num_primary_pg": 1, 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: "num_acting_pg": 1, 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: "num_up_not_acting_pg": 0 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.852 INFO:tasks.workunit.client.0.smithi180.stdout: ], 2020-03-12T03:27:39.853 INFO:tasks.workunit.client.0.smithi180.stdout: "purged_snaps": [], 2020-03-12T03:27:39.853 INFO:tasks.workunit.client.0.smithi180.stdout: "servicemap": { 2020-03-12T03:27:39.853 INFO:tasks.workunit.client.0.smithi180.stdout: "epoch": 1, 2020-03-12T03:27:39.853 INFO:tasks.workunit.client.0.smithi180.stdout: "modified": "2020-03-12T03:26:01.942805+0000", 2020-03-12T03:27:39.853 INFO:tasks.workunit.client.0.smithi180.stdout: "services": {} 2020-03-12T03:27:39.853 INFO:tasks.workunit.client.0.smithi180.stdout: }, 2020-03-12T03:27:39.853 INFO:tasks.workunit.client.0.smithi180.stdout: "paxos": { 2020-03-12T03:27:39.854 INFO:tasks.workunit.client.0.smithi180.stdout: "first_committed": 1, 2020-03-12T03:27:39.854 INFO:tasks.workunit.client.0.smithi180.stdout: "last_committed": 306, 2020-03-12T03:27:39.854 INFO:tasks.workunit.client.0.smithi180.stdout: "last_pn": 0, 2020-03-12T03:27:39.854 INFO:tasks.workunit.client.0.smithi180.stdout: "accepted_pn": 0 2020-03-12T03:27:39.854 INFO:tasks.workunit.client.0.smithi180.stdout: } 2020-03-12T03:27:39.854 INFO:tasks.workunit.client.0.smithi180.stdout:} 2020-03-12T03:27:39.855 INFO:tasks.workunit.client.0.smithi180.stderr:report 4038131168 2020-03-12T03:27:39.855 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-12T03:27:39.856 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-12T03:27:39.856 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-12T03:27:39.856 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-12T03:27:39.856 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-12T03:27:39.856 INFO:tasks.workunit.client.0.smithi180.stdout:writing non-divergent object existing_4 2020-03-12T03:27:40.029 INFO:tasks.workunit.client.0.smithi180.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-12T03:27:40.030 INFO:tasks.workunit.client.0.smithi180.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.030 INFO:tasks.workunit.client.0.smithi180.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.030 INFO:tasks.workunit.client.0.smithi180.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.030 INFO:tasks.workunit.client.0.smithi180.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.030 INFO:tasks.workunit.client.0.smithi180.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.030 INFO:tasks.workunit.client.0.smithi180.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.030 INFO:tasks.workunit.client.0.smithi180.stdout:1.19 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.031 INFO:tasks.workunit.client.0.smithi180.stdout:1.18 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.031 INFO:tasks.workunit.client.0.smithi180.stdout:1.17 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.031 INFO:tasks.workunit.client.0.smithi180.stdout:1.16 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.031 INFO:tasks.workunit.client.0.smithi180.stdout:1.15 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.031 INFO:tasks.workunit.client.0.smithi180.stdout:1.14 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.031 INFO:tasks.workunit.client.0.smithi180.stdout:1.13 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.031 INFO:tasks.workunit.client.0.smithi180.stdout:1.12 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.032 INFO:tasks.workunit.client.0.smithi180.stdout:1.11 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.032 INFO:tasks.workunit.client.0.smithi180.stdout:1.10 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.032 INFO:tasks.workunit.client.0.smithi180.stdout:1.5 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.032 INFO:tasks.workunit.client.0.smithi180.stdout:1.4 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.032 INFO:tasks.workunit.client.0.smithi180.stdout:1.3 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.032 INFO:tasks.workunit.client.0.smithi180.stdout:1.2 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.032 INFO:tasks.workunit.client.0.smithi180.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-12T03:27:06.105203+0000 28'100 38:121 [0,2] 0 [0,2] 0 0'0 2020-03-12T03:26:23.482158+0000 0'0 2020-03-12T03:26:23.482158+0000 0 2020-03-12T03:27:40.033 INFO:tasks.workunit.client.0.smithi180.stdout:1.1 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.033 INFO:tasks.workunit.client.0.smithi180.stdout:1.6 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.033 INFO:tasks.workunit.client.0.smithi180.stdout:1.7 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.033 INFO:tasks.workunit.client.0.smithi180.stdout:1.8 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.033 INFO:tasks.workunit.client.0.smithi180.stdout:1.9 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.033 INFO:tasks.workunit.client.0.smithi180.stdout:1.a 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.033 INFO:tasks.workunit.client.0.smithi180.stdout:1.b 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.034 INFO:tasks.workunit.client.0.smithi180.stdout:1.c 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.034 INFO:tasks.workunit.client.0.smithi180.stdout:1.d 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.034 INFO:tasks.workunit.client.0.smithi180.stdout:1.e 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.034 INFO:tasks.workunit.client.0.smithi180.stdout:1.f 0 0 0 0 0 0 0 0 0 0 unknown 2020-03-12T03:27:08.072632+0000 0'0 0:0 [] -1 [] -1 0'0 2020-03-12T03:27:08.072632+0000 0'0 2020-03-12T03:27:08.072632+0000 0 2020-03-12T03:27:40.034 INFO:tasks.workunit.client.0.smithi180.stdout: 2020-03-12T03:27:40.034 INFO:tasks.workunit.client.0.smithi180.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-12T03:27:40.035 INFO:tasks.workunit.client.0.smithi180.stderr:dumped pgs 2020-03-12T03:27:40.039 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-12T06:20:41.091 INFO:tasks.workunit.client.0.smithi180.stderr:Terminated 2020-03-12T06:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:1: TEST_divergent_3: teardown td/divergent-priors 1 2020-03-12T06:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-12T06:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2020-03-12T06:20:41.102 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-12T06:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-12T06:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-12T06:20:41.103 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-12T06:20:41.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-12T06:20:41.104 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-12T06:20:41.201 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-12T06:20:41.201 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-12T06:20:41.202 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-12T06:20:41.202 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-12T06:20:41.204 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-12T06:20:41.204 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-12T06:20:41.204 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-12T06:20:41.207 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T06:20:41.208 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-12T06:20:41.208 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-12T06:20:41.208 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-12T06:20:41.209 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-12T06:20:41.210 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2020-03-12T06:20:41.210 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2020-03-12T06:20:41.211 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2020-03-12T06:20:41.211 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/divergent-priors/mgr.x.log td/divergent-priors/mon.a.log td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log /home/ubuntu/cephtest/archive/log 2020-03-12T06:20:41.212 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-12T06:20:41.221 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-12T06:20:41.222 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T06:20:41.222 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T06:20:41.222 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.16628 2020-03-12T06:20:41.224 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-12T06:20:41.224 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-12T06:20:41.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-12T06:20:41.224 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-12T06:20:41.225 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-12T06:20:41.225 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-12T06:20:41.225 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-12T06:20:41.225 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-12T06:20:41.225 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-12T06:20:41.225 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-12T06:20:41.226 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-12T06:20:41.226 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-12T06:20:41.226 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-12T06:20:41.226 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-12T06:20:41.284 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-12T06:20:41.284 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-12T06:20:41.285 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-12T06:20:41.285 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-12T06:20:41.285 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-12T07:10:41.378 INFO:tasks.workunit.client.0.smithi180.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2020-03-12T07:10:41.390 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids= 2020-03-12T07:10:41.390 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-12T07:10:41.391 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-12T07:10:41.391 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T07:10:41.391 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T08:00:41.500 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test == 0 2020-03-12T08:00:41.511 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1586: test: ==: unary operator expected 2020-03-12T08:00:41.511 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-12T08:00:41.511 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-12T08:00:41.511 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-12T08:00:41.512 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-12T08:00:41.512 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-12T08:00:41.512 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-12T08:00:41.512 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-12T08:50:41.613 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean= 2020-03-12T08:50:41.624 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-12T08:50:41.625 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-12T08:50:41.625 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-12T09:40:41.721 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test = 2020-03-12T09:40:41.730 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-12T09:40:41.730 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-12T09:40:41.730 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-12T09:40:41.730 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-12T09:40:41.731 INFO:tasks.workunit.client.0.smithi180.stdout:delay recovery 2020-03-12T10:30:41.821 INFO:tasks.workunit.client.0.smithi180.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2020-03-12T10:30:41.831 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-12T10:30:41.832 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-12T10:30:41.832 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-12T10:30:41.833 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-12T10:30:41.833 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T10:30:41.833 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T10:30:41.833 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T10:30:41.833 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.0.asok 2020-03-12T10:30:41.834 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-12T10:30:41.834 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.0.asok set_recovery_delay 100000 2020-03-12T10:30:41.909 INFO:tasks.workunit.client.0.smithi180.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2020-03-12T10:30:41.917 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-12T10:30:41.918 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-12T10:30:41.918 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-12T10:30:41.918 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-12T10:30:41.918 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T10:30:41.919 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T10:30:41.919 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T10:30:41.919 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.2.asok 2020-03-12T10:30:41.919 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-12T10:30:41.919 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.2.asok set_recovery_delay 100000 2020-03-12T10:30:41.989 INFO:tasks.workunit.client.0.smithi180.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2020-03-12T10:30:41.996 INFO:tasks.workunit.client.0.smithi180.stdout:revive divergent 1 2020-03-12T10:30:41.996 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-12T10:30:41.997 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-12T11:20:42.089 INFO:tasks.workunit.client.0.smithi180.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2020-03-12T11:20:42.101 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-12T12:10:42.203 INFO:tasks.workunit.client.0.smithi180.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2020-03-12T12:10:42.216 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-12T12:10:42.216 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-12T12:10:42.216 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-12T12:10:42.217 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-12T12:10:42.217 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-12T12:10:42.217 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-12T12:10:42.217 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-12T12:10:42.217 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-12T12:10:42.217 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-12T12:10:42.218 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-12T12:10:42.218 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-12T12:10:42.218 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-12T12:10:42.218 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-12T12:10:42.218 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-12T12:10:42.218 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-12T12:10:42.218 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-12T12:10:42.219 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-12T12:10:42.219 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-12T12:10:42.219 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-12T12:10:42.219 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T12:10:42.219 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T12:10:42.219 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T12:10:42.219 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-12T12:10:42.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-12T12:10:42.221 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-12T12:10:42.222 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-12T12:10:42.222 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=3443d04a-69d5-4d26-8bbb-2314c07d4bbe --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.16628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-12T12:10:42.222 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-12T12:10:42.222 INFO:tasks.workunit.client.0.smithi180.stderr:cat: td/divergent-priors/1/whoami: No such file or directory 2020-03-12T12:10:42.222 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = '' ']' 2020-03-12T12:10:42.223 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: return 1 2020-03-12T12:10:42.223 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-12T12:10:42.223 INFO:tasks.workunit.client.0.smithi180.stdout:start osd.1 2020-03-12T12:10:42.245 INFO:tasks.workunit.client.0.smithi180.stderr:2020-03-12T12:10:42.242+0000 7f3b9bab0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-12T12:10:47.219 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-12T12:10:47.220 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-12T12:10:47.220 INFO:tasks.workunit.client.0.smithi180.stdout:delay recovery divergent 2020-03-12T13:00:42.250 INFO:tasks.workunit.client.0.smithi180.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-03-12T13:00:47.325 INFO:tasks.workunit.client.0.smithi180.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2020-03-12T13:00:47.336 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-12T13:00:47.337 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-12T13:00:47.337 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-12T13:00:47.337 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-12T13:00:47.337 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-12T13:00:47.338 INFO:tasks.workunit.client.0.smithi180.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.16628 2020-03-12T13:00:47.338 INFO:tasks.workunit.client.0.smithi180.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.16628/ceph-osd.1.asok 2020-03-12T13:00:47.338 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-12T13:00:47.338 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.16628/ceph-osd.1.asok set_recovery_delay 100000 2020-03-12T13:00:47.412 INFO:tasks.workunit.client.0.smithi180.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2020-03-12T13:00:47.420 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-12T13:50:47.513 INFO:tasks.workunit.client.0.smithi180.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2020-03-12T13:50:47.524 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-12T13:50:47.524 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-12T13:50:47.524 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-12T13:50:47.525 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-12T13:50:47.525 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-12T13:50:47.525 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T13:50:47.525 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-12T13:50:47.525 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T13:50:47.525 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T13:50:47.525 INFO:tasks.workunit.client.0.smithi180.stdout:0 2020-03-12T14:40:47.621 INFO:tasks.workunit.client.0.smithi180.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2020-03-12T14:40:47.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-12T14:40:48.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-12T14:40:48.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-12T14:40:48.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-12T14:40:48.633 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-12T14:40:48.634 INFO:tasks.workunit.client.0.smithi180.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-12T14:40:48.634 INFO:tasks.workunit.client.0.smithi180.stdout:1 2020-03-12T15:06:43.381 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-12T15:06:43.389 DEBUG:teuthology.task.console_log:Killing console logger for smithi180 2020-03-12T15:06:43.389 DEBUG:teuthology.task.console_log:Killing console logger for smithi180 2020-03-12T15:06:43.390 DEBUG:teuthology.exit:Finished running handlers