2020-03-27T13:50:36.016 INFO:root:teuthology version: 1.0.0-b3a6b1a 2020-03-27T13:50:36.017 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.9IK6Ce.tmp -- --verbose --lock --description rados/singleton/{all/resolve_stuck_peering.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_16.04.yaml}} --name kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894917 2020-03-27T13:50:36.046 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:50:36.097 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894917 branch: nautilus-kefu2-testing-2020-03-27-1709 description: rados/singleton/{all/resolve_stuck_peering.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_16.04.yaml}} email: tchaikov@gmail.com first_in_suite: false job_id: '4894917' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '16.04' overrides: admin_socket: branch: nautilus-kefu2-testing-2020-03-27-1709 ceph: conf: global: ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: nautilus-kefu2-testing-2020-03-27-1709 sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - - osd.0 - osd.1 - osd.2 - client.0 sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a suite: rados suite_branch: nautilus-kefu2-testing-2020-03-27-1709 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a tasks: - install: null - ceph: fs: xfs log-whitelist: - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - resolve_stuck_peering: null teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2759 2020-03-27T13:50:36.097 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa; will attempt to use it 2020-03-27T13:50:36.098 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-1709/qa/tasks 2020-03-27T13:50:36.098 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-27T13:50:36.100 INFO:teuthology.task.internal:Checking packages... 2020-03-27T13:50:36.113 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'e9aaac6a13136ef54c1c1edf22f45ec35b28b19a' 2020-03-27T13:50:36.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-27T13:50:36.114 INFO:teuthology.packaging:ref: None 2020-03-27T13:50:36.114 INFO:teuthology.packaging:tag: None 2020-03-27T13:50:36.114 INFO:teuthology.packaging:branch: nautilus-kefu2-testing-2020-03-27-1709 2020-03-27T13:50:36.114 INFO:teuthology.packaging:sha1: e9aaac6a13136ef54c1c1edf22f45ec35b28b19a 2020-03-27T13:50:36.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&ref=nautilus-kefu2-testing-2020-03-27-1709 2020-03-27T13:50:36.338 INFO:teuthology.task.internal:Found packages for ceph version 14.2.8-299-ge9aaac6-1xenial 2020-03-27T13:50:36.338 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-27T13:50:36.345 INFO:teuthology.task.internal:no buildpackages task found 2020-03-27T13:50:36.345 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-27T13:50:36.358 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-27T13:50:36.358 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:50:36.406 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894917', 'machine_type': 'smithi'} 2020-03-27T13:50:36.422 DEBUG:teuthology.lock.ops:locked ubuntu@smithi135.front.sepia.ceph.com, ubuntu@smithi196.front.sepia.ceph.com 2020-03-27T13:50:36.493 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-27T13:50:37.221 INFO:teuthology.provision.fog.smithi135:Scheduling deploy of ubuntu 16.04 2020-03-27T13:50:37.230 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of ubuntu 16.04 2020-03-27T13:50:37.594 INFO:teuthology.orchestra.console:Power off smithi135 2020-03-27T13:50:37.594 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-27T13:50:37.628 INFO:teuthology.orchestra.console:Power off smithi196 2020-03-27T13:50:37.628 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-27T13:50:37.637 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:37.653 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:41.665 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:41.677 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:45.777 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:45.791 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:49.892 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:49.901 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:50.102 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2020-03-27T13:50:50.113 INFO:teuthology.orchestra.console:Power off for smithi196 completed 2020-03-27T13:50:50.203 INFO:teuthology.orchestra.console:Power on smithi135 2020-03-27T13:50:50.203 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-27T13:50:50.214 INFO:teuthology.orchestra.console:Power on smithi196 2020-03-27T13:50:50.214 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-27T13:50:50.230 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:50.242 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:54.264 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:54.272 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:54.475 INFO:teuthology.orchestra.console:Power on for smithi135 completed 2020-03-27T13:50:54.575 INFO:teuthology.provision.fog.smithi135:Waiting for deploy to finish 2020-03-27T13:50:58.381 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-27T13:50:58.592 INFO:teuthology.orchestra.console:Power on for smithi196 completed 2020-03-27T13:50:58.693 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish 2020-03-27T13:53:25.628 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:29.745 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:34.695 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:38.822 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:43.751 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:47.878 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:52.805 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:53:56.934 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:01.862 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:05.990 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:10.918 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:15.046 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:19.975 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:24.102 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:48.521 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:57.581 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:54:57.698 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-03-27T13:55:03.706 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:03.710 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-03-27T13:55:09.712 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:09.717 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-03-27T13:55:15.720 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:18.745 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-03-27T13:55:24.746 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:24.883 INFO:teuthology.orchestra.run.smithi196:> hostname 2020-03-27T13:55:25.118 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com 2020-03-27T13:55:25.119 INFO:teuthology.orchestra.run.smithi196:> grep smithi196.front.sepia.ceph.com /etc/hosts 2020-03-27T13:55:25.161 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196 2020-03-27T13:55:25.162 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196 2020-03-27T13:55:25.232 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hosts 2020-03-27T13:55:25.277 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/172.21.15.196/172.21.15.196/g' /etc/hosts 2020-03-27T13:55:25.321 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hostname 2020-03-27T13:55:25.364 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196 2020-03-27T13:55:25.408 INFO:teuthology.provision.fog.smithi196:Deploy complete! 2020-03-27T13:55:25.417 DEBUG:teuthology.parallel:result is None 2020-03-27T13:55:25.979 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:26.068 INFO:teuthology.orchestra.run.smithi135:> hostname 2020-03-27T13:55:26.307 INFO:teuthology.orchestra.run.smithi135.stdout:smithi135.front.sepia.ceph.com 2020-03-27T13:55:26.308 INFO:teuthology.orchestra.run.smithi135:> grep smithi135.front.sepia.ceph.com /etc/hosts 2020-03-27T13:55:26.349 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.15.135 smithi135.front.sepia.ceph.com smithi135 2020-03-27T13:55:26.351 INFO:teuthology.orchestra.run.smithi135:> sudo hostname smithi135 2020-03-27T13:55:26.404 INFO:teuthology.orchestra.run.smithi135:> sudo sed -i -e 's/smithi135.front.sepia.ceph.com/smithi135/g' /etc/hosts 2020-03-27T13:55:26.448 INFO:teuthology.orchestra.run.smithi135:> sudo sed -i -e 's/172.21.15.135/172.21.15.135/g' /etc/hosts 2020-03-27T13:55:26.492 INFO:teuthology.orchestra.run.smithi135:> sudo sed -i -e 's/smithi135.front.sepia.ceph.com/smithi135/g' /etc/hostname 2020-03-27T13:55:26.536 INFO:teuthology.orchestra.run.smithi135:> sudo hostname smithi135 2020-03-27T13:55:26.579 INFO:teuthology.provision.fog.smithi135:Deploy complete! 2020-03-27T13:55:26.581 DEBUG:teuthology.parallel:result is None 2020-03-27T13:55:26.698 INFO:teuthology.lock.ops:Checking smithi135.front.sepia.ceph.com 2020-03-27T13:55:26.698 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-27T13:55:26.721 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com 2020-03-27T13:55:26.721 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-27T13:55:26.736 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: set os type and version on server 2020-03-27T13:55:26.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:55:26.737 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:26.831 INFO:teuthology.orchestra.run.smithi135:> true 2020-03-27T13:55:26.983 INFO:teuthology.orchestra.run.smithi135:> uname -m 2020-03-27T13:55:27.024 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2020-03-27T13:55:27.025 INFO:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2020-03-27T13:55:27.065 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="Ubuntu" 2020-03-27T13:55:27.065 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="16.04.6 LTS (Xenial Xerus)" 2020-03-27T13:55:27.065 INFO:teuthology.orchestra.run.smithi135.stdout:ID=ubuntu 2020-03-27T13:55:27.065 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE=debian 2020-03-27T13:55:27.065 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="Ubuntu 16.04.6 LTS" 2020-03-27T13:55:27.065 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="16.04" 2020-03-27T13:55:27.066 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="http://www.ubuntu.com/" 2020-03-27T13:55:27.066 INFO:teuthology.orchestra.run.smithi135.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2020-03-27T13:55:27.066 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2020-03-27T13:55:27.066 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_CODENAME=xenial 2020-03-27T13:55:27.066 INFO:teuthology.orchestra.run.smithi135.stdout:UBUNTU_CODENAME=xenial 2020-03-27T13:55:27.067 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2020-03-27T13:55:27.083 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server 2020-03-27T13:55:27.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-27T13:55:27.083 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:27.176 INFO:teuthology.orchestra.run.smithi196:> true 2020-03-27T13:55:27.360 INFO:teuthology.orchestra.run.smithi196:> uname -m 2020-03-27T13:55:27.409 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2020-03-27T13:55:27.409 INFO:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2020-03-27T13:55:27.450 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu" 2020-03-27T13:55:27.450 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="16.04.6 LTS (Xenial Xerus)" 2020-03-27T13:55:27.450 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu 2020-03-27T13:55:27.450 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian 2020-03-27T13:55:27.450 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 16.04.6 LTS" 2020-03-27T13:55:27.450 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="16.04" 2020-03-27T13:55:27.451 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="http://www.ubuntu.com/" 2020-03-27T13:55:27.451 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2020-03-27T13:55:27.451 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2020-03-27T13:55:27.451 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=xenial 2020-03-27T13:55:27.451 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=xenial 2020-03-27T13:55:27.451 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2020-03-27T13:55:27.465 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-03-27T13:55:27.495 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCaVW+IVJxfL3l4KgEjGSoB8rH5AhwrmLez3H2pef2GMmtJZ/CBwf5YK87nDGULt2Xikun88jSkD2Noas/jawsHVg9pOwwZUU9q/N2bvFJZrCx9RxZQG5QQ2A2e8d9v+BjMf76qA5rSYUf8e8bBKLtuWUsGv3ambFU77VNQWNJhTSThkdBy/ktQVqFBKFEvKzwCdEwFplf41srV2QHDfwoN0sF/9bPYYn1ljFWU5HahP26DNHKtdJfv2xVtios3M9quA9GsYJPJXucGGELeo+7TDNVf5r8vWugJNFqCbTrm9rlWTNy5R3Ya6W7jA9zvFWoG0OBS6xGYoMHpC5dQ2Q5 smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCymIoKcZw0p76k3vnudNiOoXh5GaurH1t5TZ/u257hHRfs+AGV9J9eBFs0WmUEHJn0MEyt/47EbjTIQxSJVjjPRJ6a1iSbnmo8dGAQ4W6F4QCZnB6XVbRGsGkSa0kaXmsU8FL5cabhNXhti7YAJhYVt2oyAOeUjqmwVohhgPI93Vl+XkiZQJmoXNvnL5QPDwKzF4vFQAxiHvd+QFpAUjHHtXDQqg3ZGupPrgI/Z45zTner0YjTBeuKfRYjfShWJ9HX4/txkkp8Y0ZgF6O95XmOeMtPZ9a1+YnK5i8sMMNYqknnYuP68QOwMENY8VuvOnrUt7Yy6XeExRK6i8F9w+HL 2020-03-27T13:55:27.495 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T13:55:27.548 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-27T13:55:27.571 INFO:teuthology.task.internal:Saving configuration 2020-03-27T13:55:27.586 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-27T13:55:27.615 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-27T13:55:27.629 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-27 13:50:36.410804', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi135.front.sepia.ceph.com', u'os_version': u'16.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJjI5FibhW8gqIWixKmRkzdSmAPeuh5BLqzSXdj9F2uH', u'description': u'/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894917'} 2020-03-27T13:55:27.643 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-27 13:50:36.410824', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi196.front.sepia.ceph.com', u'os_version': u'16.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAlT72XSjLV1we/K92RpUHavCEjLNygas1vHbZAqe9lN', u'description': u'/home/teuthworker/archive/kchai-2020-03-27_13:29:55-rados-nautilus-kefu2-testing-2020-03-27-1709-distro-basic-smithi/4894917'} 2020-03-27T13:55:27.644 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-27T13:55:27.652 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.a', 'mgr.x'] 2020-03-27T13:55:27.652 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-27T13:55:27.653 INFO:teuthology.run_tasks:Running task console_log... 2020-03-27T13:55:27.724 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-27T13:55:27.725 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-27T13:55:27.731 INFO:teuthology.task.internal:Opening connections... 2020-03-27T13:55:27.732 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2020-03-27T13:55:27.733 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi135.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:27.840 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2020-03-27T13:55:27.841 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi196.front.sepia.ceph.com', 'timeout': 60} 2020-03-27T13:55:27.936 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-27T13:55:27.943 INFO:teuthology.orchestra.run.smithi135:> uname -m 2020-03-27T13:55:27.962 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2020-03-27T13:55:27.962 INFO:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2020-03-27T13:55:28.004 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="Ubuntu" 2020-03-27T13:55:28.005 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="16.04.6 LTS (Xenial Xerus)" 2020-03-27T13:55:28.005 INFO:teuthology.orchestra.run.smithi135.stdout:ID=ubuntu 2020-03-27T13:55:28.005 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE=debian 2020-03-27T13:55:28.005 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="Ubuntu 16.04.6 LTS" 2020-03-27T13:55:28.005 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="16.04" 2020-03-27T13:55:28.005 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="http://www.ubuntu.com/" 2020-03-27T13:55:28.006 INFO:teuthology.orchestra.run.smithi135.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2020-03-27T13:55:28.006 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2020-03-27T13:55:28.006 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_CODENAME=xenial 2020-03-27T13:55:28.006 INFO:teuthology.orchestra.run.smithi135.stdout:UBUNTU_CODENAME=xenial 2020-03-27T13:55:28.006 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2020-03-27T13:55:28.019 INFO:teuthology.orchestra.run.smithi196:> uname -m 2020-03-27T13:55:28.077 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2020-03-27T13:55:28.078 INFO:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2020-03-27T13:55:28.117 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu" 2020-03-27T13:55:28.118 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="16.04.6 LTS (Xenial Xerus)" 2020-03-27T13:55:28.118 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu 2020-03-27T13:55:28.118 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian 2020-03-27T13:55:28.118 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 16.04.6 LTS" 2020-03-27T13:55:28.118 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="16.04" 2020-03-27T13:55:28.118 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="http://www.ubuntu.com/" 2020-03-27T13:55:28.119 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2020-03-27T13:55:28.119 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2020-03-27T13:55:28.119 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=xenial 2020-03-27T13:55:28.119 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=xenial 2020-03-27T13:55:28.121 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2020-03-27T13:55:28.136 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-27T13:55:28.151 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-27T13:55:28.159 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-27T13:55:28.159 INFO:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2020-03-27T13:55:28.162 INFO:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2020-03-27T13:55:28.165 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-27T13:55:28.202 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-27T13:55:28.202 INFO:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2020-03-27T13:55:28.207 INFO:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2020-03-27T13:55:28.214 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-27T13:55:28.258 INFO:teuthology.run_tasks:Running task kernel... 2020-03-27T13:55:28.281 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-27T13:55:28.281 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-27T13:55:28.282 INFO:teuthology.orchestra.run.smithi135:> uname -r 2020-03-27T13:55:28.285 INFO:teuthology.orchestra.run.smithi135.stdout:4.4.0-170-generic 2020-03-27T13:55:28.285 INFO:teuthology.task.kernel:Running kernel on smithi135: 4.4.0-170-generic 2020-03-27T13:55:28.285 INFO:teuthology.orchestra.run.smithi135:> sudo apt-get clean 2020-03-27T13:55:28.386 INFO:teuthology.orchestra.run.smithi135:> sudo apt-get update 2020-03-27T13:55:28.951 INFO:teuthology.orchestra.run.smithi135.stdout:Ign:1 https://chacra.ceph.com/r/valgrind/latest/HEAD/ubuntu/xenial/flavors/default xenial InRelease 2020-03-27T13:55:28.985 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:2 https://chacra.ceph.com/r/valgrind/latest/HEAD/ubuntu/xenial/flavors/default xenial Release 2020-03-27T13:55:29.027 INFO:teuthology.orchestra.run.smithi135.stdout:Ign:3 https://chacra.ceph.com/r/valgrind/latest/HEAD/ubuntu/xenial/flavors/default xenial Release.gpg 2020-03-27T13:55:29.094 INFO:teuthology.orchestra.run.smithi135.stdout:Hit:4 http://archive.ubuntu.com/ubuntu xenial InRelease 2020-03-27T13:55:29.284 INFO:teuthology.orchestra.run.smithi135.stdout:Get:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB] 2020-03-27T13:55:30.087 INFO:teuthology.orchestra.run.smithi135.stdout:Get:6 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB] 2020-03-27T13:55:30.094 INFO:teuthology.orchestra.run.smithi135.stdout:Get:7 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB] 2020-03-27T13:55:30.600 INFO:teuthology.orchestra.run.smithi135.stdout:Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1,120 kB] 2020-03-27T13:55:31.267 INFO:teuthology.orchestra.run.smithi135.stdout:Get:9 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [840 kB] 2020-03-27T13:55:31.305 INFO:teuthology.orchestra.run.smithi135.stdout:Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [907 kB] 2020-03-27T13:55:31.635 INFO:teuthology.orchestra.run.smithi135.stdout:Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [427 kB] 2020-03-27T13:55:31.646 INFO:teuthology.orchestra.run.smithi135.stdout:Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [793 kB] 2020-03-27T13:55:32.007 INFO:teuthology.orchestra.run.smithi135.stdout:Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [719 kB] 2020-03-27T13:55:32.037 INFO:teuthology.orchestra.run.smithi135.stdout:Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [331 kB] 2020-03-27T13:55:32.043 INFO:teuthology.orchestra.run.smithi135.stdout:Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [16.8 kB] 2020-03-27T13:55:32.043 INFO:teuthology.orchestra.run.smithi135.stdout:Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse i386 Packages [15.9 kB] 2020-03-27T13:55:32.190 INFO:teuthology.orchestra.run.smithi135.stdout:Get:17 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [647 kB] 2020-03-27T13:55:32.288 INFO:teuthology.orchestra.run.smithi135.stdout:Get:18 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [319 kB] 2020-03-27T13:55:32.407 INFO:teuthology.orchestra.run.smithi135.stdout:Get:19 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [487 kB] 2020-03-27T13:55:32.480 INFO:teuthology.orchestra.run.smithi135.stdout:Get:20 http://security.ubuntu.com/ubuntu xenial-security/universe i386 Packages [420 kB] 2020-03-27T13:55:32.539 INFO:teuthology.orchestra.run.smithi135.stdout:Get:21 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [199 kB] 2020-03-27T13:55:32.565 INFO:teuthology.orchestra.run.smithi135.stdout:Get:22 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [5,728 B] 2020-03-27T13:55:32.565 INFO:teuthology.orchestra.run.smithi135.stdout:Get:23 http://security.ubuntu.com/ubuntu xenial-security/multiverse i386 Packages [5,880 B] 2020-03-27T13:55:32.642 INFO:teuthology.orchestra.run.smithi135.stdout:Fetched 7,580 kB in 3s (1,926 kB/s) 2020-03-27T13:55:33.949 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2020-03-27T13:55:33.982 INFO:teuthology.orchestra.run.smithi135:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-03-27T13:55:34.012 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2020-03-27T13:55:34.217 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2020-03-27T13:55:34.217 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2020-03-27T13:55:34.242 INFO:teuthology.orchestra.run.smithi135.stderr:E: Unable to locate package linux-image-current-generic 2020-03-27T13:55:34.243 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-03-27T13:55:34.243 INFO:teuthology.orchestra.run.smithi135:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-03-27T13:55:34.312 INFO:teuthology.orchestra.run.smithi135.stdout:Reading package lists... 2020-03-27T13:55:34.521 INFO:teuthology.orchestra.run.smithi135.stdout:Building dependency tree... 2020-03-27T13:55:34.521 INFO:teuthology.orchestra.run.smithi135.stdout:Reading state information... 2020-03-27T13:55:34.654 INFO:teuthology.orchestra.run.smithi135.stdout:The following package was automatically installed and is no longer required: 2020-03-27T13:55:34.654 INFO:teuthology.orchestra.run.smithi135.stdout: linux-modules-extra-4.4.0-170-generic 2020-03-27T13:55:34.654 INFO:teuthology.orchestra.run.smithi135.stdout:Use 'sudo apt autoremove' to remove it. 2020-03-27T13:55:34.655 INFO:teuthology.orchestra.run.smithi135.stdout:The following additional packages will be installed: 2020-03-27T13:55:34.655 INFO:teuthology.orchestra.run.smithi135.stdout: linux-generic linux-headers-4.4.0-176 linux-headers-4.4.0-176-generic 2020-03-27T13:55:34.655 INFO:teuthology.orchestra.run.smithi135.stdout: linux-headers-generic linux-image-4.4.0-176-generic 2020-03-27T13:55:34.656 INFO:teuthology.orchestra.run.smithi135.stdout: linux-modules-4.4.0-176-generic linux-modules-extra-4.4.0-176-generic 2020-03-27T13:55:34.657 INFO:teuthology.orchestra.run.smithi135.stdout:Suggested packages: 2020-03-27T13:55:34.657 INFO:teuthology.orchestra.run.smithi135.stdout: fdutils linux-doc-4.4.0 | linux-source-4.4.0 linux-tools 2020-03-27T13:55:34.657 INFO:teuthology.orchestra.run.smithi135.stdout:Recommended packages: 2020-03-27T13:55:34.657 INFO:teuthology.orchestra.run.smithi135.stdout: thermald 2020-03-27T13:55:34.659 INFO:teuthology.orchestra.run.smithi135.stdout:The following NEW packages will be installed: 2020-03-27T13:55:34.659 INFO:teuthology.orchestra.run.smithi135.stdout: linux-headers-4.4.0-176 linux-headers-4.4.0-176-generic 2020-03-27T13:55:34.659 INFO:teuthology.orchestra.run.smithi135.stdout: linux-image-4.4.0-176-generic linux-modules-4.4.0-176-generic 2020-03-27T13:55:34.659 INFO:teuthology.orchestra.run.smithi135.stdout: linux-modules-extra-4.4.0-176-generic 2020-03-27T13:55:34.660 INFO:teuthology.orchestra.run.smithi135.stdout:The following packages will be upgraded: 2020-03-27T13:55:34.661 INFO:teuthology.orchestra.run.smithi135.stdout: linux-generic linux-headers-generic linux-image-generic 2020-03-27T13:55:34.672 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-27T13:55:34.673 DEBUG:teuthology.task.console_log:Killing console logger for smithi196 2020-03-27T13:55:34.673 DEBUG:teuthology.task.console_log:Killing console logger for smithi135 2020-03-27T13:55:34.673 DEBUG:teuthology.task.console_log:Killing console logger for smithi196 2020-03-27T13:55:34.673 DEBUG:teuthology.task.console_log:Killing console logger for smithi135 2020-03-27T13:55:34.674 DEBUG:teuthology.exit:Finished running handlers