2019-11-08T18:40:51.170 INFO:root:teuthology version: 1.0.0-8e58544 2019-11-08T18:40:51.171 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.8mbIBC.tmp -- --verbose --lock --description rados/singleton/{all/test_envlibrados_for_rocksdb.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} --name sage-2019-11-08_17:49:46-rados-master-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2019-11-08_17:49:46-rados-master-distro-basic-smithi/4484648 2019-11-08T18:40:51.199 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-08T18:40:51.252 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2019-11-08_17:49:46-rados-master-distro-basic-smithi/4484648 branch: master description: rados/singleton/{all/test_envlibrados_for_rocksdb.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} email: null first_in_suite: false job_id: '4484648' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2019-11-08_17:49:46-rados-master-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: master ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async osd max object name len: 460 osd max object namespace len: 64 mgr: debug monc: 10 mon: debug mon: 20 debug ms: 20 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME mon_bind_msgr2: false sha1: 457b9560a30baf1e82c12db76889a7bd6b8d7a8e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 457b9560a30baf1e82c12db76889a7bd6b8d7a8e thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: master sha1: 25e26d27cf1d18c025a717b3adeaff12c8df41a4 owner: scheduled_sage@teuthology priority: 80 repo: git://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 457b9560a30baf1e82c12db76889a7bd6b8d7a8e suite: rados suite_branch: master suite_path: /home/teuthworker/src/git.ceph.com_ceph_master/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: 25e26d27cf1d18c025a717b3adeaff12c8df41a4 tasks: - install: null - ceph: log-whitelist: - overall HEALTH_ - \(POOL_APP_NOT_ENABLED\) - workunit: clients: all: - rados/test_envlibrados_for_rocksdb.sh teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.17311 2019-11-08T18:40:51.252 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_master/qa; will attempt to use it 2019-11-08T18:40:51.253 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks 2019-11-08T18:40:51.253 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-11-08T18:40:51.254 INFO:teuthology.task.internal:Checking packages... 2019-11-08T18:40:51.267 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '457b9560a30baf1e82c12db76889a7bd6b8d7a8e' 2019-11-08T18:40:51.268 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-11-08T18:40:51.268 INFO:teuthology.packaging:ref: None 2019-11-08T18:40:51.268 INFO:teuthology.packaging:tag: None 2019-11-08T18:40:51.268 INFO:teuthology.packaging:branch: master 2019-11-08T18:40:51.268 INFO:teuthology.packaging:sha1: 457b9560a30baf1e82c12db76889a7bd6b8d7a8e 2019-11-08T18:40:51.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=master 2019-11-08T18:40:52.210 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6988-g25e26d2-1bionic 2019-11-08T18:40:52.210 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-11-08T18:40:52.221 INFO:teuthology.task.internal:no buildpackages task found 2019-11-08T18:40:52.222 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-11-08T18:40:52.241 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-11-08T18:40:52.241 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-08T18:40:52.287 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:41:02.295 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:41:12.313 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:41:22.329 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:41:32.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:41:42.357 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:41:52.367 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:42:02.379 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:42:12.397 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:42:22.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:42:32.432 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:42:42.448 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:42:52.465 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:43:02.474 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:43:12.492 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:43:22.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:43:32.527 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 0)... 2019-11-08T18:43:42.540 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:43:52.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:44:02.573 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2019-11-08T18:44:12.589 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_sage@teuthology', 'description': '/home/teuthworker/archive/sage-2019-11-08_17:49:46-rados-master-distro-basic-smithi/4484648', 'machine_type': 'smithi'} 2019-11-08T18:44:12.601 DEBUG:teuthology.lock.ops:locked ubuntu@smithi089.front.sepia.ceph.com 2019-11-08T18:44:12.662 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-11-08T18:44:12.848 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of ubuntu 18.04 2019-11-08T18:44:13.149 INFO:teuthology.orchestra.console:Power off smithi089 2019-11-08T18:44:13.150 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-11-08T18:44:13.184 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-08T18:44:17.225 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-08T18:44:21.354 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-08T18:44:21.583 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2019-11-08T18:44:21.684 INFO:teuthology.orchestra.console:Power on smithi089 2019-11-08T18:44:21.684 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-11-08T18:44:21.723 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-08T18:44:25.767 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-08T18:44:29.896 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-11-08T18:44:30.123 INFO:teuthology.orchestra.console:Power on for smithi089 completed 2019-11-08T18:44:30.224 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish 2019-11-08T18:47:46.341 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:48:10.797 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:48:19.854 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:48:28.910 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:48:37.966 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:48:47.021 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:48:56.078 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:02.080 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:02.091 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:08.099 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:08.109 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:14.119 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:14.128 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:20.137 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:20.148 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:26.157 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:26.167 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:32.176 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:32.186 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:38.195 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:38.205 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:44.214 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:44.225 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:50.235 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:50:50.244 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-11-08T18:50:56.254 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:52:02.309 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:52:10.670 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:52:19.722 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:52:28.781 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:52:37.837 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:52:46.894 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:52:55.946 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:53:05.004 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:53:14.061 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:53:23.117 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:53:32.174 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:53:41.230 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:53:50.281 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:53:59.338 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:54:08.397 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:54:17.454 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:54:26.505 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:54:35.563 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:54:44.621 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:54:53.678 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:55:02.734 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:55:11.790 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:55:20.846 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:55:29.902 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:55:38.962 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:55:48.045 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:55:57.098 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:56:06.158 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:56:15.214 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:56:24.270 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:56:33.325 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:56:42.381 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:56:51.438 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:57:00.494 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:57:09.546 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:57:18.606 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:57:27.662 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:57:36.718 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:57:45.774 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:57:54.830 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:58:03.886 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:58:12.942 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:58:21.998 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:58:31.054 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:58:40.109 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:58:49.165 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:58:58.221 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:59:07.278 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:59:16.333 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:59:25.390 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:59:34.445 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:59:43.502 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T18:59:52.556 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:00:01.614 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:00:10.670 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:00:19.724 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:00:28.782 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:00:37.838 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:00:46.894 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:00:55.950 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:01:05.002 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:01:14.060 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:01:23.118 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:01:32.173 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:01:41.229 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:01:50.286 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:01:59.338 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:02:08.397 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:02:17.450 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:02:26.510 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:02:35.565 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:02:44.622 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:02:53.682 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:03:02.766 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:03:11.822 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:03:20.882 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:03:29.963 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:03:39.022 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:03:48.078 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:03:57.133 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:04:06.187 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:04:15.243 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:04:24.302 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi089.front.sepia.ceph.com', 'timeout': 60} 2019-11-08T19:04:27.357 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/lock_machines.py", line 79, in lock_machines os_version, arch) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/ops.py", line 145, in lock_many reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/__init__.py", line 26, in reimage return fog_obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/fog.py", line 89, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/fog.py", line 261, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 132, in __call__ raise MaxWhileTries(error_msg) MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2019-11-08T19:04:27.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines 2019-11-08T19:04:27.368 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2019-11-08T19:04:27.370 INFO:teuthology.run:Summary data: description: rados/singleton/{all/test_envlibrados_for_rocksdb.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} failure_reason: reached maximum tries (100) after waiting for 600 seconds owner: scheduled_sage@teuthology status: dead success: false 2019-11-08T19:04:27.371 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-11-08T19:04:27.421 INFO:teuthology.run:DEAD