2020-09-08T03:21:14.486 INFO:root:teuthology version: 1.0.0-55bd2dc 2020-09-08T03:21:14.489 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2020-09-08_02:30:03-upgrade:mimic-x-nautilus-distro-basic-smithi --block --owner scheduled_teuthology@teuthology --lock --archive /home/teuthworker/archive/teuthology-2020-09-08_02:30:03-upgrade:mimic-x-nautilus-distro-basic-smithi/5416092 --verbose --description upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 7-final-workload objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} -- /tmp/teuthology-worker.mur0jmk_.tmp 2020-09-08T03:21:14.542 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-09-08T03:21:14.623 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2020-09-08_02:30:03-upgrade:mimic-x-nautilus-distro-basic-smithi/5416092 branch: nautilus description: upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 7-final-workload objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} email: ceph-qa@ceph.io first_in_suite: false job_id: '5416092' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'Run ceph on two nodes, with a separate client-only node. Use xfs beneath the osds. #Note-To enable RHEL runs on ovh nodes, add the following to overrides #ansible.cephlab: # skip_tags: entitlements,packages,repos ' - desc: install ceph/mimic latest - desc: 'install upgrade ceph/-x on 2/3 of cluster restart : mons, osd 0-7 ' - desc: 'randomly kill and revive osd small chance to increase the number of pgs ' - desc: 'randomized correctness test for rados operations on an erasure coded pool ' - desc: 'randomized correctness test for rados operations on an erasure coded pool using the jerasure plugin with k=3 and m=1 ' name: teuthology-2020-09-08_02:30:03-upgrade:mimic-x-nautilus-distro-basic-smithi nuke-on-error: true openstack: - machine: disk: 100 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: nautilus ceph: conf: global: enable experimental unrecoverable data corrupting features: '*' mon warn on msgr2 not enabled: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on osd down out interval zero: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max pg log entries: 2 osd min pg log entries: 1 osd objectstore: bluestore fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(MON_DOWN\) - \(MGR_DOWN\) - \(MON_MSGR2_NOT_ENABLED\) - but it is still running - wrongly marked me down - objects unfound and apparently lost - log bound mismatch - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request mon_bind_addrvec: false mon_bind_msgr2: false sha1: fee7b7e4d8927d79c886fba1b53276175c503b4b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: fee7b7e4d8927d79c886fba1b53276175c503b4b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: nautilus sha1: fee7b7e4d8927d79c886fba1b53276175c503b4b owner: scheduled_teuthology@teuthology priority: 50 repo: git://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - osd.4 - osd.5 - osd.6 - osd.7 - - mon.c - - osd.8 - osd.9 - osd.10 - osd.11 - - client.0 sha1: fee7b7e4d8927d79c886fba1b53276175c503b4b sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 min_in: 4 timeout: 1200 - print: '**** done thrashosds 3-thrash' - rados: clients: - client.0 ec_pool: true objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 write_append_excl: false - print: '**** done rados ec task' suite: upgrade:mimic-x suite_branch: nautilus suite_path: /home/teuthworker/src/git.ceph.com_ceph_nautilus/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: fee7b7e4d8927d79c886fba1b53276175c503b4b tasks: - install: branch: mimic exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-diskprediction-cloud - ceph-mgr-rook - ceph-mgr-ssh extra_packages: - librados2 - print: '**** done install mimic' - ceph: conf: global: bluestore_warn_on_legacy_statfs: false mon pg warn min per osd: 0 - exec: osd.0: - ceph osd require-osd-release mimic - ceph osd set-require-min-compat-client mimic - print: '**** done ceph' - install.upgrade: mon.a: null mon.b: null mon.c: null - print: '**** done install.upgrade first 3 nodes' - ceph.restart: daemons: - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 mon-health-to-clog: false - print: '**** done ceph.restart all mons and 2/3 of osds' - parallel: - stress-tasks - install.upgrade: client.0: null osd.8: null - ceph.restart: daemons: - osd.8 - osd.9 - osd.10 - osd.11 wait-for-healthy: false wait-for-osds-up: true - exec: osd.0: - ceph osd set pglog_hardlimit - ceph osd dump --format=json-pretty | grep "flags" - print: '**** try to set pglog_hardlimit again, should succeed' - rados: clients: - client.0 ec_pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: jerasure31profile plugin: jerasure technique: reed_sol_van objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 write_append_excl: false teuthology_branch: master timestamp: 2020-09-08_02:30:03 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.12108 2020-09-08T03:21:14.623 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_nautilus/qa; will attempt to use it 2020-09-08T03:21:14.624 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_nautilus/qa/tasks 2020-09-08T03:21:14.624 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-09-08T03:21:14.626 INFO:teuthology.task.internal:Checking packages... 2020-09-08T03:21:14.650 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'fee7b7e4d8927d79c886fba1b53276175c503b4b' 2020-09-08T03:21:14.651 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-09-08T03:21:14.651 INFO:teuthology.packaging:ref: None 2020-09-08T03:21:14.652 INFO:teuthology.packaging:tag: None 2020-09-08T03:21:14.652 INFO:teuthology.packaging:branch: nautilus 2020-09-08T03:21:14.652 INFO:teuthology.packaging:sha1: fee7b7e4d8927d79c886fba1b53276175c503b4b 2020-09-08T03:21:14.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=nautilus 2020-09-08T03:21:14.975 INFO:teuthology.task.internal:Found packages for ceph version 14.2.11-122-gfee7b7e-1bionic 2020-09-08T03:21:14.976 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-09-08T03:21:14.984 INFO:teuthology.task.internal:no buildpackages task found 2020-09-08T03:21:14.984 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-09-08T03:21:14.998 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-09-08T03:21:14.999 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-09-08T03:21:15.049 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:21:25.072 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:21:35.104 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:21:45.124 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:21:55.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:22:05.165 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:22:15.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:22:25.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:22:35.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:22:45.247 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:22:55.259 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:23:05.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:23:15.297 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:23:25.320 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:23:35.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:23:45.351 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:23:55.369 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:24:05.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:24:15.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:24:25.433 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:24:35.476 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:24:45.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:24:55.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:25:05.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:25:15.562 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:25:25.587 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:25:35.604 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:25:45.626 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:25:55.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:26:05.668 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:26:15.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:26:25.706 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:26:35.727 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:26:45.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:26:55.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:27:05.787 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:27:15.808 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:27:25.840 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:27:35.853 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:27:45.871 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:27:55.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:28:05.913 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:28:15.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 3)... 2020-09-08T03:28:25.948 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 4)... 2020-09-08T03:28:35.968 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:28:45.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:28:56.006 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:29:06.027 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:29:16.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:29:26.069 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:29:36.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:29:46.108 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:29:56.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:30:06.148 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:30:16.167 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:30:26.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:30:36.317 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:30:46.338 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:30:56.361 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:31:06.387 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:31:16.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:31:26.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:31:36.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:31:46.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:31:56.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:32:06.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:32:16.521 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:32:26.545 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:32:36.564 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:32:46.583 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:32:56.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:33:06.621 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:33:16.643 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:33:26.659 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:33:36.671 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:33:46.691 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:33:56.706 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:34:06.728 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:34:16.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:34:26.769 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:34:36.798 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:34:46.815 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:34:56.832 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:35:06.847 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:35:16.869 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:35:26.891 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:35:36.907 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:35:46.926 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:35:56.946 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:36:06.964 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:36:16.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:36:26.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:36:37.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:36:47.027 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:36:57.039 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:37:07.055 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:37:17.072 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:37:27.089 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:37:37.114 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:37:47.135 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:37:57.148 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:38:07.169 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:38:17.188 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:38:27.435 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:38:37.456 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:38:47.504 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:38:57.524 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:39:07.538 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:39:17.563 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 2)... 2020-09-08T03:39:27.582 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:39:37.592 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:39:47.607 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:39:57.624 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:40:07.642 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:40:17.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:40:27.683 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:40:37.704 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:40:47.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:40:57.756 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:41:07.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:41:17.817 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:41:27.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:41:37.871 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:41:47.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 5, have 1)... 2020-09-08T03:41:57.913 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 5, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2020-09-08_02:30:03-upgrade:mimic-x-nautilus-distro-basic-smithi/5416092'} 2020-09-08T03:41:57.939 DEBUG:teuthology.lock.ops:locked ubuntu@smithi084.front.sepia.ceph.com, ubuntu@smithi158.front.sepia.ceph.com, ubuntu@smithi045.front.sepia.ceph.com, ubuntu@smithi037.front.sepia.ceph.com, ubuntu@smithi085.front.sepia.ceph.com 2020-09-08T03:41:58.143 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fef1380d9d8>, signals=[15]) 2020-09-08T03:41:58.406 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of ubuntu 18.04 2020-09-08T03:41:58.415 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of ubuntu 18.04 2020-09-08T03:41:58.424 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of ubuntu 18.04 2020-09-08T03:41:58.447 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of ubuntu 18.04 2020-09-08T03:41:58.457 INFO:teuthology.provision.fog.smithi158:Scheduling deploy of ubuntu 18.04 2020-09-08T03:41:58.850 INFO:teuthology.orchestra.console:Power off smithi045 2020-09-08T03:41:58.851 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-09-08T03:41:58.882 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:58.904 INFO:teuthology.orchestra.console:Power off smithi084 2020-09-08T03:41:58.905 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-09-08T03:41:58.925 INFO:teuthology.orchestra.console:Power off smithi037 2020-09-08T03:41:58.926 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-09-08T03:41:58.956 INFO:teuthology.orchestra.console:Power off smithi085 2020-09-08T03:41:58.958 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-09-08T03:41:58.975 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:58.999 INFO:teuthology.orchestra.console:Power off smithi158 2020-09-08T03:41:59.000 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-09-08T03:41:59.015 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:59.028 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:59.042 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:59.089 INFO:teuthology.orchestra.console:Power off for smithi045 completed 2020-09-08T03:41:59.130 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2020-09-08T03:41:59.154 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2020-09-08T03:41:59.156 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2020-09-08T03:41:59.167 INFO:teuthology.orchestra.console:Power off for smithi158 completed 2020-09-08T03:41:59.190 INFO:teuthology.orchestra.console:Power on smithi045 2020-09-08T03:41:59.191 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-09-08T03:41:59.222 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:59.235 INFO:teuthology.orchestra.console:Power on smithi084 2020-09-08T03:41:59.236 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-09-08T03:41:59.256 INFO:teuthology.orchestra.console:Power on smithi037 2020-09-08T03:41:59.257 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-09-08T03:41:59.267 INFO:teuthology.orchestra.console:Power on smithi085 2020-09-08T03:41:59.268 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-09-08T03:41:59.277 INFO:teuthology.orchestra.console:Power on smithi158 2020-09-08T03:41:59.278 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-09-08T03:41:59.287 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:59.299 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:59.317 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:41:59.328 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:03.251 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:03.342 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:03.354 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:03.366 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:03.382 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:03.496 INFO:teuthology.orchestra.console:Power on for smithi045 completed 2020-09-08T03:42:03.581 INFO:teuthology.orchestra.console:Power on for smithi158 completed 2020-09-08T03:42:03.596 INFO:teuthology.orchestra.console:Power on for smithi037 completed 2020-09-08T03:42:03.598 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish 2020-09-08T03:42:03.683 INFO:teuthology.provision.fog.smithi158:Waiting for deploy to finish 2020-09-08T03:42:03.697 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish 2020-09-08T03:42:07.459 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:07.473 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T03:42:07.675 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2020-09-08T03:42:07.759 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2020-09-08T03:42:07.777 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2020-09-08T03:42:07.860 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2020-09-08T03:44:49.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:44:50.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:44:54.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:44:54.108 ERROR:teuthology.orchestra.connection:Error connecting to smithi084.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 740, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1580, in auth_publickey return self.auth_handler.wait_for_response(my_event) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response raise e paramiko.ssh_exception.AuthenticationException: Authentication failed. 2020-09-08T03:44:59.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:44:59.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:08.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:08.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:09.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:17.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:17.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:18.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:20.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:26.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:26.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:27.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:29.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:35.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:35.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:36.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:41.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:45.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:47.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:54.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:45:56.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:01.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:03.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:03.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:05.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:06.952 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-09-08T03:46:09.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:09.745 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-09-08T03:46:12.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:12.967 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-09-08T03:46:14.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:15.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:15.759 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-09-08T03:46:18.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:18.983 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-09-08T03:46:21.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:21.772 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-09-08T03:46:24.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:24.997 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-09-08T03:46:25.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:27.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:27.869 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-09-08T03:46:27.872 INFO:teuthology.orchestra.run.smithi045:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-09-08T03:46:28.231 INFO:teuthology.provision.fog.smithi045:Node is ready 2020-09-08T03:46:28.232 INFO:teuthology.orchestra.run.smithi045:> hostname 2020-09-08T03:46:28.281 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045.front.sepia.ceph.com 2020-09-08T03:46:28.283 INFO:teuthology.orchestra.run.smithi045:> grep smithi045.front.sepia.ceph.com /etc/hosts 2020-09-08T03:46:28.329 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45 smithi045.front.sepia.ceph.com smithi045 2020-09-08T03:46:28.330 INFO:teuthology.orchestra.run.smithi045:> sudo hostname smithi045 2020-09-08T03:46:28.399 INFO:teuthology.orchestra.run.smithi045:> sudo sed -i -e 's/smithi045.front.sepia.ceph.com/smithi045/g' /etc/hosts 2020-09-08T03:46:28.458 INFO:teuthology.orchestra.run.smithi045:> sudo sed -i -e 's/172.21.15.45/172.21.15.45/g' /etc/hosts 2020-09-08T03:46:28.517 INFO:teuthology.orchestra.run.smithi045:> sudo sed -i -e 's/smithi045.front.sepia.ceph.com/smithi045/g' /etc/hostname 2020-09-08T03:46:28.574 INFO:teuthology.orchestra.run.smithi045:> sudo hostname smithi045 2020-09-08T03:46:28.628 INFO:teuthology.provision.fog.smithi045:Deploy complete! 2020-09-08T03:46:28.629 DEBUG:teuthology.parallel:result is None 2020-09-08T03:46:31.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:31.082 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-09-08T03:46:31.083 INFO:teuthology.orchestra.run.smithi037:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-09-08T03:46:31.491 INFO:teuthology.provision.fog.smithi037:Node is ready 2020-09-08T03:46:31.491 INFO:teuthology.orchestra.run.smithi037:> hostname 2020-09-08T03:46:31.543 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com 2020-09-08T03:46:31.543 INFO:teuthology.orchestra.run.smithi037:> grep smithi037.front.sepia.ceph.com /etc/hosts 2020-09-08T03:46:31.596 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037 2020-09-08T03:46:31.596 INFO:teuthology.orchestra.run.smithi037:> sudo hostname smithi037 2020-09-08T03:46:31.740 INFO:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hosts 2020-09-08T03:46:31.759 INFO:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/172.21.15.37/172.21.15.37/g' /etc/hosts 2020-09-08T03:46:31.817 INFO:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hostname 2020-09-08T03:46:31.872 INFO:teuthology.orchestra.run.smithi037:> sudo hostname smithi037 2020-09-08T03:46:31.928 INFO:teuthology.provision.fog.smithi037:Deploy complete! 2020-09-08T03:46:31.928 DEBUG:teuthology.parallel:result is None 2020-09-08T03:46:34.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:39.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:43.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:48.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:52.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:46:52.768 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-09-08T03:46:52.769 INFO:teuthology.orchestra.run.smithi084:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-09-08T03:46:53.122 INFO:teuthology.provision.fog.smithi084:Node is ready 2020-09-08T03:46:53.123 INFO:teuthology.orchestra.run.smithi084:> hostname 2020-09-08T03:46:53.172 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2020-09-08T03:46:53.172 INFO:teuthology.orchestra.run.smithi084:> grep smithi084.front.sepia.ceph.com /etc/hosts 2020-09-08T03:46:53.221 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2020-09-08T03:46:53.222 INFO:teuthology.orchestra.run.smithi084:> sudo hostname smithi084 2020-09-08T03:46:53.365 INFO:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi084.front.sepia.ceph.com/smithi084/g' /etc/hosts 2020-09-08T03:46:53.380 INFO:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/172.21.15.84/172.21.15.84/g' /etc/hosts 2020-09-08T03:46:53.438 INFO:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi084.front.sepia.ceph.com/smithi084/g' /etc/hostname 2020-09-08T03:46:53.492 INFO:teuthology.orchestra.run.smithi084:> sudo hostname smithi084 2020-09-08T03:46:53.547 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2020-09-08T03:46:53.547 DEBUG:teuthology.parallel:result is None 2020-09-08T03:46:57.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:06.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:09.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:09.635 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-09-08T03:47:09.635 INFO:teuthology.orchestra.run.smithi085:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-09-08T03:47:10.007 INFO:teuthology.provision.fog.smithi085:Node is ready 2020-09-08T03:47:10.007 INFO:teuthology.orchestra.run.smithi085:> hostname 2020-09-08T03:47:10.054 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com 2020-09-08T03:47:10.055 INFO:teuthology.orchestra.run.smithi085:> grep smithi085.front.sepia.ceph.com /etc/hosts 2020-09-08T03:47:10.107 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085 2020-09-08T03:47:10.108 INFO:teuthology.orchestra.run.smithi085:> sudo hostname smithi085 2020-09-08T03:47:10.269 INFO:teuthology.orchestra.run.smithi085:> sudo sed -i -e 's/smithi085.front.sepia.ceph.com/smithi085/g' /etc/hosts 2020-09-08T03:47:10.291 INFO:teuthology.orchestra.run.smithi085:> sudo sed -i -e 's/172.21.15.85/172.21.15.85/g' /etc/hosts 2020-09-08T03:47:10.347 INFO:teuthology.orchestra.run.smithi085:> sudo sed -i -e 's/smithi085.front.sepia.ceph.com/smithi085/g' /etc/hostname 2020-09-08T03:47:10.404 INFO:teuthology.orchestra.run.smithi085:> sudo hostname smithi085 2020-09-08T03:47:10.457 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2020-09-08T03:47:10.457 DEBUG:teuthology.parallel:result is None 2020-09-08T03:47:15.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:15.930 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-09-08T03:47:15.931 INFO:teuthology.orchestra.run.smithi158:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-09-08T03:47:16.323 INFO:teuthology.provision.fog.smithi158:Node is ready 2020-09-08T03:47:16.323 INFO:teuthology.orchestra.run.smithi158:> hostname 2020-09-08T03:47:16.370 INFO:teuthology.orchestra.run.smithi158.stdout:smithi158.front.sepia.ceph.com 2020-09-08T03:47:16.370 INFO:teuthology.orchestra.run.smithi158:> grep smithi158.front.sepia.ceph.com /etc/hosts 2020-09-08T03:47:16.418 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158 smithi158.front.sepia.ceph.com smithi158 2020-09-08T03:47:16.419 INFO:teuthology.orchestra.run.smithi158:> sudo hostname smithi158 2020-09-08T03:47:16.590 INFO:teuthology.orchestra.run.smithi158:> sudo sed -i -e 's/smithi158.front.sepia.ceph.com/smithi158/g' /etc/hosts 2020-09-08T03:47:16.608 INFO:teuthology.orchestra.run.smithi158:> sudo sed -i -e 's/172.21.15.158/172.21.15.158/g' /etc/hosts 2020-09-08T03:47:16.662 INFO:teuthology.orchestra.run.smithi158:> sudo sed -i -e 's/smithi158.front.sepia.ceph.com/smithi158/g' /etc/hostname 2020-09-08T03:47:16.719 INFO:teuthology.orchestra.run.smithi158:> sudo hostname smithi158 2020-09-08T03:47:16.774 INFO:teuthology.provision.fog.smithi158:Deploy complete! 2020-09-08T03:47:16.775 DEBUG:teuthology.parallel:result is None 2020-09-08T03:47:17.103 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2020-09-08T03:47:17.104 INFO:teuthology.lock.ops:New key found. Updating... 2020-09-08T03:47:17.133 INFO:teuthology.lock.ops:Checking smithi158.front.sepia.ceph.com 2020-09-08T03:47:17.134 INFO:teuthology.lock.ops:New key found. Updating... 2020-09-08T03:47:17.164 INFO:teuthology.lock.ops:Checking smithi045.front.sepia.ceph.com 2020-09-08T03:47:17.164 INFO:teuthology.lock.ops:New key found. Updating... 2020-09-08T03:47:17.196 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com 2020-09-08T03:47:17.196 INFO:teuthology.lock.ops:New key found. Updating... 2020-09-08T03:47:17.220 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2020-09-08T03:47:17.220 INFO:teuthology.lock.ops:New key found. Updating... 2020-09-08T03:47:17.254 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2020-09-08T03:47:17.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-09-08T03:47:17.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:17.414 INFO:teuthology.orchestra.run.smithi084:> true 2020-09-08T03:47:17.612 INFO:teuthology.orchestra.run.smithi084:> uname -m 2020-09-08T03:47:17.668 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2020-09-08T03:47:17.669 INFO:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2020-09-08T03:47:17.718 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Ubuntu" 2020-09-08T03:47:17.718 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-09-08T03:47:17.718 INFO:teuthology.orchestra.run.smithi084.stdout:ID=ubuntu 2020-09-08T03:47:17.719 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE=debian 2020-09-08T03:47:17.719 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-09-08T03:47:17.719 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="18.04" 2020-09-08T03:47:17.720 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.ubuntu.com/" 2020-09-08T03:47:17.720 INFO:teuthology.orchestra.run.smithi084.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-09-08T03:47:17.720 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-09-08T03:47:17.721 INFO:teuthology.orchestra.run.smithi084.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-09-08T03:47:17.721 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_CODENAME=bionic 2020-09-08T03:47:17.721 INFO:teuthology.orchestra.run.smithi084.stdout:UBUNTU_CODENAME=bionic 2020-09-08T03:47:17.722 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2020-09-08T03:47:17.742 INFO:teuthology.lock.ops:Updating [smithi158.front.sepia.ceph.com]: set os type and version on server 2020-09-08T03:47:17.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-09-08T03:47:17.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:17.822 INFO:teuthology.orchestra.run.smithi158:> true 2020-09-08T03:47:18.023 INFO:teuthology.orchestra.run.smithi158:> uname -m 2020-09-08T03:47:18.069 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2020-09-08T03:47:18.070 INFO:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2020-09-08T03:47:18.121 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="Ubuntu" 2020-09-08T03:47:18.122 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-09-08T03:47:18.122 INFO:teuthology.orchestra.run.smithi158.stdout:ID=ubuntu 2020-09-08T03:47:18.122 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE=debian 2020-09-08T03:47:18.122 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-09-08T03:47:18.122 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="18.04" 2020-09-08T03:47:18.123 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://www.ubuntu.com/" 2020-09-08T03:47:18.123 INFO:teuthology.orchestra.run.smithi158.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-09-08T03:47:18.123 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-09-08T03:47:18.123 INFO:teuthology.orchestra.run.smithi158.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-09-08T03:47:18.123 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_CODENAME=bionic 2020-09-08T03:47:18.124 INFO:teuthology.orchestra.run.smithi158.stdout:UBUNTU_CODENAME=bionic 2020-09-08T03:47:18.124 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2020-09-08T03:47:18.143 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: set os type and version on server 2020-09-08T03:47:18.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-09-08T03:47:18.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:18.237 INFO:teuthology.orchestra.run.smithi045:> true 2020-09-08T03:47:18.439 INFO:teuthology.orchestra.run.smithi045:> uname -m 2020-09-08T03:47:18.487 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2020-09-08T03:47:18.487 INFO:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2020-09-08T03:47:18.559 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Ubuntu" 2020-09-08T03:47:18.559 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-09-08T03:47:18.561 INFO:teuthology.orchestra.run.smithi045.stdout:ID=ubuntu 2020-09-08T03:47:18.569 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE=debian 2020-09-08T03:47:18.570 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-09-08T03:47:18.570 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="18.04" 2020-09-08T03:47:18.570 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.ubuntu.com/" 2020-09-08T03:47:18.570 INFO:teuthology.orchestra.run.smithi045.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-09-08T03:47:18.570 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-09-08T03:47:18.571 INFO:teuthology.orchestra.run.smithi045.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-09-08T03:47:18.571 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_CODENAME=bionic 2020-09-08T03:47:18.571 INFO:teuthology.orchestra.run.smithi045.stdout:UBUNTU_CODENAME=bionic 2020-09-08T03:47:18.571 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2020-09-08T03:47:18.589 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server 2020-09-08T03:47:18.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-09-08T03:47:18.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:18.671 INFO:teuthology.orchestra.run.smithi037:> true 2020-09-08T03:47:18.863 INFO:teuthology.orchestra.run.smithi037:> uname -m 2020-09-08T03:47:18.920 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2020-09-08T03:47:18.924 INFO:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2020-09-08T03:47:18.967 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2020-09-08T03:47:18.967 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-09-08T03:47:18.967 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2020-09-08T03:47:18.968 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2020-09-08T03:47:18.968 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-09-08T03:47:18.968 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="18.04" 2020-09-08T03:47:18.968 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2020-09-08T03:47:18.968 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-09-08T03:47:18.969 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-09-08T03:47:18.969 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-09-08T03:47:18.969 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=bionic 2020-09-08T03:47:18.969 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=bionic 2020-09-08T03:47:18.970 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2020-09-08T03:47:18.987 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2020-09-08T03:47:18.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-09-08T03:47:18.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T03:47:19.068 INFO:teuthology.orchestra.run.smithi085:> true 2020-09-08T03:47:19.271 INFO:teuthology.orchestra.run.smithi085:> uname -m 2020-09-08T03:47:19.319 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2020-09-08T03:47:19.320 INFO:teuthology.orchestra.run.smithi085:> cat /etc/os-release 2020-09-08T03:47:19.371 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="Ubuntu" 2020-09-08T03:47:19.372 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-09-08T03:47:19.372 INFO:teuthology.orchestra.run.smithi085.stdout:ID=ubuntu 2020-09-08T03:47:19.375 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE=debian 2020-09-08T03:47:19.375 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-09-08T03:47:19.375 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="18.04" 2020-09-08T03:47:19.375 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.ubuntu.com/" 2020-09-08T03:47:19.375 INFO:teuthology.orchestra.run.smithi085.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-09-08T03:47:19.376 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-09-08T03:47:19.376 INFO:teuthology.orchestra.run.smithi085.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-09-08T03:47:19.376 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_CODENAME=bionic 2020-09-08T03:47:19.376 INFO:teuthology.orchestra.run.smithi085.stdout:UBUNTU_CODENAME=bionic 2020-09-08T03:47:19.377 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2020-09-08T03:47:19.406 INFO:teuthology.task.internal.lock_machines:5 smithi machines locked this try, 5/5 locked so far 2020-09-08T03:47:19.509 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4FRNssebtLSHOHmET/+XwFuJPeX6OUt7cpvBJg9vE7d9YxTmvN9TbRtuYnKuKrUhWVNKLm93tHe7WjVZ3SgLTY+isHRPp7rt3IQX0HEQBK/gdEtIFQcyMY9hhHgUUqNHx1b10xJGXydEIAxUZArBRfDgDNGFe7ASFAWQH5OsC4MMfh78CAKHppLayXEKJGhw95ALx+r+ZZgcD5Z6kJtdcmRPGyNezAc5q79kdNUyht4nIHeQA7XYZmMz13fVXzy/39LEIsD/hTFzMcTb8VVKA+Ki8TRUaCIRX5xwlsZ/MwS+74gghzcQ1227Jt/6IUS2X+fKmn403jsQK0lLLly/h smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNaL9U2zC33/cjgiB5i1jcNkjHjgiRMbRTukeFYdNmjc9qIDekmhPyLOsVEhaQflkS+dmUJUDBWsL1r5tBSM+aZqkMTQtzFV9b1DtIUhBkYTS/5Wcf+O4fXnOPwTKg4cPPYsi8BPO20miRa+06RD3qzdMc7s1jPRq/VtoXkiBO8rmpnV8psrhCvwyf4wON5Tio8FqFaoWaxWdLXBMGojnbLxTdAfZ0LIyhbtpA+UGYU4Fc4SUM26DUt8v8HX0RW2wLn2T+nZm12i07XCsuLe7tlWXenoBv/V0MxOzARcmBt+5qgfz/RX83U4WKQgjxY1130VcYhOcCzY1RkuaZ15/1 smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmS+lPfo2GyJLblFM3AuOYWkKTHky8sytwx9uvXDw3CaIswauDGKciitoL8jAAFqBW45vhe4CHRBICD0OJ/LNeJtAeK2VwTo74Q54+8Dg6M9ZIz21PKJV7uspNfd0jrVTpt4VT7UXdeOzZNoPsS8mLCwLe+R8GI1OOgS55AQfFsDzJMnHFsTImBNQKeg8KqXD7O9IOYF434thqFZy66vo/Aw4Y6l6xRMFY2lnxMcB2fO6z7aTcxlKhN45Jtu4kTVeij4IUnMqqGDXadiNsLLBkq8JdVEE7vp/6B5iQjC1OBbhhPnbcbUG1+Ei9rP6UR33GAG9c9Pni5mDf71MgCZob smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1HW58C4vi/cQa36KE5xK7ShoS7uIM6/e3247bN2j4jkgcVqYMUozvhODEAXapOom1miKCFj5oxkPGuqJL+tgaXMAn/stupCUFrchlGp31a6PJJQvUqYFknubx+85A8rt/Bprlqc+5MNXgynD2ImsnasyNyOkyZHDmBZWh0awPZjYp9+8QE3ivgUjpDqjVtNATqyWR+5HgttnmK+lEtxnahVNCB/VtTbXWB43cXBbLLzLbWjNSfrUy801aB+gK3wiXXikF0IdFT7yeMC+cq9FT5umGSqaiCJJxpFnzgW9+haUaFWWGRWUenCcP2fmTjsmRIssedfqhJsmX8y8S/46Z smithi158.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAtwaIKFWh0WNp6IOyQXoWcT+m7B1ZmOv+dw0cmfAM40M+XPDywO7z20pqh9ihhuxuR6j0qjKd9I1qmnDGIIzwW695MHX86jvJAoMcERSHCIyOX3zf4O5f3vEgF8vpL4Kncv5v09qZu+jC4CFOhUDATtnlyS7MsIQuZBFtfHM9SMA+22wXzT/KB7S8UwcKliVx4zXHu0xquJptatwwM5E7fQbGCgpcEPE71isk2EW63/X13mlhld7HxSKYxrjGVTWwiO54+owgm0ieoVLAEPAUOedLlmvpb3hK1A6lzzAFRuUfeRL68sqTwPMDOited8okbIgt2XrF+DgkYs6vkMYB 2020-09-08T03:47:19.509 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-09-08T03:47:19.573 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-09-08T03:47:19.694 INFO:teuthology.task.internal:Saving configuration 2020-09-08T03:47:19.752 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-09-08T03:47:19.784 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-09-08T03:47:19.802 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-09-08 03:41:57.921281', 'description': '/home/teuthworker/archive/teuthology-2020-09-08_02:30:03-upgrade:mimic-x-nautilus-distro-basic-smithi/5416092', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICsPW5GnqsttUqAQMIkg0w5XhVHn/U4eXkwp77LeV4Oi', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi084.front.sepia.ceph.com'} 2020-09-08T03:47:19.824 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': False, 'locked_since': None, 'description': None, 'locked_by': None, 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILlahC+k845VhoYwqBKaUhkLOP+kOlZS2MkxLmzKnvOf', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi158.front.sepia.ceph.com'} 2020-09-08T03:47:19.824 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi158.front.sepia.ceph.com is not locked 2020-09-08T03:47:19.868 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/sepia/teuthology/?query=d4e1c6fddd0f45d9bc56c8dec6c096b1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi158.front.sepia.ceph.com is not locked 2020-09-08T03:47:19.869 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines 2020-09-08T03:47:19.898 INFO:teuthology.nuke:Checking targets against current locks 2020-09-08T03:47:19.990 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/scripts/run.py", line 34, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 407, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 268, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable