2020-11-12T17:06:46.737 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-12T17:06:46.739 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616032 --verbose --lock --block --owner scheduled_yuriw@teuthology --name yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench} -- /tmp/teuthology-worker.cifrlo67.tmp 2020-11-12T17:06:46.781 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-12T17:06:46.856 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616032 branch: octopus description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench} email: ceph-qa@ceph.io first_in_suite: false job_id: '5616032' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 - volumes: size: 50 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: octopus ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_async_recovery_min_cost: 1 osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 filestore queue throttle high multiple: 2 filestore queue throttle max multiple: 10 journal throttle high multiple: 2 journal throttle max multiple: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on degraded: true osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 9 osd max markdown count: 1000 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 min_in: 4 workunit: branch: octopus sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af sleep_before_teardown: 0 suite: rados suite_branch: octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph_octopus/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph osd set-require-min-compat-client luminous - ceph balancer mode upmap - ceph balancer on - thrashosds: chance_pgnum_grow: 3 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology_branch: master timestamp: 2020-11-12_17:03:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.7071 2020-11-12T17:06:46.856 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_octopus/qa; will attempt to use it 2020-11-12T17:06:46.857 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_octopus/qa/tasks 2020-11-12T17:06:46.857 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-12T17:06:46.859 INFO:teuthology.task.internal:Checking packages... 2020-11-12T17:06:46.879 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '31d4e76d0d584790763b4b1146b29ea4cfc2e9af' 2020-11-12T17:06:46.879 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-12T17:06:46.880 INFO:teuthology.packaging:ref: None 2020-11-12T17:06:46.880 INFO:teuthology.packaging:tag: None 2020-11-12T17:06:46.880 INFO:teuthology.packaging:branch: octopus 2020-11-12T17:06:46.880 INFO:teuthology.packaging:sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af 2020-11-12T17:06:46.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2020-11-12T17:06:47.205 INFO:teuthology.task.internal:Found packages for ceph version 15.2.5-585-g31d4e76d-1bionic 2020-11-12T17:06:47.206 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-12T17:06:47.216 INFO:teuthology.task.internal:no buildpackages task found 2020-11-12T17:06:47.217 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-12T17:06:47.240 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-12T17:06:47.241 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-12T17:06:47.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:06:57.323 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:07:07.337 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:07:17.356 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:07:27.371 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:07:37.387 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:07:47.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:07:57.419 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:08:07.438 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:08:17.454 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:08:27.471 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:08:37.490 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:08:47.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:08:57.517 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:09:07.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:09:17.554 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:09:27.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:09:37.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:09:47.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:09:57.679 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:10:07.699 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:10:17.715 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:10:27.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:10:37.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:10:47.762 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:10:57.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:11:07.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:11:17.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:11:27.819 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:11:37.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:11:47.854 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:11:57.894 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:12:07.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:12:17.974 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:12:28.018 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:12:38.047 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:12:48.092 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:12:58.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:13:08.169 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:13:18.216 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:13:28.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:13:38.272 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:13:48.286 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:13:58.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:14:08.322 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:14:18.373 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:14:28.421 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:14:38.451 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:14:48.496 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:14:58.537 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:15:08.588 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:15:18.624 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:15:28.655 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:15:38.700 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:15:48.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:15:58.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:16:08.836 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:16:18.937 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:16:28.975 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:16:39.037 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:16:49.096 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:16:59.159 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:17:09.223 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:17:19.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:17:29.406 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:17:39.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:17:49.504 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:17:59.560 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:18:09.609 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:18:19.661 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:18:29.679 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:18:39.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:18:49.743 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:18:59.762 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:19:09.805 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:19:19.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:19:29.864 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:19:39.881 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:19:49.900 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:19:59.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:20:09.967 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:20:20.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:20:30.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:20:40.081 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:20:50.133 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:21:00.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:21:10.234 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:21:20.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:21:30.321 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:21:40.348 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:21:50.368 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:00.404 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:10.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:20.493 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:30.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:40.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:50.597 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:00.649 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:10.673 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:20.711 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:30.751 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:40.799 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:50.886 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuriw@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616032'} 2020-11-12T17:23:51.026 DEBUG:teuthology.lock.ops:locked ubuntu@smithi132.front.sepia.ceph.com, ubuntu@smithi168.front.sepia.ceph.com 2020-11-12T17:23:51.299 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd826c63b70>, signals=[15]) 2020-11-12T17:23:51.667 INFO:teuthology.provision.fog.smithi168:Scheduling deploy of ubuntu 18.04 2020-11-12T17:23:51.689 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of ubuntu 18.04 2020-11-12T17:23:52.291 INFO:teuthology.orchestra.console:Power off smithi168 2020-11-12T17:23:52.296 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-12T17:23:52.448 INFO:teuthology.orchestra.console:Power off smithi132 2020-11-12T17:23:52.449 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-12T17:23:52.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:23:52.490 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:23:56.510 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:23:56.525 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:00.630 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:00.674 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:00.877 INFO:teuthology.orchestra.console:Power off for smithi168 completed 2020-11-12T17:24:00.884 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2020-11-12T17:24:00.979 INFO:teuthology.orchestra.console:Power on smithi168 2020-11-12T17:24:00.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-12T17:24:01.004 INFO:teuthology.orchestra.console:Power on smithi132 2020-11-12T17:24:01.005 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-12T17:24:01.040 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:01.056 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:05.082 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:06.362 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:06.564 INFO:teuthology.orchestra.console:Power on for smithi168 completed 2020-11-12T17:24:06.575 INFO:teuthology.orchestra.console:Power on for smithi132 completed 2020-11-12T17:24:06.667 INFO:teuthology.provision.fog.smithi168:Waiting for deploy to finish 2020-11-12T17:24:06.676 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish 2020-11-12T17:26:53.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:02.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:08.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:11.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:17.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:20.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:26.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:30.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:35.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:39.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:44.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:48.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:53.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:02.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:11.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:53.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:54.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:54.288 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-12T17:28:56.435 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-12T17:28:56.539 INFO:teuthology.orchestra.run.smithi132:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-12T17:28:56.779 INFO:teuthology.provision.fog.smithi132:Node is ready 2020-11-12T17:28:56.782 INFO:teuthology.orchestra.run.smithi132:> hostname 2020-11-12T17:28:56.821 INFO:teuthology.orchestra.run.smithi132.stdout:smithi132.front.sepia.ceph.com 2020-11-12T17:28:56.822 INFO:teuthology.orchestra.run.smithi132:> grep smithi132.front.sepia.ceph.com /etc/hosts 2020-11-12T17:28:56.870 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132 smithi132.front.sepia.ceph.com smithi132 2020-11-12T17:28:56.872 INFO:teuthology.orchestra.run.smithi132:> sudo hostname smithi132 2020-11-12T17:28:56.948 INFO:teuthology.orchestra.run.smithi132:> sudo sed -i -e 's/smithi132.front.sepia.ceph.com/smithi132/g' /etc/hosts 2020-11-12T17:28:57.007 INFO:teuthology.orchestra.run.smithi132:> sudo sed -i -e 's/172.21.15.132/172.21.15.132/g' /etc/hosts 2020-11-12T17:28:57.062 INFO:teuthology.orchestra.run.smithi132:> sudo sed -i -e 's/smithi132.front.sepia.ceph.com/smithi132/g' /etc/hostname 2020-11-12T17:28:57.119 INFO:teuthology.orchestra.run.smithi132:> sudo hostname smithi132 2020-11-12T17:28:57.171 INFO:teuthology.provision.fog.smithi132:Deploy complete! 2020-11-12T17:28:57.184 DEBUG:teuthology.parallel:result is None 2020-11-12T17:29:00.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:00.324 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-12T17:29:06.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:09.420 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-12T17:29:15.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:15.666 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-12T17:29:15.667 INFO:teuthology.orchestra.run.smithi168:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-12T17:29:16.075 INFO:teuthology.provision.fog.smithi168:Node is ready 2020-11-12T17:29:16.076 INFO:teuthology.orchestra.run.smithi168:> hostname 2020-11-12T17:29:16.126 INFO:teuthology.orchestra.run.smithi168.stdout:smithi168.front.sepia.ceph.com 2020-11-12T17:29:16.126 INFO:teuthology.orchestra.run.smithi168:> grep smithi168.front.sepia.ceph.com /etc/hosts 2020-11-12T17:29:16.172 INFO:teuthology.orchestra.run.smithi168.stdout:172.21.15.168 smithi168.front.sepia.ceph.com smithi168 2020-11-12T17:29:16.173 INFO:teuthology.orchestra.run.smithi168:> sudo hostname smithi168 2020-11-12T17:29:16.282 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/smithi168.front.sepia.ceph.com/smithi168/g' /etc/hosts 2020-11-12T17:29:16.299 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/172.21.15.168/172.21.15.168/g' /etc/hosts 2020-11-12T17:29:16.357 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/smithi168.front.sepia.ceph.com/smithi168/g' /etc/hostname 2020-11-12T17:29:16.412 INFO:teuthology.orchestra.run.smithi168:> sudo hostname smithi168 2020-11-12T17:29:16.468 INFO:teuthology.provision.fog.smithi168:Deploy complete! 2020-11-12T17:29:16.471 DEBUG:teuthology.parallel:result is None 2020-11-12T17:29:16.967 INFO:teuthology.lock.ops:Checking smithi132.front.sepia.ceph.com 2020-11-12T17:29:16.968 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-12T17:29:16.993 INFO:teuthology.lock.ops:Checking smithi168.front.sepia.ceph.com 2020-11-12T17:29:16.994 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-12T17:29:17.015 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: set os type and version on server 2020-11-12T17:29:17.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-12T17:29:17.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:17.102 INFO:teuthology.orchestra.run.smithi132:> true 2020-11-12T17:29:17.340 INFO:teuthology.orchestra.run.smithi132:> uname -m 2020-11-12T17:29:17.386 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2020-11-12T17:29:17.386 INFO:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2020-11-12T17:29:17.434 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="Ubuntu" 2020-11-12T17:29:17.435 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-12T17:29:17.435 INFO:teuthology.orchestra.run.smithi132.stdout:ID=ubuntu 2020-11-12T17:29:17.435 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE=debian 2020-11-12T17:29:17.436 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-12T17:29:17.436 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="18.04" 2020-11-12T17:29:17.436 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-12T17:29:17.436 INFO:teuthology.orchestra.run.smithi132.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-12T17:29:17.436 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-12T17:29:17.436 INFO:teuthology.orchestra.run.smithi132.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-12T17:29:17.436 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_CODENAME=bionic 2020-11-12T17:29:17.437 INFO:teuthology.orchestra.run.smithi132.stdout:UBUNTU_CODENAME=bionic 2020-11-12T17:29:17.453 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2020-11-12T17:29:17.488 INFO:teuthology.lock.ops:Updating [smithi168.front.sepia.ceph.com]: set os type and version on server 2020-11-12T17:29:17.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-12T17:29:17.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:17.580 INFO:teuthology.orchestra.run.smithi168:> true 2020-11-12T17:29:17.819 INFO:teuthology.orchestra.run.smithi168:> uname -m 2020-11-12T17:29:17.866 INFO:teuthology.orchestra.run.smithi168.stdout:x86_64 2020-11-12T17:29:17.867 INFO:teuthology.orchestra.run.smithi168:> cat /etc/os-release 2020-11-12T17:29:17.915 INFO:teuthology.orchestra.run.smithi168.stdout:NAME="Ubuntu" 2020-11-12T17:29:17.915 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-12T17:29:17.915 INFO:teuthology.orchestra.run.smithi168.stdout:ID=ubuntu 2020-11-12T17:29:17.915 INFO:teuthology.orchestra.run.smithi168.stdout:ID_LIKE=debian 2020-11-12T17:29:17.916 INFO:teuthology.orchestra.run.smithi168.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-12T17:29:17.916 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_ID="18.04" 2020-11-12T17:29:17.916 INFO:teuthology.orchestra.run.smithi168.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-12T17:29:17.916 INFO:teuthology.orchestra.run.smithi168.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-12T17:29:17.916 INFO:teuthology.orchestra.run.smithi168.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-12T17:29:17.916 INFO:teuthology.orchestra.run.smithi168.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-12T17:29:17.916 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_CODENAME=bionic 2020-11-12T17:29:17.917 INFO:teuthology.orchestra.run.smithi168.stdout:UBUNTU_CODENAME=bionic 2020-11-12T17:29:17.917 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2020-11-12T17:29:17.937 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-12T17:29:18.098 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi132.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD1OmxiqckuWb2AES8DFyh98yGV+g/eI+KlgbTkN4mPXV2n1NMy4F9ecBgkvOPQyeqmbZFdi1gDukMYNOOejlpuUvfx/12rYTDf6HZq2plrSNLdDuueX2u6ew2c6WAEYn4KjlGYCE9f3mGVnW/q9ZTj2Zkz4LD0piOBHR/Q1kzFF1DfAvS1U+pQErfduhpdKzKbPuLkCzNJEhm9Dwjtpf2D4PDbsUqcEshtHXgrHtyWFzGCs5lx2pQ23SNWxj+ysq5MlBUPvg/kMlV9p0b//d8DazoIf+GOoAZPDFxvObvfTnhC1Kxw5gF1ct/U58RdzQc4Tose450tffESlnnj8BKV smithi168.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3Mvkr26c2zoDrXo/in3+FebaN8T5GkNUTpTuVXLaOaQyjt4uhfafDSGTYvnak82u2AJQX/xrB19LPScfZht/DrvBB8XLPO1oU9cl9YOHQtxBp5sAFgd6kUtg8bwD3AWFzB4YJFOiWcXryt+/2jwm8wK8gxwoiEW/KS8iA7c1+477YonE+5jLO2t0adc0NAhya+LU6VPnIavhJgPPVXusk67XVZnPBSLGjBP1wdvDji7AR0f5jnajZHHWirh6eh5MrBtdTp/NRr3wrSMy1OPaLE5zVXsRP+KYJcWWJ+CDqH07BSYtlBN+d3QQ0t8qJb9XX1zLcgMtjD/3kiHE2AtL7 2020-11-12T17:29:18.099 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-12T17:29:19.835 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-12T17:29:19.856 INFO:teuthology.task.internal:Saving configuration 2020-11-12T17:29:19.885 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-12T17:29:19.957 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-12T17:29:19.974 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-12 17:23:50.954476', 'description': '/home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616032', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL3qVs5ayRC3EE+P2FHcD2GAulovKcRcFmF0BEmdfCY4', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi132.front.sepia.ceph.com'} 2020-11-12T17:29:19.990 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-12 17:23:50.954498', 'description': '/home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616032', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdgumEwQYZZFQEYIXbnPo2hsAOJL4SUFdqgfE3Vaq8x', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi168.front.sepia.ceph.com'} 2020-11-12T17:29:19.991 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-12T17:29:20.010 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2020-11-12T17:29:20.010 INFO:teuthology.task.internal:roles: ubuntu@smithi168.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2020-11-12T17:29:20.010 INFO:teuthology.run_tasks:Running task console_log... 2020-11-12T17:29:25.098 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd826c9a840>, signals=[15]) 2020-11-12T17:29:25.101 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-12T17:29:25.110 INFO:teuthology.task.internal:Opening connections... 2020-11-12T17:29:25.111 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2020-11-12T17:29:25.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:25.197 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi168.front.sepia.ceph.com 2020-11-12T17:29:25.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:25.306 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-12T17:29:27.111 INFO:teuthology.orchestra.run.smithi132:> uname -m 2020-11-12T17:29:27.147 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2020-11-12T17:29:27.148 INFO:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2020-11-12T17:29:27.193 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="Ubuntu" 2020-11-12T17:29:27.193 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-12T17:29:27.193 INFO:teuthology.orchestra.run.smithi132.stdout:ID=ubuntu 2020-11-12T17:29:27.193 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE=debian 2020-11-12T17:29:27.194 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-12T17:29:27.194 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="18.04" 2020-11-12T17:29:27.194 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-12T17:29:27.194 INFO:teuthology.orchestra.run.smithi132.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-12T17:29:27.194 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-12T17:29:27.195 INFO:teuthology.orchestra.run.smithi132.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-12T17:29:27.195 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_CODENAME=bionic 2020-11-12T17:29:27.195 INFO:teuthology.orchestra.run.smithi132.stdout:UBUNTU_CODENAME=bionic 2020-11-12T17:29:27.195 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2020-11-12T17:29:27.212 INFO:teuthology.orchestra.run.smithi168:> uname -m 2020-11-12T17:29:27.218 INFO:teuthology.orchestra.run.smithi168.stdout:x86_64 2020-11-12T17:29:27.219 INFO:teuthology.orchestra.run.smithi168:> cat /etc/os-release 2020-11-12T17:29:27.265 INFO:teuthology.orchestra.run.smithi168.stdout:NAME="Ubuntu" 2020-11-12T17:29:27.266 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-12T17:29:27.266 INFO:teuthology.orchestra.run.smithi168.stdout:ID=ubuntu 2020-11-12T17:29:27.266 INFO:teuthology.orchestra.run.smithi168.stdout:ID_LIKE=debian 2020-11-12T17:29:27.266 INFO:teuthology.orchestra.run.smithi168.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-12T17:29:27.266 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_ID="18.04" 2020-11-12T17:29:27.266 INFO:teuthology.orchestra.run.smithi168.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-12T17:29:27.267 INFO:teuthology.orchestra.run.smithi168.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-12T17:29:27.267 INFO:teuthology.orchestra.run.smithi168.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-12T17:29:27.267 INFO:teuthology.orchestra.run.smithi168.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-12T17:29:27.267 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_CODENAME=bionic 2020-11-12T17:29:27.267 INFO:teuthology.orchestra.run.smithi168.stdout:UBUNTU_CODENAME=bionic 2020-11-12T17:29:27.268 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2020-11-12T17:29:27.280 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-12T17:29:27.337 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-12T17:29:27.348 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-12T17:29:27.351 INFO:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2020-11-12T17:29:27.355 INFO:teuthology.orchestra.run.smithi168:> test '!' -e /home/ubuntu/cephtest 2020-11-12T17:29:27.360 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-12T17:29:27.373 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-12T17:29:27.374 INFO:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2020-11-12T17:29:27.401 INFO:teuthology.orchestra.run.smithi168:> test -z $(ls -A /var/lib/ceph) 2020-11-12T17:29:27.416 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-12T17:29:27.512 INFO:teuthology.run_tasks:Running task kernel... 2020-11-12T17:29:27.591 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}} 2020-11-12T17:29:27.592 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-12T17:29:27.593 INFO:teuthology.orchestra.run.smithi132:> uname -r 2020-11-12T17:29:27.599 INFO:teuthology.orchestra.run.smithi132.stdout:4.15.0-122-generic 2020-11-12T17:29:27.600 INFO:teuthology.task.kernel:Running kernel on smithi132: 4.15.0-122-generic 2020-11-12T17:29:27.600 INFO:teuthology.orchestra.run.smithi132:> sudo apt-get clean 2020-11-12T17:29:27.753 INFO:teuthology.orchestra.run.smithi132:> sudo apt-get update 2020-11-12T17:29:28.102 INFO:teuthology.orchestra.run.smithi132.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-12T17:29:28.102 INFO:teuthology.orchestra.run.smithi132.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-12T17:29:28.166 INFO:teuthology.orchestra.run.smithi132.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-12T17:29:28.433 INFO:teuthology.orchestra.run.smithi132.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-12T17:29:28.501 INFO:teuthology.orchestra.run.smithi132.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [860 kB] 2020-11-12T17:29:28.628 INFO:teuthology.orchestra.run.smithi132.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,155 kB] 2020-11-12T17:29:28.858 INFO:teuthology.orchestra.run.smithi132.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,425 kB] 2020-11-12T17:29:28.961 INFO:teuthology.orchestra.run.smithi132.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [280 kB] 2020-11-12T17:29:28.974 INFO:teuthology.orchestra.run.smithi132.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,752 kB] 2020-11-12T17:29:28.975 INFO:teuthology.orchestra.run.smithi132.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [173 kB] 2020-11-12T17:29:28.983 INFO:teuthology.orchestra.run.smithi132.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [22.7 kB] 2020-11-12T17:29:28.984 INFO:teuthology.orchestra.run.smithi132.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,088 kB] 2020-11-12T17:29:29.028 INFO:teuthology.orchestra.run.smithi132.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [966 kB] 2020-11-12T17:29:29.053 INFO:teuthology.orchestra.run.smithi132.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [243 kB] 2020-11-12T17:29:29.059 INFO:teuthology.orchestra.run.smithi132.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [13.3 kB] 2020-11-12T17:29:29.061 INFO:teuthology.orchestra.run.smithi132.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,808 B] 2020-11-12T17:29:29.061 INFO:teuthology.orchestra.run.smithi132.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,996 B] 2020-11-12T17:29:29.077 INFO:teuthology.orchestra.run.smithi132.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [371 kB] 2020-11-12T17:29:29.094 INFO:teuthology.orchestra.run.smithi132.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [190 kB] 2020-11-12T17:29:29.103 INFO:teuthology.orchestra.run.smithi132.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [25.3 kB] 2020-11-12T17:29:29.103 INFO:teuthology.orchestra.run.smithi132.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-12T17:29:29.205 INFO:teuthology.orchestra.run.smithi132.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,546 kB] 2020-11-12T17:29:29.271 INFO:teuthology.orchestra.run.smithi132.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [355 kB] 2020-11-12T17:29:29.281 INFO:teuthology.orchestra.run.smithi132.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [12.9 kB] 2020-11-12T17:29:29.281 INFO:teuthology.orchestra.run.smithi132.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [32.3 kB] 2020-11-12T17:29:29.282 INFO:teuthology.orchestra.run.smithi132.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,012 B] 2020-11-12T17:29:30.764 INFO:teuthology.orchestra.run.smithi132.stdout:Fetched 12.5 MB in 2s (5,171 kB/s) 2020-11-12T17:29:32.163 INFO:teuthology.orchestra.run.smithi132.stdout:Reading package lists... 2020-11-12T17:29:32.229 INFO:teuthology.orchestra.run.smithi132:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-12T17:29:32.319 INFO:teuthology.orchestra.run.smithi132.stdout:Reading package lists... 2020-11-12T17:29:32.583 INFO:teuthology.orchestra.run.smithi132.stdout:Building dependency tree... 2020-11-12T17:29:32.584 INFO:teuthology.orchestra.run.smithi132.stdout:Reading state information... 2020-11-12T17:29:32.665 INFO:teuthology.orchestra.run.smithi132.stderr:E: Unable to locate package linux-image-current-generic 2020-11-12T17:29:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-12T17:29:32.682 INFO:teuthology.orchestra.run.smithi132:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-12T17:29:32.765 INFO:teuthology.orchestra.run.smithi132.stdout:Reading package lists... 2020-11-12T17:29:33.019 INFO:teuthology.orchestra.run.smithi132.stdout:Building dependency tree... 2020-11-12T17:29:33.020 INFO:teuthology.orchestra.run.smithi132.stdout:Reading state information... 2020-11-12T17:29:33.192 INFO:teuthology.orchestra.run.smithi132.stdout:The following additional packages will be installed: 2020-11-12T17:29:33.193 INFO:teuthology.orchestra.run.smithi132.stdout: linux-generic linux-headers-4.15.0-123 linux-headers-4.15.0-123-generic 2020-11-12T17:29:33.193 INFO:teuthology.orchestra.run.smithi132.stdout: linux-headers-generic linux-image-4.15.0-123-generic 2020-11-12T17:29:33.193 INFO:teuthology.orchestra.run.smithi132.stdout: linux-modules-4.15.0-123-generic linux-modules-extra-4.15.0-123-generic 2020-11-12T17:29:33.195 INFO:teuthology.orchestra.run.smithi132.stdout:Suggested packages: 2020-11-12T17:29:33.195 INFO:teuthology.orchestra.run.smithi132.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-12T17:29:33.195 INFO:teuthology.orchestra.run.smithi132.stdout:Recommended packages: 2020-11-12T17:29:33.398 INFO:teuthology.orchestra.run.smithi132.stdout: thermald 2020-11-12T17:29:33.400 INFO:teuthology.orchestra.run.smithi132.stdout:The following NEW packages will be installed: 2020-11-12T17:29:33.400 INFO:teuthology.orchestra.run.smithi132.stdout: linux-headers-4.15.0-123 linux-headers-4.15.0-123-generic 2020-11-12T17:29:33.400 INFO:teuthology.orchestra.run.smithi132.stdout: linux-image-4.15.0-123-generic linux-modules-4.15.0-123-generic 2020-11-12T17:29:33.401 INFO:teuthology.orchestra.run.smithi132.stdout: linux-modules-extra-4.15.0-123-generic 2020-11-12T17:29:33.401 INFO:teuthology.orchestra.run.smithi132.stdout:The following packages will be upgraded: 2020-11-12T17:29:33.401 INFO:teuthology.orchestra.run.smithi132.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-12T17:29:33.403 INFO:teuthology.orchestra.run.smithi132.stdout:3 upgraded, 5 newly installed, 0 to remove and 14 not upgraded. 2020-11-12T17:29:33.404 INFO:teuthology.orchestra.run.smithi132.stdout:Need to get 66.0 MB of archives. 2020-11-12T17:29:33.404 INFO:teuthology.orchestra.run.smithi132.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-12T17:29:33.404 INFO:teuthology.orchestra.run.smithi132.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-123-generic amd64 4.15.0-123.126 [13.1 MB] 2020-11-12T17:29:34.455 INFO:teuthology.orchestra.run.smithi132.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-123-generic amd64 4.15.0-123.126 [8,012 kB] 2020-11-12T17:29:34.698 INFO:teuthology.orchestra.run.smithi132.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-123-generic amd64 4.15.0-123.126 [32.8 MB] 2020-11-12T17:29:35.915 INFO:teuthology.orchestra.run.smithi132.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.123.110 [1,872 B] 2020-11-12T17:29:35.916 INFO:teuthology.orchestra.run.smithi132.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.123.110 [2,644 B] 2020-11-12T17:29:35.916 INFO:teuthology.orchestra.run.smithi132.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-123 all 4.15.0-123.126 [10.9 MB] 2020-11-12T17:29:36.345 INFO:teuthology.orchestra.run.smithi132.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-123-generic amd64 4.15.0-123.126 [1,105 kB] 2020-11-12T17:29:36.546 INFO:teuthology.orchestra.run.smithi132.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.123.110 [2,544 B] 2020-11-12T17:29:36.861 INFO:teuthology.orchestra.run.smithi132.stdout:Fetched 66.0 MB in 3s (21.0 MB/s) 2020-11-12T17:29:36.990 INFO:teuthology.orchestra.run.smithi132.stdout:Selecting previously unselected package linux-modules-4.15.0-123-generic. 2020-11-12T17:29:37.027 INFO:teuthology.orchestra.run.smithi132.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102071 files and directories currently installed.) 2020-11-12T17:29:37.030 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../0-linux-modules-4.15.0-123-generic_4.15.0-123.126_amd64.deb ... 2020-11-12T17:29:37.077 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-modules-4.15.0-123-generic (4.15.0-123.126) ... 2020-11-12T17:29:38.310 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-12T17:29:38.330 DEBUG:teuthology.task.console_log:Killing console logger for smithi132 2020-11-12T17:29:38.355 DEBUG:teuthology.task.console_log:Killing console logger for smithi168 2020-11-12T17:29:38.355 DEBUG:teuthology.task.console_log:Killing console logger for smithi132 2020-11-12T17:29:38.355 DEBUG:teuthology.task.console_log:Killing console logger for smithi168 2020-11-12T17:29:38.356 DEBUG:teuthology.exit:Finished running handlers