2020-06-12T02:24:38.919 INFO:root:teuthology version: 1.0.0-6c966fe 2020-06-12T02:24:38.921 DEBUG:teuthology.run:Teuthology command: teuthology --block --name kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi --lock --owner scheduled_kchai@teuthology --verbose --description rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} --archive /home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138040 -- /tmp/teuthology-worker.1ip0_bea.tmp 2020-06-12T02:24:38.956 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-12T02:24:39.026 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138040 branch: octopus description: rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '5138040' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: octopus ceph: conf: global: mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: octopus-py3 sha1: 1e0d5d60071d5554f87a2be72ea0bcccb272aab3 owner: scheduled_kchai@teuthology priority: 90 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 sleep_before_teardown: 0 suite: rados suite_branch: octopus-py3 suite_path: /home/teuthworker/src/github.com_tchaikov_ceph_octopus-py3/qa suite_relpath: qa suite_repo: https://github.com/tchaikov/ceph.git suite_sha1: 1e0d5d60071d5554f87a2be72ea0bcccb272aab3 tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27823 2020-06-12T02:24:39.026 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_tchaikov_ceph_octopus-py3/qa; will attempt to use it 2020-06-12T02:24:39.027 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_tchaikov_ceph_octopus-py3/qa/tasks 2020-06-12T02:24:39.028 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-06-12T02:24:39.030 INFO:teuthology.task.internal:Checking packages... 2020-06-12T02:24:39.054 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '91d24f498556f86f6d1318d2efd5210c35b36762' 2020-06-12T02:24:39.055 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-06-12T02:24:39.058 INFO:teuthology.packaging:ref: None 2020-06-12T02:24:39.058 INFO:teuthology.packaging:tag: None 2020-06-12T02:24:39.059 INFO:teuthology.packaging:branch: octopus 2020-06-12T02:24:39.059 INFO:teuthology.packaging:sha1: 91d24f498556f86f6d1318d2efd5210c35b36762 2020-06-12T02:24:39.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2020-06-12T02:24:39.328 INFO:teuthology.task.internal:Found packages for ceph version 15.2.3-298.g7488806bf1 2020-06-12T02:24:39.329 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-06-12T02:24:39.350 INFO:teuthology.task.internal:no buildpackages task found 2020-06-12T02:24:39.351 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-06-12T02:24:39.367 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-06-12T02:24:39.368 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-12T02:24:39.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-06-12T02:24:49.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:24:59.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:25:09.480 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:25:19.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:25:29.529 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:25:39.548 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:25:49.566 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:25:59.584 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:26:09.612 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:26:19.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:26:29.653 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:26:39.674 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:26:49.693 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:26:59.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:27:09.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-06-12T02:27:19.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:27:29.791 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:27:39.808 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:27:49.844 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:27:59.863 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-06-12T02:28:09.887 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-06-12T02:28:19.911 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 2)... 2020-06-12T02:28:29.937 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:28:39.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:28:49.974 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:29:00.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:29:10.024 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:29:20.051 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:29:30.087 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:29:40.105 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:29:50.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:30:00.149 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:30:10.170 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 2)... 2020-06-12T02:30:20.191 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 2)... 2020-06-12T02:30:30.213 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:30:40.233 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 2)... 2020-06-12T02:30:50.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:31:00.287 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:31:10.316 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:31:20.348 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-06-12T02:31:30.370 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:31:40.389 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:31:50.408 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:32:00.428 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:32:10.448 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:32:20.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:32:30.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:32:40.507 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-06-12T02:32:50.528 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:33:00.544 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:33:10.569 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 2)... 2020-06-12T02:33:20.592 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 3)... 2020-06-12T02:33:30.615 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:33:40.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:33:50.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:34:00.665 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:34:10.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 2)... 2020-06-12T02:34:20.706 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-06-12T02:34:30.724 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 4, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138040'} 2020-06-12T02:34:30.766 DEBUG:teuthology.lock.ops:locked ubuntu@smithi180.front.sepia.ceph.com, ubuntu@smithi005.front.sepia.ceph.com, ubuntu@smithi097.front.sepia.ceph.com, ubuntu@smithi131.front.sepia.ceph.com 2020-06-12T02:34:30.940 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd1fe41c598>, signals=[15]) 2020-06-12T02:34:31.162 INFO:teuthology.provision.fog.smithi180:Scheduling deploy of centos 8.1 2020-06-12T02:34:31.171 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.1 2020-06-12T02:34:31.175 INFO:teuthology.provision.fog.smithi131:Scheduling deploy of centos 8.1 2020-06-12T02:34:31.178 INFO:teuthology.provision.fog.smithi005:Scheduling deploy of centos 8.1 2020-06-12T02:34:31.636 INFO:teuthology.orchestra.console:Power off smithi180 2020-06-12T02:34:31.638 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-12T02:34:31.668 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:31.701 INFO:teuthology.orchestra.console:Power off smithi005 2020-06-12T02:34:31.702 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-12T02:34:31.741 INFO:teuthology.orchestra.console:Power off smithi131 2020-06-12T02:34:31.742 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-12T02:34:31.766 INFO:teuthology.orchestra.console:Power off smithi097 2020-06-12T02:34:31.767 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-12T02:34:31.780 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:31.791 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:31.804 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:35.806 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:35.821 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:35.838 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:35.852 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:39.926 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:39.952 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:39.974 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:39.987 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:40.153 INFO:teuthology.orchestra.console:Power off for smithi180 completed 2020-06-12T02:34:40.173 INFO:teuthology.orchestra.console:Power off for smithi005 completed 2020-06-12T02:34:40.189 INFO:teuthology.orchestra.console:Power off for smithi131 completed 2020-06-12T02:34:40.206 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2020-06-12T02:34:40.256 INFO:teuthology.orchestra.console:Power on smithi180 2020-06-12T02:34:40.256 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-12T02:34:40.274 INFO:teuthology.orchestra.console:Power on smithi005 2020-06-12T02:34:40.275 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-12T02:34:40.286 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:40.298 INFO:teuthology.orchestra.console:Power on smithi131 2020-06-12T02:34:40.299 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-12T02:34:40.316 INFO:teuthology.orchestra.console:Power on smithi097 2020-06-12T02:34:40.317 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-12T02:34:40.329 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:40.344 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:40.364 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:44.347 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi180.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:44.378 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:44.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:44.414 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:44.566 INFO:teuthology.orchestra.console:Power on for smithi180 completed 2020-06-12T02:34:44.667 INFO:teuthology.provision.fog.smithi180:Waiting for deploy to finish 2020-06-12T02:34:48.495 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:48.516 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:48.532 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-12T02:34:48.718 INFO:teuthology.orchestra.console:Power on for smithi005 completed 2020-06-12T02:34:48.733 INFO:teuthology.orchestra.console:Power on for smithi097 completed 2020-06-12T02:34:48.752 INFO:teuthology.orchestra.console:Power on for smithi131 completed 2020-06-12T02:34:48.820 INFO:teuthology.provision.fog.smithi005:Waiting for deploy to finish 2020-06-12T02:34:48.835 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish 2020-06-12T02:34:48.854 INFO:teuthology.provision.fog.smithi131:Waiting for deploy to finish 2020-06-12T02:37:04.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:14.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:15.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:20.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:20.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:23.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:25.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:29.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:29.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:32.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:34.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:38.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:38.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:41.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:43.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:47.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:47.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:50.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:52.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:56.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:37:56.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:38:01.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:38:05.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:38:05.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:38:14.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:38:14.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:38:56.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:38:56.686 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-12T02:38:56.688 INFO:teuthology.orchestra.run.smithi005:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-12T02:38:56.985 INFO:teuthology.provision.fog.smithi005:Node is ready 2020-06-12T02:38:56.986 INFO:teuthology.orchestra.run.smithi005:> hostname 2020-06-12T02:38:57.005 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005.front.sepia.ceph.com 2020-06-12T02:38:57.006 INFO:teuthology.orchestra.run.smithi005:> grep smithi005.front.sepia.ceph.com /etc/hosts 2020-06-12T02:38:57.060 INFO:teuthology.orchestra.run.smithi005.stdout:172.21.15.5 smithi005.front.sepia.ceph.com smithi005 2020-06-12T02:38:57.061 INFO:teuthology.orchestra.run.smithi005:> sudo hostname smithi005 2020-06-12T02:38:57.175 INFO:teuthology.orchestra.run.smithi005:> sudo sed -i -e 's/smithi005.front.sepia.ceph.com/smithi005/g' /etc/hosts 2020-06-12T02:38:57.209 INFO:teuthology.orchestra.run.smithi005:> sudo sed -i -e 's/172.21.15.5/172.21.15.5/g' /etc/hosts 2020-06-12T02:38:57.280 INFO:teuthology.orchestra.run.smithi005:> sudo sed -i -e 's/smithi005.front.sepia.ceph.com/smithi005/g' /etc/hostname 2020-06-12T02:38:57.351 INFO:teuthology.orchestra.run.smithi005:> sudo hostname smithi005 2020-06-12T02:38:57.430 INFO:teuthology.provision.fog.smithi005:Deploy complete! 2020-06-12T02:38:57.431 DEBUG:teuthology.parallel:result is None 2020-06-12T02:39:07.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:07.520 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-12T02:39:07.521 INFO:teuthology.orchestra.run.smithi180:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-12T02:39:07.855 INFO:teuthology.provision.fog.smithi180:Node is ready 2020-06-12T02:39:07.856 INFO:teuthology.orchestra.run.smithi180:> hostname 2020-06-12T02:39:07.870 INFO:teuthology.orchestra.run.smithi180.stdout:smithi180.front.sepia.ceph.com 2020-06-12T02:39:07.871 INFO:teuthology.orchestra.run.smithi180:> grep smithi180.front.sepia.ceph.com /etc/hosts 2020-06-12T02:39:07.924 INFO:teuthology.orchestra.run.smithi180.stdout:172.21.15.180 smithi180.front.sepia.ceph.com smithi180 2020-06-12T02:39:07.924 INFO:teuthology.orchestra.run.smithi180:> sudo hostname smithi180 2020-06-12T02:39:08.033 INFO:teuthology.orchestra.run.smithi180:> sudo sed -i -e 's/smithi180.front.sepia.ceph.com/smithi180/g' /etc/hosts 2020-06-12T02:39:08.062 INFO:teuthology.orchestra.run.smithi180:> sudo sed -i -e 's/172.21.15.180/172.21.15.180/g' /etc/hosts 2020-06-12T02:39:08.134 INFO:teuthology.orchestra.run.smithi180:> sudo sed -i -e 's/smithi180.front.sepia.ceph.com/smithi180/g' /etc/hostname 2020-06-12T02:39:08.201 INFO:teuthology.orchestra.run.smithi180:> sudo hostname smithi180 2020-06-12T02:39:08.273 INFO:teuthology.provision.fog.smithi180:Deploy complete! 2020-06-12T02:39:08.274 DEBUG:teuthology.parallel:result is None 2020-06-12T02:39:20.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:20.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:20.827 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-12T02:39:20.828 INFO:teuthology.orchestra.run.smithi131:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-12T02:39:20.894 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-12T02:39:20.895 INFO:teuthology.orchestra.run.smithi097:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-12T02:39:21.127 INFO:teuthology.provision.fog.smithi131:Node is ready 2020-06-12T02:39:21.128 INFO:teuthology.orchestra.run.smithi131:> hostname 2020-06-12T02:39:21.145 INFO:teuthology.orchestra.run.smithi131.stdout:smithi131.front.sepia.ceph.com 2020-06-12T02:39:21.145 INFO:teuthology.orchestra.run.smithi131:> grep smithi131.front.sepia.ceph.com /etc/hosts 2020-06-12T02:39:21.184 INFO:teuthology.provision.fog.smithi097:Node is ready 2020-06-12T02:39:21.184 INFO:teuthology.orchestra.run.smithi097:> hostname 2020-06-12T02:39:21.198 INFO:teuthology.orchestra.run.smithi131.stdout:172.21.15.131 smithi131.front.sepia.ceph.com smithi131 2020-06-12T02:39:21.199 INFO:teuthology.orchestra.run.smithi131:> sudo hostname smithi131 2020-06-12T02:39:21.200 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com 2020-06-12T02:39:21.201 INFO:teuthology.orchestra.run.smithi097:> grep smithi097.front.sepia.ceph.com /etc/hosts 2020-06-12T02:39:21.255 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097 2020-06-12T02:39:21.255 INFO:teuthology.orchestra.run.smithi097:> sudo hostname smithi097 2020-06-12T02:39:21.308 INFO:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hosts 2020-06-12T02:39:21.336 INFO:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/172.21.15.131/172.21.15.131/g' /etc/hosts 2020-06-12T02:39:21.368 INFO:teuthology.orchestra.run.smithi097:> sudo sed -i -e 's/smithi097.front.sepia.ceph.com/smithi097/g' /etc/hosts 2020-06-12T02:39:21.403 INFO:teuthology.orchestra.run.smithi097:> sudo sed -i -e 's/172.21.15.97/172.21.15.97/g' /etc/hosts 2020-06-12T02:39:21.408 INFO:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hostname 2020-06-12T02:39:21.470 INFO:teuthology.orchestra.run.smithi131:> sudo hostname smithi131 2020-06-12T02:39:21.471 INFO:teuthology.orchestra.run.smithi097:> sudo sed -i -e 's/smithi097.front.sepia.ceph.com/smithi097/g' /etc/hostname 2020-06-12T02:39:21.536 INFO:teuthology.provision.fog.smithi131:Deploy complete! 2020-06-12T02:39:21.536 DEBUG:teuthology.parallel:result is None 2020-06-12T02:39:21.538 INFO:teuthology.orchestra.run.smithi097:> sudo hostname smithi097 2020-06-12T02:39:21.604 INFO:teuthology.provision.fog.smithi097:Deploy complete! 2020-06-12T02:39:21.605 DEBUG:teuthology.parallel:result is None 2020-06-12T02:39:21.802 INFO:teuthology.lock.ops:Checking smithi180.front.sepia.ceph.com 2020-06-12T02:39:21.804 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-12T02:39:21.833 INFO:teuthology.lock.ops:Checking smithi005.front.sepia.ceph.com 2020-06-12T02:39:21.834 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-12T02:39:21.853 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com 2020-06-12T02:39:21.853 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-12T02:39:21.873 INFO:teuthology.lock.ops:Checking smithi131.front.sepia.ceph.com 2020-06-12T02:39:21.874 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-12T02:39:21.894 INFO:teuthology.lock.ops:Updating [smithi180.front.sepia.ceph.com]: set os type and version on server 2020-06-12T02:39:21.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:39:21.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:21.968 INFO:teuthology.orchestra.run.smithi180:> true 2020-06-12T02:39:22.136 INFO:teuthology.orchestra.run.smithi180:> uname -m 2020-06-12T02:39:22.190 INFO:teuthology.orchestra.run.smithi180.stdout:x86_64 2020-06-12T02:39:22.191 INFO:teuthology.orchestra.run.smithi180:> cat /etc/os-release 2020-06-12T02:39:22.245 INFO:teuthology.orchestra.run.smithi180.stdout:NAME="CentOS Linux" 2020-06-12T02:39:22.245 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION="8 (Core)" 2020-06-12T02:39:22.246 INFO:teuthology.orchestra.run.smithi180.stdout:ID="centos" 2020-06-12T02:39:22.246 INFO:teuthology.orchestra.run.smithi180.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:39:22.246 INFO:teuthology.orchestra.run.smithi180.stdout:VERSION_ID="8" 2020-06-12T02:39:22.247 INFO:teuthology.orchestra.run.smithi180.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:39:22.247 INFO:teuthology.orchestra.run.smithi180.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:39:22.248 INFO:teuthology.orchestra.run.smithi180.stdout:ANSI_COLOR="0;31" 2020-06-12T02:39:22.248 INFO:teuthology.orchestra.run.smithi180.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:39:22.248 INFO:teuthology.orchestra.run.smithi180.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:39:22.250 INFO:teuthology.orchestra.run.smithi180.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:39:22.251 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-06-12T02:39:22.251 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:39:22.251 INFO:teuthology.orchestra.run.smithi180.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:39:22.252 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:39:22.252 INFO:teuthology.orchestra.run.smithi180.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:39:22.252 INFO:teuthology.orchestra.run.smithi180.stdout: 2020-06-12T02:39:22.254 INFO:teuthology.lock.ops:Updating smithi180.front.sepia.ceph.com on lock server 2020-06-12T02:39:22.281 INFO:teuthology.lock.ops:Updating [smithi005.front.sepia.ceph.com]: set os type and version on server 2020-06-12T02:39:22.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:39:22.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:22.394 INFO:teuthology.orchestra.run.smithi005:> true 2020-06-12T02:39:22.587 INFO:teuthology.orchestra.run.smithi005:> uname -m 2020-06-12T02:39:22.642 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2020-06-12T02:39:22.643 INFO:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2020-06-12T02:39:22.698 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Linux" 2020-06-12T02:39:22.699 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="8 (Core)" 2020-06-12T02:39:22.699 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2020-06-12T02:39:22.699 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:39:22.700 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="8" 2020-06-12T02:39:22.700 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:39:22.700 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:39:22.700 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2020-06-12T02:39:22.700 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:39:22.700 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:39:22.701 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:39:22.701 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-06-12T02:39:22.701 INFO:teuthology.orchestra.run.smithi005.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:39:22.701 INFO:teuthology.orchestra.run.smithi005.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:39:22.702 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:39:22.702 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:39:22.710 INFO:teuthology.orchestra.run.smithi005.stdout: 2020-06-12T02:39:22.711 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2020-06-12T02:39:22.747 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server 2020-06-12T02:39:22.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:39:22.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:22.819 INFO:teuthology.orchestra.run.smithi097:> true 2020-06-12T02:39:22.988 INFO:teuthology.orchestra.run.smithi097:> uname -m 2020-06-12T02:39:23.042 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2020-06-12T02:39:23.043 INFO:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2020-06-12T02:39:23.097 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Linux" 2020-06-12T02:39:23.097 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8 (Core)" 2020-06-12T02:39:23.098 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2020-06-12T02:39:23.098 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:39:23.098 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2020-06-12T02:39:23.098 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:39:23.098 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:39:23.099 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2020-06-12T02:39:23.099 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:39:23.099 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:39:23.099 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:39:23.099 INFO:teuthology.orchestra.run.smithi097.stdout: 2020-06-12T02:39:23.100 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:39:23.100 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:39:23.100 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:39:23.100 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:39:23.100 INFO:teuthology.orchestra.run.smithi097.stdout: 2020-06-12T02:39:23.101 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2020-06-12T02:39:23.131 INFO:teuthology.lock.ops:Updating [smithi131.front.sepia.ceph.com]: set os type and version on server 2020-06-12T02:39:23.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-12T02:39:23.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:23.205 INFO:teuthology.orchestra.run.smithi131:> true 2020-06-12T02:39:23.295 INFO:teuthology.orchestra.run.smithi131:> uname -m 2020-06-12T02:39:23.349 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2020-06-12T02:39:23.350 INFO:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2020-06-12T02:39:23.405 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Linux" 2020-06-12T02:39:23.405 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="8 (Core)" 2020-06-12T02:39:23.405 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2020-06-12T02:39:23.405 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2020-06-12T02:39:23.406 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="8" 2020-06-12T02:39:23.406 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el8" 2020-06-12T02:39:23.406 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-06-12T02:39:23.406 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2020-06-12T02:39:23.406 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-06-12T02:39:23.407 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://www.centos.org/" 2020-06-12T02:39:23.407 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-12T02:39:23.407 INFO:teuthology.orchestra.run.smithi131.stdout: 2020-06-12T02:39:23.407 INFO:teuthology.orchestra.run.smithi131.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-06-12T02:39:23.407 INFO:teuthology.orchestra.run.smithi131.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-06-12T02:39:23.408 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-12T02:39:23.408 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-06-12T02:39:23.408 INFO:teuthology.orchestra.run.smithi131.stdout: 2020-06-12T02:39:23.409 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2020-06-12T02:39:23.434 INFO:teuthology.task.internal.lock_machines:4 smithi machines locked this try, 4/4 locked so far 2020-06-12T02:39:23.508 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1mZrXLvKH9A9kUSsheMjsVjwz9Y2k3dQStPRVZiKBE3IbfOhMn7mmuKKPNU4El8id5IX9ltKrTVe0cfCwj6/UhOh/Gw2aRb4cYLfjQnRyNsIt1CNHlldmftBT5u8zzEY7mIofNbfhNGFJwwWTNt3xvktISaq2s0D2M6DOAv3ROJYnZcuj1DZ3ZDQy6uAY/aCcamZSAKbs0POKUMENh0koXcYFOE8oCRcpj3ab44jzQUo/f0kz3K8aUpOzcKANCn9tinIu2SlU6UxiekV/vmf8BkCSzM71NLiWASvtLMeBkZeXN4LAxWPGIqAMZTE66OvI5dv8PHwD1B8bMWyz7okxJwNIJh/vPD6LBHImLzPYER5AmDOKR5HJ3IotCimHK0sPcEOc0cYwYDoVGZc+3ep95A+XjQATjqpY19WkD0Sv9VjVIirgEJWf+1qNhVg/EnWSrAJDgf9HFA9uXu8LeuysvW4HxiRxYQlFvfV3M2M8qB50qQX41jWDTyf9KIpF888= smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6mlGobCxExFn3q3Tm3wIBoAYZP/M7WS94s/sQ7DML8ajQoHbS1Gr+i5KrymckwhuqhRDdsgPtiOiWpyKw3Z4pmBS+83fTAu6WfBahGXFZVTOVGwzVc/H7dVnaiwwQ++0YecX458oxzH0GyEOGf4OyZoTnV1zSNdQh6Y3RAQdNKCYroBJcCUiTBpV/i4/GGHruCz7vjbCxsiCLCwmAjXxoQQs8omQsAZWmqfFJ1uyGmWvgrIScikquVNpBF6z6PleA4CG6dhSNJpn8k6MPLUvvMlUS+9GSKxavMMcB3F3bDC3uSxIzJWkCwSlOxx/Cz6XZk0umfkg/HK3oJL4UyEbZNe/dF+lz26/zavJFy76tBuMyyCLDeCi0gxfmHerZwcPZKjW44U9FPjnjP50WjTu7ZcdULXe93Rkm5D1cZ+zFBx24jjeL9eKqpGISG8VpYURhTellz0qh3vvVLDtAoDYHIZDMlArn3W6w1TmeUeRl5yfnIeQmSnL1YgqeDmxAXkU= smithi131.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMkZZTRsBZAonwDG+bRB/PR+PcFNQg/TW88P2aXnmuK0uFfC1f0f4dSItttVunkUipwH0NaTwltuiHxRTp394jiw428sIgDJ8dTc6nFVOaegGhUKzyMdfyK2PDKQ9Kq02yikylUUZfwbEdUk80f1mwiHeMgOQNquAIhrqKC5oS6r8m8uSHSGxG8JFsi48ZiW2rcE1Xi9lHWIufYFBUtRlrHjOaT2ns2rtok36I4ZDk13Zd29jxJ1wtPFUtelUnpR24MgUzxMzudKPv9iyqkmlw9kI2X81yBP9XaNiLfhHh9S5/NNgQ7k6GvTvjttTexZc5co5Zcf1qmLpoJZ+MVYQr2UdimqjrvNIdPKMqJZkpEmP6B4hnUGF+3+iWX7LmYnzZcOioHjFQPFeFU5PWzf/6kq7v3NBZh8CmkqYDicf/z2MucL/g5/fdX01frodenKs78G0Cj1m5qOv8DAtg4m1Im/zbQzvnQaGHngLlKUxyAE9yF4JEWk3gzFJaQqrqBAM= smithi180.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1hDtqUzJIQReMJW0egVSJeHNaucZrnx3TSMHiPNIs2pHlGHQ4BEJPAz5c/wBIE/oMx6loJJCkWm4zh6PBE0C0nPVCEa1hY3myyRphLiu/pn8FIossHPF5vZN4hKnJk6XFyTeOcF9YbYULi0aWO3F5CrOoCbnWezO6xoZf8VTHjjjuupv4k8GhOMebFRgH6YvIsGdsRVVlT8sVg181aChaJ5uOd+DQ+55Tf+I0Mgjki2n7zl3mQ6HxTt2+o98VyTeOcjh6Dw6O1rJxcdeSgdI/va+aA8eukYR/Hez/5Oz2/llod+UHjbRpCXh9CTAnKgWNvWUDAFx3I6CDkZl90r5rTwNnPpKRzHpzqciOW/BodPDF35/bUf4O/Kx6a0Sdukj3V4ncqXAGOMl99PWEQvJGgvZZ1c9/vFa8eOGHGFKf2PoH9Ok7mJmQK5brdeYoyH3v1y8AaPwhqRQtL1t2G/9pQgMhSIJqVmRvpTIleBKlSM1NJ6hrUwtKie4e+olICmc= 2020-06-12T02:39:23.509 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-12T02:39:23.568 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-06-12T02:39:23.577 INFO:teuthology.task.internal:Saving configuration 2020-06-12T02:39:23.599 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-06-12T02:39:23.613 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-06-12T02:39:23.630 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-12 02:34:30.733688', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138040', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuOANNmEI+r3hljfjE6qYBiZms+W9R/6JJ6w8Lbt6cXK/ysrCuJySYXfPuOKpksZdk8gqsHmhfbR8x3d5Yizlo=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi180.front.sepia.ceph.com'} 2020-06-12T02:39:23.646 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-12 02:34:30.733709', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138040', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM8qliNUc/tNNVIy059bYfeKwgEdcVUNhsKsWMqcoUCtUidBPhTSGeDhHrR0eY1lMutU42ioWnZninBcuPpFlCs=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi005.front.sepia.ceph.com'} 2020-06-12T02:39:23.661 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-12 02:34:30.733725', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138040', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJ6r9X7ACSegNPXTUJUeXoBT+G1ib49AT9gqNkeaJqBcim5+aEXLo+1tsSytavf6jkkouHOtWHsUqjVxILQwTI=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi097.front.sepia.ceph.com'} 2020-06-12T02:39:23.676 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-12 02:34:30.733740', 'description': '/home/teuthworker/archive/kchai-2020-06-11_10:13:29-rados-octopus-distro-basic-smithi/5138040', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM3VLwNT8UMJiQWwnYHYJ0PmA3Q6rmQeZoKkbGL8FvjGgxbkn3a34uNhippfPieVgJDH/d+xkLTNdNPN/DpqhYE=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi131.front.sepia.ceph.com'} 2020-06-12T02:39:23.677 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-06-12T02:39:23.690 INFO:teuthology.task.internal:roles: ubuntu@smithi180.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2020-06-12T02:39:23.690 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2020-06-12T02:39:23.690 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2020-06-12T02:39:23.691 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2020-06-12T02:39:23.691 INFO:teuthology.run_tasks:Running task console_log... 2020-06-12T02:39:23.863 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd20241c0d0>, signals=[15]) 2020-06-12T02:39:23.865 INFO:teuthology.run_tasks:Running task internal.connect... 2020-06-12T02:39:23.882 INFO:teuthology.task.internal:Opening connections... 2020-06-12T02:39:23.883 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi180.front.sepia.ceph.com 2020-06-12T02:39:23.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi180.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:23.960 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2020-06-12T02:39:23.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:24.049 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2020-06-12T02:39:24.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:24.123 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2020-06-12T02:39:24.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-12T02:39:24.197 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-06-12T02:39:24.207 INFO:teuthology.orchestra.run.smithi180:> uname -m 2020-06-12T02:39:24.223 INFO:teuthology.orchestra.run.smithi180.stdout:x86_64 2020-06-12T02:39:24.224 INFO:teuthology.orchestra.run.smithi180:> cat /etc/os-release 2020-06-12T02:39:24.239 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-06-12T02:39:24.240 DEBUG:teuthology.task.console_log:Killing console logger for smithi180 2020-06-12T02:39:24.240 DEBUG:teuthology.task.console_log:Killing console logger for smithi005 2020-06-12T02:39:24.241 DEBUG:teuthology.task.console_log:Killing console logger for smithi097 2020-06-12T02:39:24.241 DEBUG:teuthology.task.console_log:Killing console logger for smithi131 2020-06-12T02:39:24.242 DEBUG:teuthology.task.console_log:Killing console logger for smithi180 2020-06-12T02:39:24.242 DEBUG:teuthology.task.console_log:Killing console logger for smithi005 2020-06-12T02:39:24.242 DEBUG:teuthology.task.console_log:Killing console logger for smithi097 2020-06-12T02:39:24.243 DEBUG:teuthology.task.console_log:Killing console logger for smithi131 2020-06-12T02:39:24.243 DEBUG:teuthology.exit:Finished running handlers