2020-08-31T22:06:05.209 INFO:root:teuthology version: 1.0.0-ec650ed 2020-08-31T22:06:05.212 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --lock --owner scheduled_pdonnell@teuthology --description kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_dbench} --block --name pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi --archive /home/teuthworker/archive/pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396249 -- /tmp/teuthology-worker.ri9xiowh.tmp 2020-08-31T22:06:05.488 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-31T22:06:05.558 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396249 branch: wip-pdonnell-testing-20200829.172552 description: kcephfs/cephfs/{begin clusters/1-mds-1-client conf/{client mds mon osd} inline/no kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kclient_workunit_suites_dbench} email: pdonnell@redhat.com first_in_suite: false job_id: '5396249' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-pdonnell-testing-20200829.172552 ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20200829.172552 sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf sleep_before_teardown: 0 suite: kcephfs suite_branch: wip-pdonnell-testing-20200829.172552 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.172552/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - workunit: clients: all: - suites/dbench.sh teuthology_branch: refs/pull/1554/merge timestamp: 2020-08-31_21:26:19 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.12113 2020-08-31T22:06:05.558 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.172552/qa; will attempt to use it 2020-08-31T22:06:05.559 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.172552/qa/tasks 2020-08-31T22:06:05.560 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-08-31T22:06:05.562 INFO:teuthology.task.internal:Checking packages... 2020-08-31T22:06:05.587 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf' 2020-08-31T22:06:05.588 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-08-31T22:06:05.588 INFO:teuthology.packaging:ref: None 2020-08-31T22:06:05.589 INFO:teuthology.packaging:tag: None 2020-08-31T22:06:05.589 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20200829.172552 2020-08-31T22:06:05.590 INFO:teuthology.packaging:sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf 2020-08-31T22:06:05.590 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20200829.172552 2020-08-31T22:06:05.765 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-4885.gc5af0d5 2020-08-31T22:06:05.765 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-08-31T22:06:05.777 INFO:teuthology.task.internal:no buildpackages task found 2020-08-31T22:06:05.778 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-08-31T22:06:05.887 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-08-31T22:06:05.888 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-31T22:06:05.943 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:06:15.959 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:06:25.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:06:35.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:06:46.013 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:06:56.037 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:07:06.059 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:07:16.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-08-31T22:07:26.099 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-08-31T22:07:36.114 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-08-31T22:07:46.132 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-08-31T22:07:56.151 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396249'} 2020-08-31T22:07:56.162 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 3 smithi nodes. 2020-08-31T22:07:56.163 ERROR:teuthology.lock.ops:{"message": "only 0 nodes available"} 2020-08-31T22:07:56.163 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/3 locked so far 2020-08-31T22:07:56.164 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 3 more 2020-08-31T22:07:56.164 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting... 2020-08-31T22:08:06.193 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:08:16.217 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:08:26.235 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:08:36.257 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-08-31T22:08:46.276 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-08-31T22:08:56.295 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-08-31T22:09:06.312 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-08-31T22:09:16.333 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396249'} 2020-08-31T22:09:16.353 DEBUG:teuthology.lock.ops:locked ubuntu@smithi146.front.sepia.ceph.com, ubuntu@smithi172.front.sepia.ceph.com, ubuntu@smithi085.front.sepia.ceph.com 2020-08-31T22:09:16.445 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f09aaefdea0>, signals=[15]) 2020-08-31T22:09:16.702 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of centos 8.1 2020-08-31T22:09:16.723 INFO:teuthology.provision.fog.smithi172:Scheduling deploy of centos 8.1 2020-08-31T22:09:16.726 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of centos 8.1 2020-08-31T22:09:17.104 INFO:teuthology.orchestra.console:Power off smithi085 2020-08-31T22:09:17.105 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T22:09:17.133 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:17.156 INFO:teuthology.orchestra.console:Power off smithi172 2020-08-31T22:09:17.158 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T22:09:17.178 INFO:teuthology.orchestra.console:Power off smithi146 2020-08-31T22:09:17.179 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T22:09:17.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:17.203 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:17.303 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2020-08-31T22:09:17.316 INFO:teuthology.orchestra.console:Power off for smithi172 completed 2020-08-31T22:09:17.328 INFO:teuthology.orchestra.console:Power off for smithi146 completed 2020-08-31T22:09:17.405 INFO:teuthology.orchestra.console:Power on smithi085 2020-08-31T22:09:17.406 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T22:09:17.419 INFO:teuthology.orchestra.console:Power on smithi172 2020-08-31T22:09:17.420 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T22:09:17.431 INFO:teuthology.orchestra.console:Power on smithi146 2020-08-31T22:09:17.431 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T22:09:17.442 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:17.454 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:17.465 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:21.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:21.494 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:21.506 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:21.708 INFO:teuthology.orchestra.console:Power on for smithi172 completed 2020-08-31T22:09:21.810 INFO:teuthology.provision.fog.smithi172:Waiting for deploy to finish 2020-08-31T22:09:25.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi085.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:25.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T22:09:25.813 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2020-08-31T22:09:25.833 INFO:teuthology.orchestra.console:Power on for smithi146 completed 2020-08-31T22:09:25.914 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2020-08-31T22:09:25.935 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish 2020-08-31T22:11:57.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:11:57.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:06.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:06.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:08.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:15.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:15.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:17.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:24.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:24.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:26.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:33.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:33.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:35.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:42.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:42.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:44.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:51.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:51.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:12:53.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:13:00.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:13:02.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:13:11.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:13:57.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:13:58.178 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-08-31T22:13:58.181 INFO:teuthology.orchestra.run.smithi085:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-08-31T22:13:58.515 INFO:teuthology.provision.fog.smithi085:Node is ready 2020-08-31T22:13:58.515 INFO:teuthology.orchestra.run.smithi085:> hostname 2020-08-31T22:13:58.530 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com 2020-08-31T22:13:58.531 INFO:teuthology.orchestra.run.smithi085:> grep smithi085.front.sepia.ceph.com /etc/hosts 2020-08-31T22:13:58.585 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085 2020-08-31T22:13:58.586 INFO:teuthology.orchestra.run.smithi085:> sudo hostname smithi085 2020-08-31T22:13:58.719 INFO:teuthology.orchestra.run.smithi085:> sudo sed -i -e 's/smithi085.front.sepia.ceph.com/smithi085/g' /etc/hosts 2020-08-31T22:13:58.747 INFO:teuthology.orchestra.run.smithi085:> sudo sed -i -e 's/172.21.15.85/172.21.15.85/g' /etc/hosts 2020-08-31T22:13:58.815 INFO:teuthology.orchestra.run.smithi085:> sudo sed -i -e 's/smithi085.front.sepia.ceph.com/smithi085/g' /etc/hostname 2020-08-31T22:13:58.884 INFO:teuthology.orchestra.run.smithi085:> sudo hostname smithi085 2020-08-31T22:13:58.950 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2020-08-31T22:13:58.951 DEBUG:teuthology.parallel:result is None 2020-08-31T22:14:06.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:07.190 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-08-31T22:14:07.191 INFO:teuthology.orchestra.run.smithi146:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-08-31T22:14:07.499 INFO:teuthology.provision.fog.smithi146:Node is ready 2020-08-31T22:14:07.499 INFO:teuthology.orchestra.run.smithi146:> hostname 2020-08-31T22:14:07.523 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com 2020-08-31T22:14:07.524 INFO:teuthology.orchestra.run.smithi146:> grep smithi146.front.sepia.ceph.com /etc/hosts 2020-08-31T22:14:07.578 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146 2020-08-31T22:14:07.578 INFO:teuthology.orchestra.run.smithi146:> sudo hostname smithi146 2020-08-31T22:14:07.698 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hosts 2020-08-31T22:14:07.727 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/172.21.15.146/172.21.15.146/g' /etc/hosts 2020-08-31T22:14:07.794 INFO:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hostname 2020-08-31T22:14:07.861 INFO:teuthology.orchestra.run.smithi146:> sudo hostname smithi146 2020-08-31T22:14:07.927 INFO:teuthology.provision.fog.smithi146:Deploy complete! 2020-08-31T22:14:07.927 DEBUG:teuthology.parallel:result is None 2020-08-31T22:14:17.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:18.067 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-08-31T22:14:18.067 INFO:teuthology.orchestra.run.smithi172:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-08-31T22:14:18.395 INFO:teuthology.provision.fog.smithi172:Node is ready 2020-08-31T22:14:18.395 INFO:teuthology.orchestra.run.smithi172:> hostname 2020-08-31T22:14:18.410 INFO:teuthology.orchestra.run.smithi172.stdout:smithi172.front.sepia.ceph.com 2020-08-31T22:14:18.411 INFO:teuthology.orchestra.run.smithi172:> grep smithi172.front.sepia.ceph.com /etc/hosts 2020-08-31T22:14:18.466 INFO:teuthology.orchestra.run.smithi172.stdout:172.21.15.172 smithi172.front.sepia.ceph.com smithi172 2020-08-31T22:14:18.466 INFO:teuthology.orchestra.run.smithi172:> sudo hostname smithi172 2020-08-31T22:14:18.576 INFO:teuthology.orchestra.run.smithi172:> sudo sed -i -e 's/smithi172.front.sepia.ceph.com/smithi172/g' /etc/hosts 2020-08-31T22:14:18.607 INFO:teuthology.orchestra.run.smithi172:> sudo sed -i -e 's/172.21.15.172/172.21.15.172/g' /etc/hosts 2020-08-31T22:14:18.676 INFO:teuthology.orchestra.run.smithi172:> sudo sed -i -e 's/smithi172.front.sepia.ceph.com/smithi172/g' /etc/hostname 2020-08-31T22:14:18.745 INFO:teuthology.orchestra.run.smithi172:> sudo hostname smithi172 2020-08-31T22:14:18.817 INFO:teuthology.provision.fog.smithi172:Deploy complete! 2020-08-31T22:14:18.827 DEBUG:teuthology.parallel:result is None 2020-08-31T22:14:19.235 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com 2020-08-31T22:14:19.236 INFO:teuthology.lock.ops:New key found. Updating... 2020-08-31T22:14:19.277 INFO:teuthology.lock.ops:Checking smithi172.front.sepia.ceph.com 2020-08-31T22:14:19.277 INFO:teuthology.lock.ops:New key found. Updating... 2020-08-31T22:14:19.308 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2020-08-31T22:14:19.308 INFO:teuthology.lock.ops:New key found. Updating... 2020-08-31T22:14:19.343 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server 2020-08-31T22:14:19.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-08-31T22:14:19.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:19.503 INFO:teuthology.orchestra.run.smithi146:> true 2020-08-31T22:14:19.592 INFO:teuthology.orchestra.run.smithi146:> uname -m 2020-08-31T22:14:19.648 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2020-08-31T22:14:19.649 INFO:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2020-08-31T22:14:19.705 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Linux" 2020-08-31T22:14:19.705 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8 (Core)" 2020-08-31T22:14:19.706 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2020-08-31T22:14:19.706 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2020-08-31T22:14:19.706 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8" 2020-08-31T22:14:19.706 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2020-08-31T22:14:19.706 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-08-31T22:14:19.707 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2020-08-31T22:14:19.707 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-08-31T22:14:19.707 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.centos.org/" 2020-08-31T22:14:19.707 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-08-31T22:14:19.707 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-08-31T22:14:19.708 INFO:teuthology.orchestra.run.smithi146.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-08-31T22:14:19.708 INFO:teuthology.orchestra.run.smithi146.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-08-31T22:14:19.708 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-08-31T22:14:19.708 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-08-31T22:14:19.708 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-08-31T22:14:19.710 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2020-08-31T22:14:19.727 INFO:teuthology.lock.ops:Updating [smithi172.front.sepia.ceph.com]: set os type and version on server 2020-08-31T22:14:19.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-08-31T22:14:19.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:19.845 INFO:teuthology.orchestra.run.smithi172:> true 2020-08-31T22:14:20.005 INFO:teuthology.orchestra.run.smithi172:> uname -m 2020-08-31T22:14:20.059 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2020-08-31T22:14:20.059 INFO:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2020-08-31T22:14:20.114 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Linux" 2020-08-31T22:14:20.115 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8 (Core)" 2020-08-31T22:14:20.115 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2020-08-31T22:14:20.115 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2020-08-31T22:14:20.115 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8" 2020-08-31T22:14:20.115 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2020-08-31T22:14:20.116 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-08-31T22:14:20.116 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2020-08-31T22:14:20.116 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-08-31T22:14:20.116 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.centos.org/" 2020-08-31T22:14:20.116 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-08-31T22:14:20.117 INFO:teuthology.orchestra.run.smithi172.stdout: 2020-08-31T22:14:20.117 INFO:teuthology.orchestra.run.smithi172.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-08-31T22:14:20.117 INFO:teuthology.orchestra.run.smithi172.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-08-31T22:14:20.117 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-08-31T22:14:20.117 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-08-31T22:14:20.118 INFO:teuthology.orchestra.run.smithi172.stdout: 2020-08-31T22:14:20.118 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2020-08-31T22:14:20.135 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2020-08-31T22:14:20.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-08-31T22:14:20.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:20.250 INFO:teuthology.orchestra.run.smithi085:> true 2020-08-31T22:14:20.413 INFO:teuthology.orchestra.run.smithi085:> uname -m 2020-08-31T22:14:20.467 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2020-08-31T22:14:20.467 INFO:teuthology.orchestra.run.smithi085:> cat /etc/os-release 2020-08-31T22:14:20.521 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="CentOS Linux" 2020-08-31T22:14:20.522 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8 (Core)" 2020-08-31T22:14:20.522 INFO:teuthology.orchestra.run.smithi085.stdout:ID="centos" 2020-08-31T22:14:20.522 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="rhel fedora" 2020-08-31T22:14:20.522 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8" 2020-08-31T22:14:20.523 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8" 2020-08-31T22:14:20.523 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-08-31T22:14:20.523 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31" 2020-08-31T22:14:20.523 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-08-31T22:14:20.523 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.centos.org/" 2020-08-31T22:14:20.524 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-08-31T22:14:20.524 INFO:teuthology.orchestra.run.smithi085.stdout: 2020-08-31T22:14:20.524 INFO:teuthology.orchestra.run.smithi085.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-08-31T22:14:20.524 INFO:teuthology.orchestra.run.smithi085.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-08-31T22:14:20.524 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-08-31T22:14:20.525 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-08-31T22:14:20.525 INFO:teuthology.orchestra.run.smithi085.stdout: 2020-08-31T22:14:20.526 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2020-08-31T22:14:20.547 INFO:teuthology.task.internal.lock_machines:3 smithi machines locked this try, 3/3 locked so far 2020-08-31T22:14:20.626 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8GvcHezjlMuMnx1BNLVXKIOjS9QYJZt2A7yruANfoV4FO9rAtyseHPOJW2BnlTB95R1JW9YQfSTE5IJexdmRFFrI9ME2HB1bzqPGBKKqijBOk95Ll+EEfw3umgAlfsxkCYR1bQKjfbYZ01KEqRe+jdNqCBWgVF59hQfNsxDIN4JbecZg1WJecLaUGSnSNAz6cMDgkw82M8MUK05utCVYzA7zdAQNHO87L6FPh0VA7ZwyoNNN7Xjrwit3FXuPjJcUZvZji3OIUikVezTkmBUd7hPlHSD30/z+KxQKJzUXZuuPBySYXzU/VNMo3tdhGeVGMivpnW7xGIm4lBjyAlt0byTiP809vnHK77gL06n27zbgWj1MnzBJ4qoiF9GFSqrlsMzkGhgmUL7IX/OgWY38E8vlx8vPgoOnLZJEpPDeZFzkt2A2SJagsrPxjHaEy7414MyJoLgnBqedcud/fWuoWLDzOPbrO2JaUv/nY98/7A8DaiADEWzhN5eKJF4Yx7Rs= smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EdUYp1+lXlqxIzoc7mP1cYrZALoQX/udQUJBcU0hqtm83gnJ0unJ8OlX4ZNXvXzV7+ELqOsRX1d6G78uI5NqXvM9vBl660l0ok03JpqIMaeNUYL26NFc9BXNYfJEkFdrgKm2ZJVAlyXbIfAG9EYDTPt+Y9uTY/LpO7PTUhe3mXDmCpKKbPCg6Fc51udcRih75XKfBRXyok9sOJ/9eK/83T1z9IlAnPKllWFk2DpwBqTg8mTFVBEKlobDFDRw0jbOad0qTNy4RIe9rCEmsxaxQOjYbYdHnh6Qx32gxwIH8PdKM2MQlbWiXsPSKvyHeE0QJDrW4Ze626trfuKOO3FQJsye5p8ZOH7ktarJPn1v/Kx+MO1xocyRiIIyCSK7LLrp+AbX79H6hq7qCGzLhirv3yXOYf/uWEXLaT3771uCVxej2zJnHjnHzLeVvll4BQi8YD9rolqx3H/ZebGceWBbtGEDGWaY9mDMd4IdZmQkMohsABKXz1h7t+1/D5F44z8= smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQVKiD/Tm8SmNXd/afSVRgEgnofH+gXt7fIzl5noQ6atdbmAFF4iW7MFwtjyDyT/borS2QyivXQlCQQdEM2vQtEJZTGiHN5S9wTI4alae02aLMH7ZGdu/q0xVdZ4lX3uX+qBH3uEh3uwkfNs0mSjvi+pGn8hh2j7z1/6O4a6TQqGLayyl+VNGNRpLU3UQHc/4KOO1he+hYwrZG6t8c8IHvJIv4pUpAHY3Ltwp3CLUfH3vZEOy9SUFwDHaevLTk7OAK72QOZr9hQo+S5bQPjIoAibHHGEJgHunzNQHoubZOTvRIUJpBhZKVGYzcsS4T2gmkITpVHG2lO5MmmhF5ThnPzsQeeRVYDKuBjL1sj/IvDl+cfsjtTF3+yAIHBs9P/sY4eh3ftGipBkL5jkVog7eFJN+yBKWF8XHsYrUsuoIxKtYgV3yRMa/CQid0NpmVKaHOkV3m14Rd/O0684p4ZLLO15cRWdaT/dkf2/9wVmfI/h8rxmvMNaxHN6PkKJsm0h0= 2020-08-31T22:14:20.626 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-31T22:14:20.681 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-08-31T22:14:21.130 INFO:teuthology.task.internal:Saving configuration 2020-08-31T22:14:21.215 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-08-31T22:14:21.248 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-08-31T22:14:21.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-08-31 22:09:16.338848', 'description': '/home/teuthworker/archive/pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396249', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBO9gvQ5g5GWyzpzjW34Y5VGnlLYlribGBQA2SD5I6Ae0vQZ17CiWWQbhkF+9AT477Zfy2RJ1h4+2InPt7/+AOE=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi146.front.sepia.ceph.com'} 2020-08-31T22:14:21.283 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-08-31 22:09:16.338868', 'description': '/home/teuthworker/archive/pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396249', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCvRmKB7Isqb7+L641QjGr0qErukH8azHFjyTcKLAjHb6EtCHL69k6PGAUcJIOEWimk3gCvFDvVFYcgdfhAoTEg=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi172.front.sepia.ceph.com'} 2020-08-31T22:14:21.299 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-08-31 22:09:16.338896', 'description': '/home/teuthworker/archive/pdonnell-2020-08-31_21:26:19-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396249', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFOn+McIm98at/uYjiTLUEoh+0iOfgzX6RfZSf9d/neM6NzoBeNgJhPOvopbvaQiMMnRxSZclpYhftH/W4y1PLA=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi085.front.sepia.ceph.com'} 2020-08-31T22:14:21.300 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-08-31T22:14:21.315 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2020-08-31T22:14:21.316 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2020-08-31T22:14:21.316 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['client.0'] 2020-08-31T22:14:21.316 INFO:teuthology.run_tasks:Running task console_log... 2020-08-31T22:14:21.444 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f09adb40d90>, signals=[15]) 2020-08-31T22:14:21.446 INFO:teuthology.run_tasks:Running task internal.connect... 2020-08-31T22:14:21.457 INFO:teuthology.task.internal:Opening connections... 2020-08-31T22:14:21.458 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2020-08-31T22:14:21.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:21.536 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2020-08-31T22:14:21.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:21.612 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi085.front.sepia.ceph.com 2020-08-31T22:14:21.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi085.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T22:14:21.686 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-08-31T22:14:21.696 INFO:teuthology.orchestra.run.smithi146:> uname -m 2020-08-31T22:14:21.716 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2020-08-31T22:14:21.716 INFO:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2020-08-31T22:14:21.771 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Linux" 2020-08-31T22:14:21.772 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8 (Core)" 2020-08-31T22:14:21.772 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2020-08-31T22:14:21.772 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2020-08-31T22:14:21.772 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8" 2020-08-31T22:14:21.773 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2020-08-31T22:14:21.773 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-08-31T22:14:21.773 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2020-08-31T22:14:21.773 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-08-31T22:14:21.773 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.centos.org/" 2020-08-31T22:14:21.774 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-08-31T22:14:21.774 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-08-31T22:14:21.774 INFO:teuthology.orchestra.run.smithi146.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-08-31T22:14:21.774 INFO:teuthology.orchestra.run.smithi146.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-08-31T22:14:21.775 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-08-31T22:14:21.775 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-08-31T22:14:21.775 INFO:teuthology.orchestra.run.smithi146.stdout: 2020-08-31T22:14:21.775 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2020-08-31T22:14:21.793 INFO:teuthology.orchestra.run.smithi172:> uname -m 2020-08-31T22:14:21.809 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2020-08-31T22:14:21.809 INFO:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2020-08-31T22:14:21.863 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Linux" 2020-08-31T22:14:21.864 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8 (Core)" 2020-08-31T22:14:21.864 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2020-08-31T22:14:21.864 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2020-08-31T22:14:21.864 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8" 2020-08-31T22:14:21.864 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2020-08-31T22:14:21.865 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-08-31T22:14:21.865 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2020-08-31T22:14:21.865 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-08-31T22:14:21.865 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.centos.org/" 2020-08-31T22:14:21.865 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-08-31T22:14:21.866 INFO:teuthology.orchestra.run.smithi172.stdout: 2020-08-31T22:14:21.866 INFO:teuthology.orchestra.run.smithi172.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-08-31T22:14:21.866 INFO:teuthology.orchestra.run.smithi172.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-08-31T22:14:21.866 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-08-31T22:14:21.866 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-08-31T22:14:21.867 INFO:teuthology.orchestra.run.smithi172.stdout: 2020-08-31T22:14:21.867 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2020-08-31T22:14:21.884 INFO:teuthology.orchestra.run.smithi085:> uname -m 2020-08-31T22:14:21.899 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2020-08-31T22:14:21.900 INFO:teuthology.orchestra.run.smithi085:> cat /etc/os-release 2020-08-31T22:14:21.954 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="CentOS Linux" 2020-08-31T22:14:21.955 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8 (Core)" 2020-08-31T22:14:21.955 INFO:teuthology.orchestra.run.smithi085.stdout:ID="centos" 2020-08-31T22:14:21.955 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="rhel fedora" 2020-08-31T22:14:21.955 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8" 2020-08-31T22:14:21.956 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8" 2020-08-31T22:14:21.956 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-08-31T22:14:21.956 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31" 2020-08-31T22:14:21.956 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-08-31T22:14:21.957 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://www.centos.org/" 2020-08-31T22:14:21.957 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-08-31T22:14:21.957 INFO:teuthology.orchestra.run.smithi085.stdout: 2020-08-31T22:14:21.957 INFO:teuthology.orchestra.run.smithi085.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-08-31T22:14:21.957 INFO:teuthology.orchestra.run.smithi085.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-08-31T22:14:21.958 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-08-31T22:14:21.958 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-08-31T22:14:21.958 INFO:teuthology.orchestra.run.smithi085.stdout: 2020-08-31T22:14:21.958 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2020-08-31T22:14:21.973 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-08-31T22:14:21.988 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-08-31T22:14:21.997 INFO:teuthology.task.internal:Checking for old test directory... 2020-08-31T22:14:21.998 INFO:teuthology.orchestra.run.smithi085:> test '!' -e /home/ubuntu/cephtest 2020-08-31T22:14:22.001 INFO:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2020-08-31T22:14:22.004 INFO:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2020-08-31T22:14:22.018 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-08-31T22:14:22.037 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-08-31T22:14:22.038 INFO:teuthology.orchestra.run.smithi085:> test -z $(ls -A /var/lib/ceph) 2020-08-31T22:14:22.055 INFO:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2020-08-31T22:14:22.058 INFO:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2020-08-31T22:14:22.084 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-08-31T22:14:22.146 INFO:teuthology.run_tasks:Running task kernel... 2020-08-31T22:14:22.168 DEBUG:teuthology.task.kernel:normalized config {'client.0': {'branch': 'testing'}} 2020-08-31T22:14:22.169 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2020-08-31T22:14:22.169 INFO:teuthology.orchestra.run.smithi085:> sudo lsb_release -is 2020-08-31T22:14:22.229 INFO:teuthology.orchestra.run.smithi085.stdout:CentOS 2020-08-31T22:14:22.230 DEBUG:teuthology.misc:System to be installed: CentOS 2020-08-31T22:14:22.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2020-08-31T22:14:22.514 DEBUG:teuthology.task.kernel:sha1 for client.0 is 7b4196f8b36937040ee59cf376f44604f885ef7e 2020-08-31T22:14:22.514 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "7b4196f8b36937040ee59cf376f44604f885ef7e"... 2020-08-31T22:14:22.515 INFO:teuthology.orchestra.run.smithi085:> uname -r 2020-08-31T22:14:22.531 INFO:teuthology.orchestra.run.smithi085.stdout:4.18.0-193.14.2.el8_2.x86_64 2020-08-31T22:14:22.532 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-193.14.2.el8_2.x86_64 vs 7b4196f8b36937040ee59cf376f44604f885ef7e 2020-08-31T22:14:22.533 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2020-08-31T22:14:22.534 INFO:teuthology.orchestra.run.smithi085:> sudo yum upgrade -y linux-firmware 2020-08-31T22:14:25.140 INFO:teuthology.orchestra.run.smithi085.stdout:CentOS-8 - AppStream 41 MB/s | 7.0 MB 00:00 2020-08-31T22:14:26.344 INFO:teuthology.orchestra.run.smithi085.stdout:CentOS-8 - Base 25 MB/s | 2.2 MB 00:00 2020-08-31T22:14:26.785 INFO:teuthology.orchestra.run.smithi085.stdout:CentOS-8 - Extras 87 kB/s | 5.5 kB 00:00 2020-08-31T22:14:27.190 INFO:teuthology.orchestra.run.smithi085.stdout:Extra Packages for Enterprise Linux 22 MB/s | 7.9 MB 00:00 2020-08-31T22:14:28.817 INFO:teuthology.orchestra.run.smithi085.stdout:lab-extras 3.6 MB/s | 20 kB 00:00 2020-08-31T22:14:29.964 INFO:teuthology.orchestra.run.smithi085.stdout:Dependencies resolved. 2020-08-31T22:14:29.965 INFO:teuthology.orchestra.run.smithi085.stdout:Nothing to do. 2020-08-31T22:14:29.965 INFO:teuthology.orchestra.run.smithi085.stdout:Complete! 2020-08-31T22:14:30.090 INFO:teuthology.task.kernel:Downloading kernel 7b4196f8b36937040ee59cf376f44604f885ef7e on client.0... 2020-08-31T22:14:30.091 INFO:teuthology.orchestra.run.smithi085:> sudo lsb_release -is 2020-08-31T22:14:30.127 INFO:teuthology.orchestra.run.smithi085.stdout:CentOS 2020-08-31T22:14:30.128 DEBUG:teuthology.misc:System to be installed: CentOS 2020-08-31T22:14:30.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7b4196f8b36937040ee59cf376f44604f885ef7e 2020-08-31T22:14:30.292 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/7b4196f8b36937040ee59cf376f44604f885ef7e/centos/8/flavors/default/x86_64/ 2020-08-31T22:14:30.293 INFO:teuthology.orchestra.run.smithi085:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.9.0_rc2_ceph_g7b4196f8b369-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/7b4196f8b36937040ee59cf376f44604f885ef7e/centos/8/flavors/default/x86_64/ --input-file=- 2020-08-31T22:14:30.296 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi085.front.sepia.ceph.com to complete... 2020-08-31T22:17:26.296 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-08-31T22:17:26.301 DEBUG:teuthology.task.console_log:Killing console logger for smithi146 2020-08-31T22:17:26.303 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2020-08-31T22:17:26.304 DEBUG:teuthology.task.console_log:Killing console logger for smithi085 2020-08-31T22:17:26.304 DEBUG:teuthology.task.console_log:Killing console logger for smithi146 2020-08-31T22:17:26.305 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2020-08-31T22:17:26.305 DEBUG:teuthology.task.console_log:Killing console logger for smithi085 2020-08-31T22:17:26.314 DEBUG:teuthology.exit:Finished running handlers