2020-02-14T15:35:15.355 INFO:root:teuthology version: 1.0.0-ee19bca 2020-02-14T15:35:15.355 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.D0BQ6n.tmp -- --verbose --lock --description rados/dashboard/{clusters/{2-node-mgr.yaml} debug/mgr.yaml objectstore/filestore-xfs.yaml supported-random-distro$/{rhel_8.yaml} tasks/dashboard.yaml} --name kchai-2020-02-14_15:08:49-rados-wip-kefu-testing-2020-02-14-2045-distro-basic-smithi --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2020-02-14_15:08:49-rados-wip-kefu-testing-2020-02-14-2045-distro-basic-smithi/4763804 2020-02-14T15:35:15.391 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-14T15:35:15.460 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-02-14_15:08:49-rados-wip-kefu-testing-2020-02-14-2045-distro-basic-smithi/4763804 branch: wip-kefu-testing-2020-02-14-2045 description: rados/dashboard/{clusters/{2-node-mgr.yaml} debug/mgr.yaml objectstore/filestore-xfs.yaml supported-random-distro$/{rhel_8.yaml} tasks/dashboard.yaml} email: null first_in_suite: false job_id: '4763804' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: kchai-2020-02-14_15:08:49-rados-wip-kefu-testing-2020-02-14-2045-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 2 size: 30 os_type: rhel os_version: '8.1' overrides: admin_socket: branch: wip-kefu-testing-2020-02-14-2045 ceph: conf: client: debug client: 20 debug mgrc: 20 debug ms: 1 mds: debug mgrc: 20 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug mgrc: 20 debug ms: 1 debug osd: 25 osd objectstore: filestore osd sloppy crc: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a075f279f1948ef2c8122c5ac06d522688c3eb90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: a075f279f1948ef2c8122c5ac06d522688c3eb90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2020-02-14-2045 sha1: a075f279f1948ef2c8122c5ac06d522688c3eb90 owner: scheduled_kchai@teuthology priority: 30 repo: git://git.ceph.com/ceph-ci.git roles: - - mgr.x - mon.a - mon.c - mds.a - mds.c - osd.0 - client.0 - - mgr.y - mgr.z - mon.b - mds.b - osd.1 - osd.2 - osd.3 - client.1 sha1: a075f279f1948ef2c8122c5ac06d522688c3eb90 suite: rados suite_branch: wip-kefu-testing-2020-02-14-2045 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-02-14-2045/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: a075f279f1948ef2c8122c5ac06d522688c3eb90 tasks: - install: null - ceph: log-whitelist: - overall HEALTH_ - \(MGR_DOWN\) - \(PG_ - replacing it with standby - No standby daemons available - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_DOWN\) - \(OSD_HOST_DOWN\) - \(POOL_APP_NOT_ENABLED\) - \(OSDMAP_FLAGS\) - pauserd,pausewr flag\(s\) set - Monitor daemon marked osd\.[[:digit:]]+ down, but it is still running - evicting unresponsive client .+ wait-for-scrub: false - rgw: - client.0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.mgr.test_dashboard - tasks.mgr.dashboard.test_auth - tasks.mgr.dashboard.test_cephfs - tasks.mgr.dashboard.test_cluster_configuration - tasks.mgr.dashboard.test_erasure_code_profile - tasks.mgr.dashboard.test_ganesha - tasks.mgr.dashboard.test_health - tasks.mgr.dashboard.test_host - tasks.mgr.dashboard.test_logs - tasks.mgr.dashboard.test_mgr_module - tasks.mgr.dashboard.test_monitor - tasks.mgr.dashboard.test_orchestrator - tasks.mgr.dashboard.test_osd - tasks.mgr.dashboard.test_perf_counters - tasks.mgr.dashboard.test_pool - tasks.mgr.dashboard.test_rbd - tasks.mgr.dashboard.test_rbd_mirroring - tasks.mgr.dashboard.test_requests - tasks.mgr.dashboard.test_rgw - tasks.mgr.dashboard.test_role - tasks.mgr.dashboard.test_settings - tasks.mgr.dashboard.test_summary - tasks.mgr.dashboard.test_user teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.17164 2020-02-14T15:35:15.461 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-02-14-2045/qa; will attempt to use it 2020-02-14T15:35:15.461 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-02-14-2045/qa/tasks 2020-02-14T15:35:15.461 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-02-14T15:35:15.463 INFO:teuthology.task.internal:Checking packages... 2020-02-14T15:35:15.476 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'a075f279f1948ef2c8122c5ac06d522688c3eb90' 2020-02-14T15:35:15.476 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-02-14T15:35:15.476 INFO:teuthology.packaging:ref: None 2020-02-14T15:35:15.476 INFO:teuthology.packaging:tag: None 2020-02-14T15:35:15.476 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-02-14-2045 2020-02-14T15:35:15.476 INFO:teuthology.packaging:sha1: a075f279f1948ef2c8122c5ac06d522688c3eb90 2020-02-14T15:35:15.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2020-02-14-2045 2020-02-14T15:35:15.643 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-735.ga075f27 2020-02-14T15:35:15.643 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-02-14T15:35:15.663 INFO:teuthology.task.internal:no buildpackages task found 2020-02-14T15:35:15.663 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-02-14T15:35:15.675 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-02-14T15:35:15.676 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-14T15:35:15.741 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:35:25.766 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:35:35.786 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:35:45.803 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:35:55.819 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:36:05.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:36:15.853 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:36:25.872 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:36:35.890 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:36:45.907 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:36:55.924 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:37:05.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:37:15.964 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:37:25.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:37:36.003 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:37:46.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:37:56.045 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:38:06.062 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:38:16.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:38:26.102 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:38:36.123 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:38:46.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:38:56.162 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:39:06.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:39:16.200 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:39:26.220 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:39:36.254 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:39:46.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-14T15:39:56.291 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:40:06.311 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:40:16.328 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:40:26.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:40:36.357 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-02-14T15:40:46.378 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2020-02-14_15:08:49-rados-wip-kefu-testing-2020-02-14-2045-distro-basic-smithi/4763804', 'machine_type': 'smithi'} 2020-02-14T15:40:46.409 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 2 smithi nodes. 2020-02-14T15:40:46.409 ERROR:teuthology.lock.ops:{"message": "only 0 nodes available"} 2020-02-14T15:40:46.409 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/2 locked so far 2020-02-14T15:40:46.409 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 2 more 2020-02-14T15:40:46.409 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting... 2020-02-14T15:40:56.428 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:41:06.445 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:41:16.465 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:41:26.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:41:36.500 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:41:46.519 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:41:56.535 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:42:06.553 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:42:16.572 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:42:26.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:42:36.610 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 0)... 2020-02-14T15:42:46.629 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:42:56.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-14T15:43:06.674 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2020-02-14_15:08:49-rados-wip-kefu-testing-2020-02-14-2045-distro-basic-smithi/4763804', 'machine_type': 'smithi'} 2020-02-14T15:43:06.697 DEBUG:teuthology.lock.ops:locked ubuntu@smithi138.front.sepia.ceph.com, ubuntu@smithi142.front.sepia.ceph.com 2020-02-14T15:43:06.767 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-02-14T15:43:06.974 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of rhel 8.1 2020-02-14T15:43:07.008 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of rhel 8.1 2020-02-14T15:43:07.308 INFO:teuthology.orchestra.console:Power off smithi138 2020-02-14T15:43:07.309 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-14T15:43:07.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:07.354 INFO:teuthology.orchestra.console:Power off smithi142 2020-02-14T15:43:07.354 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-14T15:43:07.381 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:11.366 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:11.413 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:15.478 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:15.522 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:15.691 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2020-02-14T15:43:15.732 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2020-02-14T15:43:15.793 INFO:teuthology.orchestra.console:Power on smithi138 2020-02-14T15:43:15.793 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-14T15:43:15.814 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:15.833 INFO:teuthology.orchestra.console:Power on smithi142 2020-02-14T15:43:15.833 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-14T15:43:15.858 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:19.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:19.881 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:20.104 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2020-02-14T15:43:20.212 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2020-02-14T15:43:23.960 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-14T15:43:24.246 INFO:teuthology.orchestra.console:Power on for smithi138 completed 2020-02-14T15:43:24.348 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish 2020-02-14T15:43:59.723 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2020-02-14T15:43:59.723 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2020-02-14T15:43:59.724 DEBUG:teuthology.task.console_log:Killing console logger for smithi138 2020-02-14T15:43:59.724 DEBUG:teuthology.exit:Finished running handlers