2020-11-16T22:00:01.555 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-16T22:00:01.556 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631938 --description multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_multimds_misc} --owner scheduled_pdonnell@teuthology --block --verbose --name pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi --lock -- /tmp/teuthology-worker.cl_bsmph.tmp 2020-11-16T22:00:01.594 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T22:00:01.657 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631938 branch: wip-pdonnell-testing-20201116.174825 description: multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_multimds_misc} email: pdonnell@redhat.com first_in_suite: false job_id: '5631938' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '8.1' overrides: admin_socket: branch: wip-pdonnell-testing-20201116.174825 ceph: cephfs: max_mds: 3 session_timeout: 300 cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 fuse default permissions: false rados mon op timeout: 15m rados osd op timeout: 15m global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Scrub error on inode log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 6959059e6aae2fb5241382a4eeeca62785867604 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 6959059e6aae2fb5241382a4eeeca62785867604 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20201116.174825 sha1: 6959059e6aae2fb5241382a4eeeca62785867604 owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mgr.x - mds.b - mds.c - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - client.1 sha1: 6959059e6aae2fb5241382a4eeeca62785867604 sleep_before_teardown: 0 suite: multimds suite_branch: wip-pdonnell-testing-20201116.174825 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201116.174825/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 6959059e6aae2fb5241382a4eeeca62785867604 tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - check-counter: counters: mds: - mds.exported - mds.imported - cephfs_test_runner: modules: - tasks.cephfs.test_multimds_misc teuthology_branch: master timestamp: 2020-11-16_20:59:43 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.11585 2020-11-16T22:00:01.657 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201116.174825/qa; will attempt to use it 2020-11-16T22:00:01.658 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201116.174825/qa/tasks 2020-11-16T22:00:01.658 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-16T22:00:01.660 INFO:teuthology.task.internal:Checking packages... 2020-11-16T22:00:01.680 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '6959059e6aae2fb5241382a4eeeca62785867604' 2020-11-16T22:00:01.681 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-16T22:00:01.681 INFO:teuthology.packaging:ref: None 2020-11-16T22:00:01.681 INFO:teuthology.packaging:tag: None 2020-11-16T22:00:01.682 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20201116.174825 2020-11-16T22:00:01.682 INFO:teuthology.packaging:sha1: 6959059e6aae2fb5241382a4eeeca62785867604 2020-11-16T22:00:01.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20201116.174825 2020-11-16T22:00:01.845 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7226.g6959059e 2020-11-16T22:00:01.846 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-16T22:00:01.855 INFO:teuthology.task.internal:no buildpackages task found 2020-11-16T22:00:01.855 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-16T22:00:01.869 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-16T22:00:01.869 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T22:00:01.920 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:00:11.939 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:00:21.954 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:00:31.970 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:00:41.988 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:00:51.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:02.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:12.024 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:22.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:32.057 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:42.074 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:52.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:02.106 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:12.119 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:22.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:32.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:42.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:52.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:02.201 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:12.219 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:22.234 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:32.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:42.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:52.491 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:02.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:12.519 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:22.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:32.550 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:42.569 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:52.586 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:02.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:12.615 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:22.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:32.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:42.668 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:52.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:02.702 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:12.715 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:22.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:32.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:42.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:52.785 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:02.804 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:12.818 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:22.834 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:32.853 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:42.870 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:52.888 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:02.903 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:12.918 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:22.931 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:32.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:42.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:52.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:02.997 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:13.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:23.032 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:33.050 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:43.069 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:53.088 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:10:03.106 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:10:13.124 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:10:23.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:10:33.158 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:10:43.176 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:10:53.194 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:03.208 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:13.226 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:23.245 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:33.256 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:43.275 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:53.292 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:03.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:13.320 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:23.330 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:33.346 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:43.362 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:53.381 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:03.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:13.410 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:23.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:33.445 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:43.462 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:53.471 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:03.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:13.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:23.520 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:33.538 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:43.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:53.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:03.595 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:13.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:23.626 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:33.639 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:43.655 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:53.669 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:03.692 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:13.710 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:23.727 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:33.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:43.763 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:53.781 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:03.793 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:13.811 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:23.829 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:33.847 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:43.860 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:53.878 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:03.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:13.913 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:23.924 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:33.938 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:43.953 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:53.971 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:19:03.989 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:26.363 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:36.382 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:46.401 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:56.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:06.423 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:16.441 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:26.458 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:36.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:46.487 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:56.506 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:06.523 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:16.541 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:26.549 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:36.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:46.585 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:56.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:06.621 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:16.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:26.649 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:36.667 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:46.684 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:56.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:06.719 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:16.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:26.753 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:36.763 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:46.782 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:56.799 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:06.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:16.828 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:26.846 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:36.864 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:46.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:56.899 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:06.909 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:16.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:26.944 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:36.961 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:46.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:56.995 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:07.013 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:17.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:27.047 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:37.064 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:47.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:57.096 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:07.112 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:17.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:27.149 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:37.172 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:47.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:57.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:07.224 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:17.242 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:27.257 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:37.276 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631938'} 2020-11-16T22:34:37.300 DEBUG:teuthology.lock.ops:locked ubuntu@smithi035.front.sepia.ceph.com, ubuntu@smithi025.front.sepia.ceph.com, ubuntu@smithi055.front.sepia.ceph.com 2020-11-16T22:34:37.377 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc233e80158>, signals=[15]) 2020-11-16T22:34:37.578 INFO:teuthology.provision.fog.smithi035:Scheduling deploy of rhel 8.1 2020-11-16T22:34:37.586 INFO:teuthology.provision.fog.smithi055:Scheduling deploy of rhel 8.1 2020-11-16T22:34:37.588 INFO:teuthology.provision.fog.smithi025:Scheduling deploy of rhel 8.1 2020-11-16T22:34:37.905 INFO:teuthology.orchestra.console:Power off smithi035 2020-11-16T22:34:37.906 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-16T22:34:37.922 INFO:teuthology.orchestra.console:Power off smithi025 2020-11-16T22:34:37.922 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-16T22:34:37.937 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:37.947 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:37.962 INFO:teuthology.orchestra.console:Power off smithi055 2020-11-16T22:34:37.963 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-16T22:34:37.984 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:41.974 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:41.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:42.006 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:46.088 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:46.098 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:46.116 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:50.202 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:50.212 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:50.227 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:50.415 INFO:teuthology.orchestra.console:Power off for smithi025 completed 2020-11-16T22:34:50.423 INFO:teuthology.orchestra.console:Power off for smithi035 completed 2020-11-16T22:34:50.437 INFO:teuthology.orchestra.console:Power off for smithi055 completed 2020-11-16T22:34:50.516 INFO:teuthology.orchestra.console:Power on smithi025 2020-11-16T22:34:50.516 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-16T22:34:50.525 INFO:teuthology.orchestra.console:Power on smithi035 2020-11-16T22:34:50.526 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-16T22:34:50.538 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:50.545 INFO:teuthology.orchestra.console:Power on smithi055 2020-11-16T22:34:50.546 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-16T22:34:50.553 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:50.564 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:54.566 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi025.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:58.723 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi035.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:58.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-16T22:34:58.924 INFO:teuthology.orchestra.console:Power on for smithi025 completed 2020-11-16T22:34:58.934 INFO:teuthology.orchestra.console:Power on for smithi035 completed 2020-11-16T22:34:58.943 INFO:teuthology.orchestra.console:Power on for smithi055 completed 2020-11-16T22:34:59.026 INFO:teuthology.provision.fog.smithi025:Waiting for deploy to finish 2020-11-16T22:34:59.035 INFO:teuthology.provision.fog.smithi035:Waiting for deploy to finish 2020-11-16T22:34:59.044 INFO:teuthology.provision.fog.smithi055:Waiting for deploy to finish 2020-11-16T22:37:15.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:37:15.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:37:39.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:37:39.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:37:48.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:37:48.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:37:57.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:37:57.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:00.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:06.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:06.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:24.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:34.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:38.630 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-16T22:38:38.632 INFO:teuthology.orchestra.run.smithi035:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-16T22:38:43.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:52.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:38:53.494 INFO:teuthology.provision.fog.smithi035:Node is ready 2020-11-16T22:38:53.494 INFO:teuthology.orchestra.run.smithi035:> hostname 2020-11-16T22:38:53.513 INFO:teuthology.orchestra.run.smithi035.stdout:smithi035.front.sepia.ceph.com 2020-11-16T22:38:53.514 INFO:teuthology.orchestra.run.smithi035:> grep smithi035.front.sepia.ceph.com /etc/hosts 2020-11-16T22:38:53.570 INFO:teuthology.orchestra.run.smithi035.stdout:172.21.15.35 smithi035.front.sepia.ceph.com smithi035 2020-11-16T22:38:53.571 INFO:teuthology.orchestra.run.smithi035:> sudo hostname smithi035 2020-11-16T22:38:53.712 INFO:teuthology.orchestra.run.smithi035:> sudo sed -i -e 's/smithi035.front.sepia.ceph.com/smithi035/g' /etc/hosts 2020-11-16T22:38:53.746 INFO:teuthology.orchestra.run.smithi035:> sudo sed -i -e 's/172.21.15.35/172.21.15.35/g' /etc/hosts 2020-11-16T22:38:53.819 INFO:teuthology.orchestra.run.smithi035:> sudo sed -i -e 's/smithi035.front.sepia.ceph.com/smithi035/g' /etc/hostname 2020-11-16T22:38:53.891 INFO:teuthology.orchestra.run.smithi035:> sudo hostname smithi035 2020-11-16T22:38:53.962 INFO:teuthology.provision.fog.smithi035:Deploy complete! 2020-11-16T22:38:53.963 DEBUG:teuthology.parallel:result is None 2020-11-16T22:39:01.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:39:10.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:39:12.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:39:13.125 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-16T22:39:13.126 INFO:teuthology.orchestra.run.smithi055:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-16T22:39:13.430 INFO:teuthology.provision.fog.smithi055:Node is ready 2020-11-16T22:39:13.430 INFO:teuthology.orchestra.run.smithi055:> hostname 2020-11-16T22:39:13.444 INFO:teuthology.orchestra.run.smithi055.stdout:smithi055.front.sepia.ceph.com 2020-11-16T22:39:13.445 INFO:teuthology.orchestra.run.smithi055:> grep smithi055.front.sepia.ceph.com /etc/hosts 2020-11-16T22:39:13.499 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.15.55 smithi055.front.sepia.ceph.com smithi055 2020-11-16T22:39:13.499 INFO:teuthology.orchestra.run.smithi055:> sudo hostname smithi055 2020-11-16T22:39:13.621 INFO:teuthology.orchestra.run.smithi055:> sudo sed -i -e 's/smithi055.front.sepia.ceph.com/smithi055/g' /etc/hosts 2020-11-16T22:39:13.652 INFO:teuthology.orchestra.run.smithi055:> sudo sed -i -e 's/172.21.15.55/172.21.15.55/g' /etc/hosts 2020-11-16T22:39:13.724 INFO:teuthology.orchestra.run.smithi055:> sudo sed -i -e 's/smithi055.front.sepia.ceph.com/smithi055/g' /etc/hostname 2020-11-16T22:39:13.794 INFO:teuthology.orchestra.run.smithi055:> sudo hostname smithi055 2020-11-16T22:39:13.864 INFO:teuthology.provision.fog.smithi055:Deploy complete! 2020-11-16T22:39:13.865 DEBUG:teuthology.parallel:result is None 2020-11-16T22:40:16.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:40:16.587 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-16T22:40:16.588 INFO:teuthology.orchestra.run.smithi025:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-16T22:40:16.892 INFO:teuthology.provision.fog.smithi025:Node is ready 2020-11-16T22:40:16.892 INFO:teuthology.orchestra.run.smithi025:> hostname 2020-11-16T22:40:16.906 INFO:teuthology.orchestra.run.smithi025.stdout:smithi025.front.sepia.ceph.com 2020-11-16T22:40:16.907 INFO:teuthology.orchestra.run.smithi025:> grep smithi025.front.sepia.ceph.com /etc/hosts 2020-11-16T22:40:16.961 INFO:teuthology.orchestra.run.smithi025.stdout:172.21.15.25 smithi025.front.sepia.ceph.com smithi025 2020-11-16T22:40:16.962 INFO:teuthology.orchestra.run.smithi025:> sudo hostname smithi025 2020-11-16T22:40:17.090 INFO:teuthology.orchestra.run.smithi025:> sudo sed -i -e 's/smithi025.front.sepia.ceph.com/smithi025/g' /etc/hosts 2020-11-16T22:40:17.122 INFO:teuthology.orchestra.run.smithi025:> sudo sed -i -e 's/172.21.15.25/172.21.15.25/g' /etc/hosts 2020-11-16T22:40:17.193 INFO:teuthology.orchestra.run.smithi025:> sudo sed -i -e 's/smithi025.front.sepia.ceph.com/smithi025/g' /etc/hostname 2020-11-16T22:40:17.264 INFO:teuthology.orchestra.run.smithi025:> sudo hostname smithi025 2020-11-16T22:40:17.334 INFO:teuthology.provision.fog.smithi025:Deploy complete! 2020-11-16T22:40:17.335 DEBUG:teuthology.parallel:result is None 2020-11-16T22:40:58.927 INFO:teuthology.lock.ops:Checking smithi035.front.sepia.ceph.com 2020-11-16T22:40:59.334 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-16T22:40:59.350 INFO:teuthology.lock.ops:Checking smithi025.front.sepia.ceph.com 2020-11-16T22:40:59.350 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-16T22:40:59.362 INFO:teuthology.lock.ops:Checking smithi055.front.sepia.ceph.com 2020-11-16T22:40:59.363 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-16T22:40:59.376 INFO:teuthology.lock.ops:Updating [smithi035.front.sepia.ceph.com]: set os type and version on server 2020-11-16T22:40:59.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T22:40:59.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:40:59.445 INFO:teuthology.orchestra.run.smithi035:> true 2020-11-16T22:40:59.624 INFO:teuthology.orchestra.run.smithi035:> uname -m 2020-11-16T22:40:59.678 INFO:teuthology.orchestra.run.smithi035.stdout:x86_64 2020-11-16T22:40:59.679 INFO:teuthology.orchestra.run.smithi035:> cat /etc/os-release 2020-11-16T22:40:59.732 INFO:teuthology.orchestra.run.smithi035.stdout:NAME="Red Hat Enterprise Linux" 2020-11-16T22:40:59.732 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION="8.1 (Ootpa)" 2020-11-16T22:40:59.732 INFO:teuthology.orchestra.run.smithi035.stdout:ID="rhel" 2020-11-16T22:40:59.733 INFO:teuthology.orchestra.run.smithi035.stdout:ID_LIKE="fedora" 2020-11-16T22:40:59.733 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION_ID="8.1" 2020-11-16T22:40:59.733 INFO:teuthology.orchestra.run.smithi035.stdout:PLATFORM_ID="platform:el8" 2020-11-16T22:40:59.733 INFO:teuthology.orchestra.run.smithi035.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)" 2020-11-16T22:40:59.733 INFO:teuthology.orchestra.run.smithi035.stdout:ANSI_COLOR="0;31" 2020-11-16T22:40:59.733 INFO:teuthology.orchestra.run.smithi035.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA" 2020-11-16T22:40:59.734 INFO:teuthology.orchestra.run.smithi035.stdout:HOME_URL="https://www.redhat.com/" 2020-11-16T22:40:59.734 INFO:teuthology.orchestra.run.smithi035.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-16T22:40:59.734 INFO:teuthology.orchestra.run.smithi035.stdout: 2020-11-16T22:40:59.734 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-16T22:40:59.734 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1 2020-11-16T22:40:59.734 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-16T22:40:59.734 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1" 2020-11-16T22:40:59.736 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server 2020-11-16T22:40:59.751 INFO:teuthology.lock.ops:Updating [smithi025.front.sepia.ceph.com]: set os type and version on server 2020-11-16T22:40:59.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T22:40:59.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:40:59.822 INFO:teuthology.orchestra.run.smithi025:> true 2020-11-16T22:41:00.004 INFO:teuthology.orchestra.run.smithi025:> uname -m 2020-11-16T22:41:00.059 INFO:teuthology.orchestra.run.smithi025.stdout:x86_64 2020-11-16T22:41:00.060 INFO:teuthology.orchestra.run.smithi025:> cat /etc/os-release 2020-11-16T22:41:00.114 INFO:teuthology.orchestra.run.smithi025.stdout:NAME="Red Hat Enterprise Linux" 2020-11-16T22:41:00.115 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION="8.1 (Ootpa)" 2020-11-16T22:41:00.115 INFO:teuthology.orchestra.run.smithi025.stdout:ID="rhel" 2020-11-16T22:41:00.115 INFO:teuthology.orchestra.run.smithi025.stdout:ID_LIKE="fedora" 2020-11-16T22:41:00.115 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_ID="8.1" 2020-11-16T22:41:00.115 INFO:teuthology.orchestra.run.smithi025.stdout:PLATFORM_ID="platform:el8" 2020-11-16T22:41:00.115 INFO:teuthology.orchestra.run.smithi025.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)" 2020-11-16T22:41:00.116 INFO:teuthology.orchestra.run.smithi025.stdout:ANSI_COLOR="0;31" 2020-11-16T22:41:00.116 INFO:teuthology.orchestra.run.smithi025.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA" 2020-11-16T22:41:00.116 INFO:teuthology.orchestra.run.smithi025.stdout:HOME_URL="https://www.redhat.com/" 2020-11-16T22:41:00.116 INFO:teuthology.orchestra.run.smithi025.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-16T22:41:00.116 INFO:teuthology.orchestra.run.smithi025.stdout: 2020-11-16T22:41:00.116 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-16T22:41:00.117 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1 2020-11-16T22:41:00.117 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-16T22:41:00.117 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1" 2020-11-16T22:41:00.117 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server 2020-11-16T22:41:00.134 INFO:teuthology.lock.ops:Updating [smithi055.front.sepia.ceph.com]: set os type and version on server 2020-11-16T22:41:00.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-16T22:41:00.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:41:00.208 INFO:teuthology.orchestra.run.smithi055:> true 2020-11-16T22:41:00.392 INFO:teuthology.orchestra.run.smithi055:> uname -m 2020-11-16T22:41:00.447 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2020-11-16T22:41:00.448 INFO:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2020-11-16T22:41:00.502 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="Red Hat Enterprise Linux" 2020-11-16T22:41:00.502 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="8.1 (Ootpa)" 2020-11-16T22:41:00.503 INFO:teuthology.orchestra.run.smithi055.stdout:ID="rhel" 2020-11-16T22:41:00.503 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="fedora" 2020-11-16T22:41:00.503 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="8.1" 2020-11-16T22:41:00.503 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el8" 2020-11-16T22:41:00.503 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)" 2020-11-16T22:41:00.503 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2020-11-16T22:41:00.504 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA" 2020-11-16T22:41:00.504 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://www.redhat.com/" 2020-11-16T22:41:00.504 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-16T22:41:00.504 INFO:teuthology.orchestra.run.smithi055.stdout: 2020-11-16T22:41:00.504 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-16T22:41:00.504 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1 2020-11-16T22:41:00.505 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-16T22:41:00.505 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1" 2020-11-16T22:41:00.505 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2020-11-16T22:41:00.520 INFO:teuthology.task.internal.lock_machines:3 smithi machines locked this try, 3/3 locked so far 2020-11-16T22:41:00.569 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaP4Q085lgHKi8iqpuM1Raet6UKXvfQM4UqwnVxTIOrNG3fRCpX2VTu2DLFwubohflgrH1PBJ4NMZfbfHeMzGHMcPXXxDnaQX410j35x+srqKpo1MOEy+RsQcXxF7Hb8G5FigF1SslrcF7xd+MyGxXcq+rzmzWdK6bMavSXQCkONWbEZvT2NWeGk9FJmtV/OPeSHWGdUv+SFcPOjhLwwtmSHubQw7f2zvtEZXu+XJsAZGjHFaiuSA/3N+9WE/7ilwGF6IKVbwxcmnt1C6mg+QPapVTkeYPjSBIRJEpwt1/tFEjbDiLJOlABkFy3Ibh5g8EdFAWItzcTppF8nuCzYcCZcfdSWx7dRI2ZYoMUrbHv5ixbj+aqigOSgvYbdZw3mz+pOZecXrKYo9aK7H8IiXNnDDaL97tjIdLoSgoJ0DjCSIyKJQTqnShtKezYx/sZqmi8dsTe+rMabjfmpHMCqNnyv4uVSzZI6Ab65l6T5D8x0EK7frlRn1GiEARUX/WBpU= smithi035.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhCKyOsBqe0fU3GGue8Y42+uIRuMcPclpHrLhq8buvV5MUv/kWgnWh7FdJZX8EPKUf+uf4B6kqnv2kPwRU9sOpp6su1waG4fkhpwnk6I+QII3fbcXdRx6/R92/MS8fepK/vEgVSkTAi0UbBLPiJGsImxXftdrUTEz1Wj1bHWuX0HxUksUUwEVeKlbwT78jhnMbccUjNoXM6ivnSJ6ijJfqd8DiFRA/zXWKMquHKYe3lNV2ztXGHsISCfN0HcOwARwKewSlfo2gDa3O/AiA2HUbippHHIDIiDytzu2DSgZ+OsO93PaelzXwLK5fSV31HpgnEiXXRx4QyGrlgsDcdwl7uvEnazn82rp9sdXchcrGMske7fWYVh3Xqc7kF/Xbg7TdE4QENqrICEFV7WR8gs3LtLJp96urf11dssV6qEpWuUmH9/mP1rjkxpp045dt3lKxxmwrdTrkloBEc1tjS/Cuz5uB2VV0TrAYyRiMj+VjN3TOJHYGgOjDUCwmlkhyc9k= smithi055.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc5AZwZkQaQ5OcMg19eTvaVi/n47bnljGIUjGyWv+qUItLWypSxPKluu+9p265rWbPHxIs/G3qf3SzYBaPZ2FX8z8miRHPV5iUmsVHLPE7QqzQWhS9Nvs5b6ZDwFuxXqaB0BeAcOKv10n5Qg+YLrbRdzw9epQ0EDcHQumcvvIArczMJr+c+NlMovp2oWkimn8za34tv77Zkjqpn0h3ED+1uC6ngzL4co8UYQJiabMYRRkj1BWJUGVMjwl6Y8umS/HXatyw2F/zsm5JJP8v5ujBs8thz0SkXKquOWEZ1UUp3JWULx6gVzoUtY6/ztjFiHGsvswkahtq2a0EJGR7pY+8iMV+hjrA0OuNAsjI2HAiYoDX9NxPdgTaUbTgR+VI8awD26qgc81OG1jBL2TSetokX+86yqDHOZdSlvLBp1JH7WNMCSszv0vRaT9x5BiEbvh1lv7EtjEwjgzoZs9Bz6rXBy99CnoDbqLm/8PDPdJWQzmd8CodzGjxS2wGN90p9xk= 2020-11-16T22:41:00.569 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T22:41:00.621 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-16T22:41:00.830 INFO:teuthology.task.internal:Saving configuration 2020-11-16T22:41:01.254 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-16T22:41:01.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-16T22:41:01.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-16 22:34:37.300043', 'description': '/home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631938', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObVZSVUWjOuOwzfmgWfwDbJrpgD3I5lWuOc9WIgOlCQ1ZKBsXliI8AsSwEWxJI2fu94FLlDHhpArKdAdfASWLo=', 'os_version': '8.1', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi035.front.sepia.ceph.com'} 2020-11-16T22:41:01.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-16 22:34:37.300063', 'description': '/home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631938', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGYBUMweC8EBoycAbHLhCeZmlbqV0CjTqHtt91iXU1C5Q0avoFz7RoPKzVfDMnebobi7I9syTovsBsYCzSRWR0o=', 'os_version': '8.1', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi025.front.sepia.ceph.com'} 2020-11-16T22:41:01.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-16 22:34:37.300078', 'description': '/home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631938', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGjQyC/BVtk/d0aYDXgdEnaCvtz3RWmRfRUmCiSF9Rz0S0H1PhpfZ1YR4sC35tWyMiSnad/mL0TbdnRrCPSHUMc=', 'os_version': '8.1', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi055.front.sepia.ceph.com'} 2020-11-16T22:41:01.514 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-16T22:41:01.726 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2020-11-16T22:41:01.727 INFO:teuthology.task.internal:roles: ubuntu@smithi025.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'mds.b', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2020-11-16T22:41:01.727 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['client.0', 'client.1'] 2020-11-16T22:41:01.727 INFO:teuthology.run_tasks:Running task console_log... 2020-11-16T22:41:01.825 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc233dfe1e0>, signals=[15]) 2020-11-16T22:41:01.825 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-16T22:41:01.833 INFO:teuthology.task.internal:Opening connections... 2020-11-16T22:41:01.833 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi035.front.sepia.ceph.com 2020-11-16T22:41:01.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi035.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:41:01.909 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi025.front.sepia.ceph.com 2020-11-16T22:41:01.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:41:02.398 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2020-11-16T22:41:02.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-16T22:41:02.472 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-16T22:41:14.522 INFO:teuthology.orchestra.run.smithi035:> uname -m 2020-11-16T22:41:14.541 INFO:teuthology.orchestra.run.smithi035.stdout:x86_64 2020-11-16T22:41:14.542 INFO:teuthology.orchestra.run.smithi035:> cat /etc/os-release 2020-11-16T22:41:14.598 INFO:teuthology.orchestra.run.smithi035.stdout:NAME="Red Hat Enterprise Linux" 2020-11-16T22:41:14.598 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION="8.1 (Ootpa)" 2020-11-16T22:41:14.598 INFO:teuthology.orchestra.run.smithi035.stdout:ID="rhel" 2020-11-16T22:41:14.598 INFO:teuthology.orchestra.run.smithi035.stdout:ID_LIKE="fedora" 2020-11-16T22:41:14.598 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION_ID="8.1" 2020-11-16T22:41:14.599 INFO:teuthology.orchestra.run.smithi035.stdout:PLATFORM_ID="platform:el8" 2020-11-16T22:41:14.599 INFO:teuthology.orchestra.run.smithi035.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)" 2020-11-16T22:41:14.599 INFO:teuthology.orchestra.run.smithi035.stdout:ANSI_COLOR="0;31" 2020-11-16T22:41:14.599 INFO:teuthology.orchestra.run.smithi035.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA" 2020-11-16T22:41:14.599 INFO:teuthology.orchestra.run.smithi035.stdout:HOME_URL="https://www.redhat.com/" 2020-11-16T22:41:14.599 INFO:teuthology.orchestra.run.smithi035.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-16T22:41:14.599 INFO:teuthology.orchestra.run.smithi035.stdout: 2020-11-16T22:41:14.600 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-16T22:41:14.600 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1 2020-11-16T22:41:14.600 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-16T22:41:14.600 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1" 2020-11-16T22:41:14.600 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server 2020-11-16T22:41:14.818 INFO:teuthology.orchestra.run.smithi025:> uname -m 2020-11-16T22:41:14.864 INFO:teuthology.orchestra.run.smithi025.stdout:x86_64 2020-11-16T22:41:14.864 INFO:teuthology.orchestra.run.smithi025:> cat /etc/os-release 2020-11-16T22:41:14.920 INFO:teuthology.orchestra.run.smithi025.stdout:NAME="Red Hat Enterprise Linux" 2020-11-16T22:41:14.920 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION="8.1 (Ootpa)" 2020-11-16T22:41:14.921 INFO:teuthology.orchestra.run.smithi025.stdout:ID="rhel" 2020-11-16T22:41:14.921 INFO:teuthology.orchestra.run.smithi025.stdout:ID_LIKE="fedora" 2020-11-16T22:41:14.921 INFO:teuthology.orchestra.run.smithi025.stdout:VERSION_ID="8.1" 2020-11-16T22:41:14.921 INFO:teuthology.orchestra.run.smithi025.stdout:PLATFORM_ID="platform:el8" 2020-11-16T22:41:14.921 INFO:teuthology.orchestra.run.smithi025.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)" 2020-11-16T22:41:14.921 INFO:teuthology.orchestra.run.smithi025.stdout:ANSI_COLOR="0;31" 2020-11-16T22:41:14.922 INFO:teuthology.orchestra.run.smithi025.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA" 2020-11-16T22:41:14.922 INFO:teuthology.orchestra.run.smithi025.stdout:HOME_URL="https://www.redhat.com/" 2020-11-16T22:41:14.922 INFO:teuthology.orchestra.run.smithi025.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-16T22:41:14.922 INFO:teuthology.orchestra.run.smithi025.stdout: 2020-11-16T22:41:14.922 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-16T22:41:14.922 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1 2020-11-16T22:41:14.922 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-16T22:41:14.923 INFO:teuthology.orchestra.run.smithi025.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1" 2020-11-16T22:41:14.923 INFO:teuthology.lock.ops:Updating smithi025.front.sepia.ceph.com on lock server 2020-11-16T22:41:14.936 INFO:teuthology.orchestra.run.smithi055:> uname -m 2020-11-16T22:41:14.954 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2020-11-16T22:41:14.955 INFO:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2020-11-16T22:41:15.011 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="Red Hat Enterprise Linux" 2020-11-16T22:41:15.011 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="8.1 (Ootpa)" 2020-11-16T22:41:15.011 INFO:teuthology.orchestra.run.smithi055.stdout:ID="rhel" 2020-11-16T22:41:15.012 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="fedora" 2020-11-16T22:41:15.012 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="8.1" 2020-11-16T22:41:15.012 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el8" 2020-11-16T22:41:15.012 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)" 2020-11-16T22:41:15.012 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2020-11-16T22:41:15.012 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA" 2020-11-16T22:41:15.012 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://www.redhat.com/" 2020-11-16T22:41:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-16T22:41:15.013 INFO:teuthology.orchestra.run.smithi055.stdout: 2020-11-16T22:41:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-16T22:41:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1 2020-11-16T22:41:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-16T22:41:15.013 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1" 2020-11-16T22:41:15.014 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2020-11-16T22:41:15.027 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-16T22:41:15.242 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-16T22:41:15.654 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-16T22:41:15.655 INFO:teuthology.orchestra.run.smithi025:> test '!' -e /home/ubuntu/cephtest 2020-11-16T22:41:15.658 INFO:teuthology.orchestra.run.smithi035:> test '!' -e /home/ubuntu/cephtest 2020-11-16T22:41:15.660 INFO:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2020-11-16T22:41:15.676 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-16T22:41:15.886 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-16T22:41:15.886 INFO:teuthology.orchestra.run.smithi025:> test -z $(ls -A /var/lib/ceph) 2020-11-16T22:41:15.889 INFO:teuthology.orchestra.run.smithi035:> test -z $(ls -A /var/lib/ceph) 2020-11-16T22:41:15.891 INFO:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2020-11-16T22:41:15.922 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-16T22:41:15.980 INFO:teuthology.run_tasks:Running task kernel... 2020-11-16T22:41:16.001 DEBUG:teuthology.task.kernel:normalized config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2020-11-16T22:41:16.002 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2020-11-16T22:41:16.002 INFO:teuthology.orchestra.run.smithi055:> uname -r 2020-11-16T22:41:16.019 INFO:teuthology.orchestra.run.smithi055.stdout:4.18.0-193.el8.x86_64 2020-11-16T22:41:16.019 INFO:teuthology.task.kernel:Running kernel on smithi055: 4.18.0-193.el8.x86_64 2020-11-16T22:41:16.019 INFO:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2020-11-16T22:41:17.923 INFO:teuthology.orchestra.run.smithi055.stdout:Updating Subscription Management repositories. 2020-11-16T22:41:17.940 INFO:teuthology.orchestra.run.smithi055.stdout:Unable to read consumer identity 2020-11-16T22:41:18.077 INFO:teuthology.orchestra.run.smithi055.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-16T22:41:19.537 INFO:teuthology.orchestra.run.smithi055.stdout:Copr repo for ceph-el8 owned by ktdreyer 18 kB/s | 2.9 kB 00:00 2020-11-16T22:41:41.380 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 19 MB/s | 8.3 MB 00:00 2020-11-16T22:41:41.380 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 3.9 MB/s | 23 kB 00:00 2020-11-16T22:41:41.380 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-4.18.0-193.el8.x86_64 is already installed. 2020-11-16T22:41:41.380 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-11-16T22:41:41.381 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2020-11-16T22:41:41.381 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2020-11-16T22:41:41.381 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2020-11-16T22:41:41.381 INFO:teuthology.orchestra.run.smithi055.stderr:Modular dependency problem: 2020-11-16T22:41:41.381 INFO:teuthology.orchestra.run.smithi055.stderr: 2020-11-16T22:41:41.381 INFO:teuthology.orchestra.run.smithi055.stderr: Problem: conflicting requests 2020-11-16T22:41:41.382 INFO:teuthology.orchestra.run.smithi055.stderr: - nothing provides module(perl:5.26) needed by module perl-YAML:1.24:8010020190322131811:a5949e2e-0.x86_64 2020-11-16T22:41:41.382 INFO:teuthology.orchestra.run.smithi055:> echo no | sudo yum reinstall kernel || true 2020-11-16T22:41:41.750 INFO:teuthology.orchestra.run.smithi055.stdout:Updating Subscription Management repositories. 2020-11-16T22:41:41.750 INFO:teuthology.orchestra.run.smithi055.stdout:Unable to read consumer identity 2020-11-16T22:41:41.804 INFO:teuthology.orchestra.run.smithi055.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-16T22:41:41.975 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:20 ago on Mon 16 Nov 2020 10:41:21 PM UTC. 2020-11-16T22:41:42.005 INFO:teuthology.orchestra.run.smithi055.stderr:Modular dependency problem: 2020-11-16T22:41:42.006 INFO:teuthology.orchestra.run.smithi055.stderr: 2020-11-16T22:41:42.006 INFO:teuthology.orchestra.run.smithi055.stderr: Problem: conflicting requests 2020-11-16T22:41:42.006 INFO:teuthology.orchestra.run.smithi055.stderr: - nothing provides module(perl:5.26) needed by module perl-YAML:1.24:8010020190322131811:a5949e2e-0.x86_64 2020-11-16T22:41:42.100 INFO:teuthology.orchestra.run.smithi055.stdout:Installed package kernel-4.18.0-193.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2020-11-16T22:41:42.507 INFO:teuthology.orchestra.run.smithi055.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-16T22:41:42.507 INFO:teuthology.orchestra.run.smithi055.stderr:Error: No packages marked for reinstall. 2020-11-16T22:41:42.508 INFO:teuthology.orchestra.run.smithi055:> sudo yum reinstall -y kernel || true 2020-11-16T22:41:42.862 INFO:teuthology.orchestra.run.smithi055.stdout:Updating Subscription Management repositories. 2020-11-16T22:41:42.863 INFO:teuthology.orchestra.run.smithi055.stdout:Unable to read consumer identity 2020-11-16T22:41:42.910 INFO:teuthology.orchestra.run.smithi055.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-16T22:41:43.078 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:22 ago on Mon 16 Nov 2020 10:41:21 PM UTC. 2020-11-16T22:41:47.363 INFO:teuthology.orchestra.run.smithi055.stderr:Modular dependency problem: 2020-11-16T22:41:47.364 INFO:teuthology.orchestra.run.smithi055.stderr: 2020-11-16T22:41:47.364 INFO:teuthology.orchestra.run.smithi055.stderr: Problem: conflicting requests 2020-11-16T22:41:47.364 INFO:teuthology.orchestra.run.smithi055.stderr: - nothing provides module(perl:5.26) needed by module perl-YAML:1.24:8010020190322131811:a5949e2e-0.x86_64 2020-11-16T22:41:47.364 INFO:teuthology.orchestra.run.smithi055.stderr:Error: No packages marked for reinstall. 2020-11-16T22:41:47.364 INFO:teuthology.orchestra.run.smithi055.stdout:Installed package kernel-4.18.0-193.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2020-11-16T22:41:47.364 INFO:teuthology.orchestra.run.smithi055.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-16T22:41:47.365 INFO:teuthology.orchestra.run.smithi055:> rpm -q kernel --last | head -n 1 2020-11-16T22:41:47.447 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-4.18.0-193.el8.x86_64 Sun 10 May 2020 01:56:55 PM UTC 2020-11-16T22:41:47.448 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-193.el8.x86_64 2020-11-16T22:41:47.449 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-16T22:41:47.449 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-16T22:41:47.458 INFO:teuthology.task.internal:Creating test directory... 2020-11-16T22:41:47.459 INFO:teuthology.orchestra.run.smithi025:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-16T22:41:47.461 INFO:teuthology.orchestra.run.smithi035:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-16T22:41:47.464 INFO:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-16T22:41:47.479 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-16T22:41:47.492 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-16T22:41:47.502 INFO:teuthology.task.internal:Creating archive directory... 2020-11-16T22:41:47.503 INFO:teuthology.orchestra.run.smithi025:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-16T22:41:47.519 INFO:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-16T22:41:47.521 INFO:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-16T22:41:47.558 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-16T22:41:47.770 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-16T22:41:47.771 INFO:teuthology.orchestra.run.smithi025:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-16T22:41:47.773 INFO:teuthology.orchestra.run.smithi035:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-16T22:41:47.775 INFO:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-16T22:41:47.825 INFO:teuthology.orchestra.run.smithi035.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:41:47.826 INFO:teuthology.orchestra.run.smithi025.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:41:47.851 INFO:teuthology.orchestra.run.smithi035.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:41:47.852 INFO:teuthology.orchestra.run.smithi025.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:41:47.862 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:41:47.888 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-16T22:41:47.889 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-16T22:41:47.897 INFO:teuthology.task.internal:Configuring sudo... 2020-11-16T22:41:47.897 INFO:teuthology.orchestra.run.smithi025:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-16T22:41:47.900 INFO:teuthology.orchestra.run.smithi035:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-16T22:41:47.902 INFO:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-16T22:41:47.937 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-16T22:41:47.947 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-16T22:41:47.948 INFO:teuthology.orchestra.run.smithi025:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-16T22:41:47.974 INFO:teuthology.orchestra.run.smithi035:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-16T22:41:47.976 INFO:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-16T22:41:47.993 INFO:teuthology.orchestra.run.smithi035:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T22:41:48.063 INFO:teuthology.orchestra.run.smithi035:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T22:41:48.137 INFO:teuthology.orchestra.run.smithi035:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T22:41:48.169 INFO:teuthology.orchestra.run.smithi035:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T22:41:48.224 INFO:teuthology.orchestra.run.smithi035:> set -ex 2020-11-16T22:41:48.224 INFO:teuthology.orchestra.run.smithi035:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T22:41:48.278 INFO:teuthology.orchestra.run.smithi035.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T22:41:48.294 INFO:teuthology.orchestra.run.smithi035.stderr:0+1 records in 2020-11-16T22:41:48.294 INFO:teuthology.orchestra.run.smithi035.stderr:0+1 records out 2020-11-16T22:41:48.294 INFO:teuthology.orchestra.run.smithi035.stderr:152 bytes copied, 3.4269e-05 s, 4.4 MB/s 2020-11-16T22:41:48.295 INFO:teuthology.orchestra.run.smithi025:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T22:41:48.331 INFO:teuthology.orchestra.run.smithi025:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T22:41:48.401 INFO:teuthology.orchestra.run.smithi025:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T22:41:48.476 INFO:teuthology.orchestra.run.smithi025:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T22:41:56.482 INFO:teuthology.orchestra.run.smithi025:> set -ex 2020-11-16T22:41:56.483 INFO:teuthology.orchestra.run.smithi025:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T22:41:56.538 INFO:teuthology.orchestra.run.smithi025.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T22:41:56.556 INFO:teuthology.orchestra.run.smithi025.stderr:0+1 records in 2020-11-16T22:41:56.556 INFO:teuthology.orchestra.run.smithi025.stderr:0+1 records out 2020-11-16T22:41:56.556 INFO:teuthology.orchestra.run.smithi025.stderr:152 bytes copied, 3.9433e-05 s, 3.9 MB/s 2020-11-16T22:41:56.557 INFO:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T22:41:56.595 INFO:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-16T22:41:56.668 INFO:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T22:41:56.704 INFO:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-16T22:42:11.446 INFO:teuthology.orchestra.run.smithi055:> set -ex 2020-11-16T22:42:11.447 INFO:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T22:42:11.501 INFO:teuthology.orchestra.run.smithi055.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-16T22:42:11.519 INFO:teuthology.orchestra.run.smithi055.stderr:0+1 records in 2020-11-16T22:42:11.519 INFO:teuthology.orchestra.run.smithi055.stderr:0+1 records out 2020-11-16T22:42:11.520 INFO:teuthology.orchestra.run.smithi055.stderr:152 bytes copied, 3.7128e-05 s, 4.1 MB/s 2020-11-16T22:42:11.520 INFO:teuthology.orchestra.run.smithi025:> sudo service rsyslog restart 2020-11-16T22:42:11.523 INFO:teuthology.orchestra.run.smithi035:> sudo service rsyslog restart 2020-11-16T22:42:11.525 INFO:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2020-11-16T22:42:11.606 INFO:teuthology.orchestra.run.smithi035.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-16T22:42:11.611 INFO:teuthology.orchestra.run.smithi025.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-16T22:42:11.624 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-16T22:42:12.050 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-16T22:42:13.482 INFO:teuthology.task.internal:Starting timer... 2020-11-16T22:42:14.506 INFO:teuthology.run_tasks:Running task pcp... 2020-11-16T22:42:17.369 INFO:teuthology.run_tasks:Running task selinux... 2020-11-16T22:42:17.579 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-11-16T22:42:17.580 INFO:teuthology.orchestra.run.smithi025:> sudo service auditd rotate 2020-11-16T22:42:17.655 INFO:teuthology.orchestra.run.smithi025.stdout:Rotating logs: [ OK ] 2020-11-16T22:42:17.656 INFO:teuthology.orchestra.run.smithi035:> sudo service auditd rotate 2020-11-16T22:42:17.735 INFO:teuthology.orchestra.run.smithi035.stdout:Rotating logs: [ OK ] 2020-11-16T22:42:17.737 INFO:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2020-11-16T22:42:18.018 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: [ OK ] 2020-11-16T22:42:18.019 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-16T22:42:18.020 INFO:teuthology.orchestra.run.smithi035:> /usr/sbin/getenforce 2020-11-16T22:42:18.051 INFO:teuthology.orchestra.run.smithi035.stdout:Permissive 2020-11-16T22:42:18.051 INFO:teuthology.orchestra.run.smithi025:> /usr/sbin/getenforce 2020-11-16T22:42:18.083 INFO:teuthology.orchestra.run.smithi025.stdout:Permissive 2020-11-16T22:42:18.084 INFO:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2020-11-16T22:42:18.107 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2020-11-16T22:42:18.108 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi035.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi025.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi055.front.sepia.ceph.com': 'permissive'} 2020-11-16T22:42:18.108 INFO:teuthology.orchestra.run.smithi035:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-16T22:42:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-16T22:42:18.145 INFO:teuthology.orchestra.run.smithi025:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-16T22:42:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-16T22:42:18.182 INFO:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-16T22:42:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-16T22:42:18.219 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-16T22:42:18.219 INFO:teuthology.orchestra.run.smithi035:> sudo /usr/sbin/setenforce permissive 2020-11-16T22:42:18.254 INFO:teuthology.orchestra.run.smithi025:> sudo /usr/sbin/setenforce permissive 2020-11-16T22:42:18.288 INFO:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2020-11-16T22:42:18.356 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-16T22:42:18.771 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-16T22:42:39.193 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-16T22:42:39.243 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-16T22:42:44.355 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-16T22:42:44.442 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi035.front.sepia.ceph.com,smithi025.front.sepia.ceph.com,smithi055.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-16T22:42:44.749 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-16T22:42:45.539 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-16T22:59:59.481 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* Monday 16 November 2020 22:42:47 +0000 (0:00:00.064) 0:00:00.064 ******* Monday 16 November 2020 22:42:47 +0000 (0:00:00.014) 0:00:00.079 ******* Monday 16 November 2020 22:42:47 +0000 (0:00:00.014) 0:00:00.093 ******* TASK [Gathering Facts] ********************************************************* ok: [smithi035.front.sepia.ceph.com] Monday 16 November 2020 22:42:49 +0000 (0:00:01.780) 0:00:01.874 ******* ok: [smithi025.front.sepia.ceph.com] ok: [smithi055.front.sepia.ceph.com] Monday 16 November 2020 22:42:57 +0000 (0:00:08.160) 0:00:10.034 ******* Monday 16 November 2020 22:42:57 +0000 (0:00:00.028) 0:00:10.063 ******* TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi025.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi035.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} Monday 16 November 2020 22:42:57 +0000 (0:00:00.218) 0:00:10.281 ******* Monday 16 November 2020 22:42:57 +0000 (0:00:00.028) 0:00:10.310 ******* ok: [smithi055.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} Monday 16 November 2020 22:42:58 +0000 (0:00:00.038) 0:00:10.348 ******* TASK [ansible-managed : Create the ansible user.] ****************************** ok: [smithi035.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [smithi025.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", 2020-11-16T22:59:59.687 INFO:teuthology.task.ansible.out:"state": "present", "uid": 1100} ok: [smithi055.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Monday 16 November 2020 22:42:58 +0000 (0:00:00.369) 0:00:10.717 ******* Monday 16 November 2020 22:42:58 +0000 (0:00:00.028) 0:00:10.746 ******* Monday 16 November 2020 22:42:58 +0000 (0:00:00.027) 0:00:10.774 ******* TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi035.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 16 November 2020 22:42:58 +0000 (0:00:00.036) 0:00:10.810 ******* Monday 16 November 2020 22:42:58 +0000 (0:00:00.026) 0:00:10.837 ******* skipping: [smithi055.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 16 November 2020 22:42:58 +0000 (0:00:00.040) 0:00:10.877 ******* TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi025.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi055.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi035.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Monday 16 November 2020 22:42:58 +0000 (0:00:00.276) 0:00:11.154 ******* Monday 16 November 2020 22:42:58 +0000 (0:00:00.028) 0:00:11.182 ******* Monday 16 November 2020 22:42:58 +0000 (0:00:00.027) 0:00:11.210 ******* TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi035.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephl 2020-11-16T22:59:59.891 INFO:teuthology.task.ansible.out:ab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605566578.8578608-41607413808583/source", "state": "file", "uid": 0} changed: [smithi025.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605566578.829592-154144141341790/source", "state": "file", "uid": 0} Monday 16 November 2020 22:42:59 +0000 (0:00:00.803) 0:00:12.014 ******* Monday 16 November 2020 22:42:59 +0000 (0:00:00.029) 0:00:12.043 ******* changed: [smithi055.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605566578.8857603-274772498693337/source", "state": "file", "uid": 0} Monday 16 November 2020 22:42:59 +0000 (0:00:00.038) 0:00:12.082 ******* TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi055.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@