2020-06-15T13:50:11.467 INFO:root:teuthology version: 1.0.0-b8c39ea 2020-06-15T13:50:11.470 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150372 --description rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/mimic.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_7.6.yaml} msgr-failures/few.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} --verbose --lock --name kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi --owner scheduled_kchai@teuthology --block -- /tmp/teuthology-worker.6uk8p4nx.tmp 2020-06-15T13:50:11.507 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-15T13:50:11.581 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150372 branch: wip-kefu-testing-2020-06-15-1050 description: rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/mimic.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_7.6.yaml} msgr-failures/few.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '5150372' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-kefu-testing-2020-06-15-1050 ceph: conf: global: mon client directed command retry: 5 mon warn on msgr2 not enabled: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd backoff on degraded: true osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request mon_bind_msgr2: false sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2020-06-15-1050 sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2020-06-15-1050 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-06-15-1050/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 tasks: - install: branch: mimic exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-immutable-object-cache - ceph-base - python3-rados - python3-rgw - python3-rbd - python3-cephfs - librados-devel extra_packages: - librados2 - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: null - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - rados: clients: - client.2 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.7016 2020-06-15T13:50:11.581 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-06-15-1050/qa; will attempt to use it 2020-06-15T13:50:11.582 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-06-15-1050/qa/tasks 2020-06-15T13:50:11.583 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-06-15T13:50:11.586 INFO:teuthology.task.internal:Checking packages... 2020-06-15T13:50:11.626 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '76f35ab4e1d5fd49cb6131e102a9258075936e41' 2020-06-15T13:50:11.626 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-06-15T13:50:11.627 INFO:teuthology.packaging:ref: None 2020-06-15T13:50:11.627 INFO:teuthology.packaging:tag: None 2020-06-15T13:50:11.627 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-06-15-1050 2020-06-15T13:50:11.628 INFO:teuthology.packaging:sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 2020-06-15T13:50:11.628 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2020-06-15-1050 2020-06-15T13:50:11.837 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-2495.g76f35ab 2020-06-15T13:50:11.839 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-06-15T13:50:11.850 INFO:teuthology.task.internal:no buildpackages task found 2020-06-15T13:50:11.850 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-06-15T13:50:11.870 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-06-15T13:50:11.871 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-15T13:50:11.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-06-15T13:50:21.944 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:50:31.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:50:41.992 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150372'} 2020-06-15T13:50:42.020 DEBUG:teuthology.lock.ops:locked ubuntu@smithi096.front.sepia.ceph.com, ubuntu@smithi145.front.sepia.ceph.com, ubuntu@smithi109.front.sepia.ceph.com 2020-06-15T13:50:42.166 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe2bb6cb158>, signals=[15]) 2020-06-15T13:50:42.407 INFO:teuthology.provision.fog.smithi096:Scheduling deploy of centos 7.6 2020-06-15T13:50:42.412 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of centos 7.6 2020-06-15T13:50:42.415 INFO:teuthology.provision.fog.smithi109:Scheduling deploy of centos 7.6 2020-06-15T13:50:42.771 INFO:teuthology.orchestra.console:Power off smithi145 2020-06-15T13:50:42.772 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-15T13:50:42.813 INFO:teuthology.orchestra.console:Power off smithi096 2020-06-15T13:50:42.815 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-15T13:50:42.830 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:42.865 INFO:teuthology.orchestra.console:Power off smithi109 2020-06-15T13:50:42.866 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-15T13:50:42.880 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:42.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:42.980 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2020-06-15T13:50:43.015 INFO:teuthology.orchestra.console:Power off for smithi096 completed 2020-06-15T13:50:43.032 INFO:teuthology.orchestra.console:Power off for smithi109 completed 2020-06-15T13:50:43.082 INFO:teuthology.orchestra.console:Power on smithi145 2020-06-15T13:50:43.082 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-15T13:50:43.115 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:43.136 INFO:teuthology.orchestra.console:Power on smithi096 2020-06-15T13:50:43.137 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-15T13:50:43.149 INFO:teuthology.orchestra.console:Power on smithi109 2020-06-15T13:50:43.151 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-15T13:50:43.168 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:43.186 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:47.186 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:47.208 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:47.227 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:51.304 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:51.329 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:51.343 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi109.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-15T13:50:51.535 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2020-06-15T13:50:51.544 INFO:teuthology.orchestra.console:Power on for smithi096 completed 2020-06-15T13:50:51.629 INFO:teuthology.orchestra.console:Power on for smithi109 completed 2020-06-15T13:50:51.637 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2020-06-15T13:50:51.645 INFO:teuthology.provision.fog.smithi096:Waiting for deploy to finish 2020-06-15T13:50:51.730 INFO:teuthology.provision.fog.smithi109:Waiting for deploy to finish 2020-06-15T13:53:22.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:22.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:22.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:31.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:31.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:32.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:41.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:41.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:41.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:50.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:50.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:50.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:59.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:59.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:53:59.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:08.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:08.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:08.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:17.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:17.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:17.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:26.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:26.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:54.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:54:55.187 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-15T13:54:55.189 INFO:teuthology.orchestra.run.smithi109:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-15T13:55:03.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:04.102 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-15T13:55:04.103 INFO:teuthology.orchestra.run.smithi096:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-15T13:55:10.532 INFO:teuthology.provision.fog.smithi109:Node is ready 2020-06-15T13:55:10.533 INFO:teuthology.orchestra.run.smithi109:> hostname 2020-06-15T13:55:10.658 INFO:teuthology.orchestra.run.smithi109.stdout:smithi109.front.sepia.ceph.com 2020-06-15T13:55:10.658 INFO:teuthology.orchestra.run.smithi109:> grep smithi109.front.sepia.ceph.com /etc/hosts 2020-06-15T13:55:10.720 INFO:teuthology.orchestra.run.smithi109.stdout:172.21.15.109 smithi109.front.sepia.ceph.com smithi109 2020-06-15T13:55:10.721 INFO:teuthology.orchestra.run.smithi109:> sudo hostname smithi109 2020-06-15T13:55:10.850 INFO:teuthology.orchestra.run.smithi109:> sudo sed -i -e 's/smithi109.front.sepia.ceph.com/smithi109/g' /etc/hosts 2020-06-15T13:55:10.960 INFO:teuthology.orchestra.run.smithi109:> sudo sed -i -e 's/172.21.15.109/172.21.15.109/g' /etc/hosts 2020-06-15T13:55:11.045 INFO:teuthology.orchestra.run.smithi109:> sudo sed -i -e 's/smithi109.front.sepia.ceph.com/smithi109/g' /etc/hostname 2020-06-15T13:55:11.115 INFO:teuthology.orchestra.run.smithi109:> sudo hostname smithi109 2020-06-15T13:55:11.185 INFO:teuthology.provision.fog.smithi109:Deploy complete! 2020-06-15T13:55:11.187 DEBUG:teuthology.parallel:result is None 2020-06-15T13:55:19.540 INFO:teuthology.provision.fog.smithi096:Node is ready 2020-06-15T13:55:19.541 INFO:teuthology.orchestra.run.smithi096:> hostname 2020-06-15T13:55:19.667 INFO:teuthology.orchestra.run.smithi096.stdout:smithi125 2020-06-15T13:55:19.667 INFO:teuthology.orchestra.run.smithi096:> grep smithi125 /etc/hosts 2020-06-15T13:55:19.749 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.125 smithi125.front.sepia.ceph.com smithi125 2020-06-15T13:55:19.750 INFO:teuthology.orchestra.run.smithi096:> sudo hostname smithi096 2020-06-15T13:55:19.908 INFO:teuthology.orchestra.run.smithi096:> sudo sed -i -e 's/smithi125/smithi096/g' /etc/hosts 2020-06-15T13:55:19.999 INFO:teuthology.orchestra.run.smithi096:> sudo sed -i -e 's/172.21.15.125/172.21.15.96/g' /etc/hosts 2020-06-15T13:55:20.080 INFO:teuthology.orchestra.run.smithi096:> sudo sed -i -e 's/smithi125/smithi096/g' /etc/hostname 2020-06-15T13:55:20.143 INFO:teuthology.orchestra.run.smithi096:> sudo hostname smithi096 2020-06-15T13:55:20.201 INFO:teuthology.provision.fog.smithi096:Deploy complete! 2020-06-15T13:55:20.202 DEBUG:teuthology.parallel:result is None 2020-06-15T13:55:32.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:32.738 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-15T13:55:32.739 INFO:teuthology.orchestra.run.smithi145:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-15T13:55:33.303 INFO:teuthology.provision.fog.smithi145:Node is ready 2020-06-15T13:55:33.304 INFO:teuthology.orchestra.run.smithi145:> hostname 2020-06-15T13:55:33.388 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145 2020-06-15T13:55:33.389 INFO:teuthology.orchestra.run.smithi145:> grep smithi145 /etc/hosts 2020-06-15T13:55:33.466 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145 2020-06-15T13:55:33.467 INFO:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2020-06-15T13:55:33.623 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145/smithi145/g' /etc/hosts 2020-06-15T13:55:33.730 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/172.21.15.145/172.21.15.145/g' /etc/hosts 2020-06-15T13:55:33.840 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145/smithi145/g' /etc/hostname 2020-06-15T13:55:33.939 INFO:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2020-06-15T13:55:34.020 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2020-06-15T13:55:34.021 DEBUG:teuthology.parallel:result is None 2020-06-15T13:55:34.262 INFO:teuthology.lock.ops:Checking smithi096.front.sepia.ceph.com 2020-06-15T13:55:34.264 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-15T13:55:34.289 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com 2020-06-15T13:55:34.289 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-15T13:55:34.308 INFO:teuthology.lock.ops:Checking smithi109.front.sepia.ceph.com 2020-06-15T13:55:34.308 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-15T13:55:34.328 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: set os type and version on server 2020-06-15T13:55:34.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-15T13:55:34.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:34.430 INFO:teuthology.orchestra.run.smithi096:> true 2020-06-15T13:55:34.814 INFO:teuthology.orchestra.run.smithi096:> uname -m 2020-06-15T13:55:34.884 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2020-06-15T13:55:34.884 INFO:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2020-06-15T13:55:34.954 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Linux" 2020-06-15T13:55:34.955 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="7 (Core)" 2020-06-15T13:55:34.955 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2020-06-15T13:55:34.955 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2020-06-15T13:55:34.956 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="7" 2020-06-15T13:55:34.956 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-06-15T13:55:34.956 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2020-06-15T13:55:34.956 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-06-15T13:55:34.956 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.centos.org/" 2020-06-15T13:55:34.957 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-15T13:55:34.957 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:34.957 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-06-15T13:55:34.957 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-06-15T13:55:34.958 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-15T13:55:34.958 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-06-15T13:55:34.958 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:34.960 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2020-06-15T13:55:34.982 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server 2020-06-15T13:55:34.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-15T13:55:34.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:35.093 INFO:teuthology.orchestra.run.smithi145:> true 2020-06-15T13:55:35.476 INFO:teuthology.orchestra.run.smithi145:> uname -m 2020-06-15T13:55:35.554 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2020-06-15T13:55:35.554 INFO:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2020-06-15T13:55:35.625 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Linux" 2020-06-15T13:55:35.625 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="7 (Core)" 2020-06-15T13:55:35.626 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2020-06-15T13:55:35.626 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2020-06-15T13:55:35.626 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="7" 2020-06-15T13:55:35.626 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-06-15T13:55:35.626 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2020-06-15T13:55:35.627 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-06-15T13:55:35.627 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.centos.org/" 2020-06-15T13:55:35.627 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-15T13:55:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:55:35.627 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-06-15T13:55:35.628 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-06-15T13:55:35.628 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-15T13:55:35.628 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-06-15T13:55:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:55:35.629 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2020-06-15T13:55:35.653 INFO:teuthology.lock.ops:Updating [smithi109.front.sepia.ceph.com]: set os type and version on server 2020-06-15T13:55:35.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-15T13:55:35.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:35.795 INFO:teuthology.orchestra.run.smithi109:> true 2020-06-15T13:55:36.190 INFO:teuthology.orchestra.run.smithi109:> uname -m 2020-06-15T13:55:36.271 INFO:teuthology.orchestra.run.smithi109.stdout:x86_64 2020-06-15T13:55:36.271 INFO:teuthology.orchestra.run.smithi109:> cat /etc/os-release 2020-06-15T13:55:36.345 INFO:teuthology.orchestra.run.smithi109.stdout:NAME="CentOS Linux" 2020-06-15T13:55:36.346 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION="7 (Core)" 2020-06-15T13:55:36.346 INFO:teuthology.orchestra.run.smithi109.stdout:ID="centos" 2020-06-15T13:55:36.346 INFO:teuthology.orchestra.run.smithi109.stdout:ID_LIKE="rhel fedora" 2020-06-15T13:55:36.346 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION_ID="7" 2020-06-15T13:55:36.347 INFO:teuthology.orchestra.run.smithi109.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-06-15T13:55:36.347 INFO:teuthology.orchestra.run.smithi109.stdout:ANSI_COLOR="0;31" 2020-06-15T13:55:36.347 INFO:teuthology.orchestra.run.smithi109.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-06-15T13:55:36.347 INFO:teuthology.orchestra.run.smithi109.stdout:HOME_URL="https://www.centos.org/" 2020-06-15T13:55:36.347 INFO:teuthology.orchestra.run.smithi109.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-15T13:55:36.348 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:55:36.348 INFO:teuthology.orchestra.run.smithi109.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-06-15T13:55:36.348 INFO:teuthology.orchestra.run.smithi109.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-06-15T13:55:36.348 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-15T13:55:36.348 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-06-15T13:55:36.349 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:55:36.349 INFO:teuthology.lock.ops:Updating smithi109.front.sepia.ceph.com on lock server 2020-06-15T13:55:36.375 INFO:teuthology.task.internal.lock_machines:3 smithi machines locked this try, 3/3 locked so far 2020-06-15T13:55:36.424 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDL2ra3dNbR7P+hUODFgay9FpdpVbhomDsFclOpEAOjxW5i6E7SSR8OQEmSrDaSWPo1TPsDfC5TsUpoGNRwILc/9JptfdqQcea6pOEmImMbVWyEKQYWaDsIBKTOkhFeqyJQCuIS0ylpExrbq1SVgnlZZF48mR+POyjgoC/pWc2aRjohw/6zzlkCY0YeuutHRIigj80xh+Y6SBOTpeyAiANo8O4hEQef3r2FJkgCdsE5b+r2RERh4GJwc/SIl4sMJZ1n2EIC8OtwURwResjS6T3b2UCML+czHJj03Xrsc82/hv4PfkySLhq0JpZxfDZaPEFUjX7gCqDQ1OsWPgGCF/qx smithi109.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQClVa5CicFsEU0LHTvC0zfy8VRww4qHtK/pFoOJe/IcI7rsSGeK4UvyzBmHdTmqpoN7l1Muw46v/heUpwI9fbSK9BbbD3yNLaWu4Qa+6CZLdPdtK1Y3lPUq0hIxsevljxeNF8s27eXbWzHOnrb6mxZ6zwLeYS98HR+qPdzj6Z2ltcPZ7iGxJnEjAv5mvylqkccMmo3/1jKfhFySFLLlfoXXw13eRvt/ffAfLQ7wC8rNoFA+dTO9bAyqj6A0SSyvvCVUpLmMkJd/MudPHdrJGJRhQMznmFwdU0f37ECbu7/BYB65rHZRco69ydTRuD9K8MEtRRUBXvcMcoyb/a/afvN9 smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEn128vjLe7toQ0Cauc/+EFkl2gshEL1cRWPdCa5x6mNSttLQIdQs6gKUbKWg/tJlbHye70pLmdEj+6NRKnxUaafp29Fa0Ztm92bQVDbU3hQGPwUMNwzmqOVRUjYEgI1j4JTNzuibqtXLjhxHPdK5curDAw6DsTVUxikQcQDpMhBIoiy++QCXOkB+czWWar8SQIVR8mhSthU4fTGdMdMclFcw3M+b0XXaaLdrvpHx+fziQx1UCnH0XEXCVMcQ7dppRro+D99TCryihcpzCsSsU2gpB9YyFxeoE21MrnL0fpmw2P3AsLEb1c9gHNzwYJiV/KAT+wt1ptCKliR420RcP 2020-06-15T13:55:36.424 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-15T13:55:36.482 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-06-15T13:55:36.491 INFO:teuthology.task.internal:Saving configuration 2020-06-15T13:55:36.523 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-06-15T13:55:36.535 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-06-15T13:55:36.551 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-15 13:50:42.002719', 'description': '/home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150372', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsB9IdyRNvQzYTQ+cVVzBuDDKUVtSA9ULnYFOXwfdBA4YhXAdMxATObjWomTMQLQ+LNZnrACJB+HoNGUeQgUmA=', 'os_version': '7', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi096.front.sepia.ceph.com'} 2020-06-15T13:55:36.567 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-15 13:50:42.002739', 'description': '/home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150372', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBz3ch+vtonDnSq8ve4lgaOTuMmLW18I2sTmVo/HLlblYhddRFCqvfqreHZXIJIux8UhLvBEnfncwdzhBMMn0Gs=', 'os_version': '7', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi145.front.sepia.ceph.com'} 2020-06-15T13:55:36.582 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-15 13:50:42.002754', 'description': '/home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150372', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCe/ifbjxnonUCQbLSQcfFpCIuVjAXn7FeID0Ca+Iw+nUV34A52zikFksMzP7vZopAinPsYDk/pvhCeGxeSS+l8=', 'os_version': '7', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi109.front.sepia.ceph.com'} 2020-06-15T13:55:36.583 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-06-15T13:55:36.593 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-06-15T13:55:36.593 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2020-06-15T13:55:36.594 INFO:teuthology.task.internal:roles: ubuntu@smithi109.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2020-06-15T13:55:36.594 INFO:teuthology.run_tasks:Running task console_log... 2020-06-15T13:55:36.773 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe2bafe3bf8>, signals=[15]) 2020-06-15T13:55:36.774 INFO:teuthology.run_tasks:Running task internal.connect... 2020-06-15T13:55:36.785 INFO:teuthology.task.internal:Opening connections... 2020-06-15T13:55:36.786 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2020-06-15T13:55:36.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:36.912 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2020-06-15T13:55:36.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:37.041 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi109.front.sepia.ceph.com 2020-06-15T13:55:37.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-15T13:55:37.151 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-06-15T13:55:37.168 INFO:teuthology.orchestra.run.smithi096:> uname -m 2020-06-15T13:55:37.297 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2020-06-15T13:55:37.298 INFO:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2020-06-15T13:55:37.366 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Linux" 2020-06-15T13:55:37.367 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="7 (Core)" 2020-06-15T13:55:37.367 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2020-06-15T13:55:37.367 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2020-06-15T13:55:37.367 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="7" 2020-06-15T13:55:37.368 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-06-15T13:55:37.368 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2020-06-15T13:55:37.368 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-06-15T13:55:37.368 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.centos.org/" 2020-06-15T13:55:37.368 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-15T13:55:37.369 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-06-15T13:55:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-06-15T13:55:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-15T13:55:37.369 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-06-15T13:55:37.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:37.371 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2020-06-15T13:55:37.399 INFO:teuthology.orchestra.run.smithi145:> uname -m 2020-06-15T13:55:37.514 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2020-06-15T13:55:37.515 INFO:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2020-06-15T13:55:37.585 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Linux" 2020-06-15T13:55:37.585 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="7 (Core)" 2020-06-15T13:55:37.586 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2020-06-15T13:55:37.586 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2020-06-15T13:55:37.586 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="7" 2020-06-15T13:55:37.586 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-06-15T13:55:37.586 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2020-06-15T13:55:37.587 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-06-15T13:55:37.587 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.centos.org/" 2020-06-15T13:55:37.587 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-15T13:55:37.587 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:55:37.587 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-06-15T13:55:37.588 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-06-15T13:55:37.588 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-15T13:55:37.588 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-06-15T13:55:37.588 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:55:37.589 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2020-06-15T13:55:37.625 INFO:teuthology.orchestra.run.smithi109:> uname -m 2020-06-15T13:55:37.748 INFO:teuthology.orchestra.run.smithi109.stdout:x86_64 2020-06-15T13:55:37.749 INFO:teuthology.orchestra.run.smithi109:> cat /etc/os-release 2020-06-15T13:55:37.823 INFO:teuthology.orchestra.run.smithi109.stdout:NAME="CentOS Linux" 2020-06-15T13:55:37.823 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION="7 (Core)" 2020-06-15T13:55:37.824 INFO:teuthology.orchestra.run.smithi109.stdout:ID="centos" 2020-06-15T13:55:37.824 INFO:teuthology.orchestra.run.smithi109.stdout:ID_LIKE="rhel fedora" 2020-06-15T13:55:37.824 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION_ID="7" 2020-06-15T13:55:37.824 INFO:teuthology.orchestra.run.smithi109.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-06-15T13:55:37.824 INFO:teuthology.orchestra.run.smithi109.stdout:ANSI_COLOR="0;31" 2020-06-15T13:55:37.825 INFO:teuthology.orchestra.run.smithi109.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-06-15T13:55:37.825 INFO:teuthology.orchestra.run.smithi109.stdout:HOME_URL="https://www.centos.org/" 2020-06-15T13:55:37.825 INFO:teuthology.orchestra.run.smithi109.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-06-15T13:55:37.825 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:55:37.825 INFO:teuthology.orchestra.run.smithi109.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-06-15T13:55:37.826 INFO:teuthology.orchestra.run.smithi109.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-06-15T13:55:37.826 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-06-15T13:55:37.826 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-06-15T13:55:37.826 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:55:37.827 INFO:teuthology.lock.ops:Updating smithi109.front.sepia.ceph.com on lock server 2020-06-15T13:55:37.861 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-06-15T13:55:37.877 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-06-15T13:55:37.900 INFO:teuthology.task.internal:Checking for old test directory... 2020-06-15T13:55:37.900 INFO:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2020-06-15T13:55:37.905 INFO:teuthology.orchestra.run.smithi109:> test '!' -e /home/ubuntu/cephtest 2020-06-15T13:55:37.910 INFO:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2020-06-15T13:55:37.997 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-06-15T13:55:38.008 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-06-15T13:55:38.008 INFO:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2020-06-15T13:55:38.013 INFO:teuthology.orchestra.run.smithi109:> test -z $(ls -A /var/lib/ceph) 2020-06-15T13:55:38.019 INFO:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2020-06-15T13:55:38.127 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-06-15T13:55:38.238 INFO:teuthology.run_tasks:Running task kernel... 2020-06-15T13:55:38.257 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'osd.8': {'kdb': True, 'sha1': 'distro'}, 'osd.9': {'kdb': True, 'sha1': 'distro'}, 'osd.10': {'kdb': True, 'sha1': 'distro'}, 'osd.11': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}} 2020-06-15T13:55:38.259 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-06-15T13:55:38.260 INFO:teuthology.orchestra.run.smithi096:> uname -r 2020-06-15T13:55:38.349 INFO:teuthology.orchestra.run.smithi096.stdout:3.10.0-1062.12.1.el7.x86_64 2020-06-15T13:55:38.350 INFO:teuthology.task.kernel:Running kernel on smithi096: 3.10.0-1062.12.1.el7.x86_64 2020-06-15T13:55:38.350 INFO:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2020-06-15T13:55:39.232 INFO:teuthology.orchestra.run.smithi096.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-06-15T13:55:39.331 INFO:teuthology.orchestra.run.smithi096.stdout:Determining fastest mirrors 2020-06-15T13:55:42.915 INFO:teuthology.orchestra.run.smithi096.stdout: * base: mirror.siena.edu 2020-06-15T13:55:42.916 INFO:teuthology.orchestra.run.smithi096.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-06-15T13:55:42.917 INFO:teuthology.orchestra.run.smithi096.stdout: * extras: mirror.facebook.net 2020-06-15T13:55:42.918 INFO:teuthology.orchestra.run.smithi096.stdout: * updates: mirror.umd.edu 2020-06-15T13:55:46.939 INFO:teuthology.orchestra.run.smithi096.stdout:Resolving Dependencies 2020-06-15T13:55:46.940 INFO:teuthology.orchestra.run.smithi096.stdout:--> Running transaction check 2020-06-15T13:55:46.940 INFO:teuthology.orchestra.run.smithi096.stdout:---> Package kernel.x86_64 0:3.10.0-1127.10.1.el7 will be installed 2020-06-15T13:55:48.193 INFO:teuthology.orchestra.run.smithi096.stdout:--> Finished Dependency Resolution 2020-06-15T13:55:48.942 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:48.943 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies Resolved 2020-06-15T13:55:48.944 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:48.944 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2020-06-15T13:55:48.944 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2020-06-15T13:55:48.945 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2020-06-15T13:55:48.945 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2020-06-15T13:55:48.945 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 3.10.0-1127.10.1.el7 updates 50 M 2020-06-15T13:55:48.945 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:48.946 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2020-06-15T13:55:48.946 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2020-06-15T13:55:48.946 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2020-06-15T13:55:48.946 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:55:48.947 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 50 M 2020-06-15T13:55:48.947 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 64 M 2020-06-15T13:55:48.947 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading packages: 2020-06-15T13:55:48.954 INFO:teuthology.orchestra.run.smithi096.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-06-15T13:55:50.896 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2020-06-15T13:55:51.069 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2020-06-15T13:55:51.911 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded 2020-06-15T13:55:51.911 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2020-06-15T13:56:30.628 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-3.10.0-1127.10.1.el7.x86_64 1/1 2020-06-15T13:56:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-3.10.0-1127.10.1.el7.x86_64 1/1 2020-06-15T13:56:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:56:31.208 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2020-06-15T13:56:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: kernel.x86_64 0:3.10.0-1127.10.1.el7 2020-06-15T13:56:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: 2020-06-15T13:56:31.209 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2020-06-15T13:56:31.264 INFO:teuthology.orchestra.run.smithi096:> rpm -q kernel --last | head -n 1 2020-06-15T13:56:31.659 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-3.10.0-1127.10.1.el7.x86_64 Mon 15 Jun 2020 01:55:54 PM UTC 2020-06-15T13:56:31.660 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1127.10.1.el7.x86_64 2020-06-15T13:56:31.661 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.12.1.el7.x86_64 Expected: 3.10.0-1127.10.1.el7.x86_64 2020-06-15T13:56:31.661 INFO:teuthology.orchestra.run.smithi145:> uname -r 2020-06-15T13:56:31.770 INFO:teuthology.orchestra.run.smithi145.stdout:3.10.0-1062.12.1.el7.x86_64 2020-06-15T13:56:31.770 INFO:teuthology.task.kernel:Running kernel on smithi145: 3.10.0-1062.12.1.el7.x86_64 2020-06-15T13:56:31.771 INFO:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2020-06-15T13:56:32.682 INFO:teuthology.orchestra.run.smithi145.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-06-15T13:56:32.792 INFO:teuthology.orchestra.run.smithi145.stdout:Determining fastest mirrors 2020-06-15T13:56:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: * base: mirror.siena.edu 2020-06-15T13:56:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-06-15T13:56:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: * extras: mirror.facebook.net 2020-06-15T13:56:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: * updates: mirror.umd.edu 2020-06-15T13:56:43.692 INFO:teuthology.orchestra.run.smithi145.stdout:Resolving Dependencies 2020-06-15T13:56:43.693 INFO:teuthology.orchestra.run.smithi145.stdout:--> Running transaction check 2020-06-15T13:56:43.694 INFO:teuthology.orchestra.run.smithi145.stdout:---> Package kernel.x86_64 0:3.10.0-1127.10.1.el7 will be installed 2020-06-15T13:56:45.061 INFO:teuthology.orchestra.run.smithi145.stdout:--> Finished Dependency Resolution 2020-06-15T13:56:45.882 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:56:45.882 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies Resolved 2020-06-15T13:56:45.901 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:56:45.902 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-06-15T13:56:45.902 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2020-06-15T13:56:45.902 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-06-15T13:56:45.903 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2020-06-15T13:56:45.903 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 3.10.0-1127.10.1.el7 updates 50 M 2020-06-15T13:56:45.903 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:56:45.903 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2020-06-15T13:56:45.903 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2020-06-15T13:56:45.904 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2020-06-15T13:56:45.904 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:56:45.904 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 50 M 2020-06-15T13:56:45.904 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 64 M 2020-06-15T13:56:45.905 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading packages: 2020-06-15T13:56:45.912 INFO:teuthology.orchestra.run.smithi145.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-06-15T13:56:47.772 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2020-06-15T13:56:47.965 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2020-06-15T13:56:48.952 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded 2020-06-15T13:56:48.952 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2020-06-15T13:57:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-3.10.0-1127.10.1.el7.x86_64 1/1 2020-06-15T13:57:27.167 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-3.10.0-1127.10.1.el7.x86_64 1/1 2020-06-15T13:57:27.167 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:57:27.168 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2020-06-15T13:57:27.168 INFO:teuthology.orchestra.run.smithi145.stdout: kernel.x86_64 0:3.10.0-1127.10.1.el7 2020-06-15T13:57:27.170 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-06-15T13:57:27.170 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2020-06-15T13:57:27.215 INFO:teuthology.orchestra.run.smithi145:> rpm -q kernel --last | head -n 1 2020-06-15T13:57:27.585 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-3.10.0-1127.10.1.el7.x86_64 Mon 15 Jun 2020 01:56:51 PM UTC 2020-06-15T13:57:27.598 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1127.10.1.el7.x86_64 2020-06-15T13:57:27.599 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.12.1.el7.x86_64 Expected: 3.10.0-1127.10.1.el7.x86_64 2020-06-15T13:57:27.600 INFO:teuthology.orchestra.run.smithi109:> uname -r 2020-06-15T13:57:27.727 INFO:teuthology.orchestra.run.smithi109.stdout:3.10.0-1062.12.1.el7.x86_64 2020-06-15T13:57:27.728 INFO:teuthology.task.kernel:Running kernel on smithi109: 3.10.0-1062.12.1.el7.x86_64 2020-06-15T13:57:27.728 INFO:teuthology.orchestra.run.smithi109:> sudo yum install -y kernel 2020-06-15T13:57:28.639 INFO:teuthology.orchestra.run.smithi109.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-06-15T13:57:28.740 INFO:teuthology.orchestra.run.smithi109.stdout:Determining fastest mirrors 2020-06-15T13:57:35.193 INFO:teuthology.orchestra.run.smithi109.stdout: * base: mirror.siena.edu 2020-06-15T13:57:35.200 INFO:teuthology.orchestra.run.smithi109.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-06-15T13:57:35.201 INFO:teuthology.orchestra.run.smithi109.stdout: * extras: mirror.facebook.net 2020-06-15T13:57:35.202 INFO:teuthology.orchestra.run.smithi109.stdout: * updates: mirror.umd.edu 2020-06-15T13:57:39.711 INFO:teuthology.orchestra.run.smithi109.stdout:Resolving Dependencies 2020-06-15T13:57:39.712 INFO:teuthology.orchestra.run.smithi109.stdout:--> Running transaction check 2020-06-15T13:57:39.712 INFO:teuthology.orchestra.run.smithi109.stdout:---> Package kernel.x86_64 0:3.10.0-1127.10.1.el7 will be installed 2020-06-15T13:57:41.021 INFO:teuthology.orchestra.run.smithi109.stdout:--> Finished Dependency Resolution 2020-06-15T13:57:41.751 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:57:41.751 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies Resolved 2020-06-15T13:57:41.759 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:57:41.760 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2020-06-15T13:57:41.760 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2020-06-15T13:57:41.760 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2020-06-15T13:57:41.760 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2020-06-15T13:57:41.761 INFO:teuthology.orchestra.run.smithi109.stdout: kernel x86_64 3.10.0-1127.10.1.el7 updates 50 M 2020-06-15T13:57:41.761 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:57:41.761 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2020-06-15T13:57:41.761 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2020-06-15T13:57:41.761 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2020-06-15T13:57:41.761 INFO:teuthology.orchestra.run.smithi109.stdout: 2020-06-15T13:57:41.762 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 50 M 2020-06-15T13:57:41.762 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 64 M 2020-06-15T13:57:41.762 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading packages: 2020-06-15T13:57:41.763 INFO:teuthology.orchestra.run.smithi109.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-06-15T13:57:43.124 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-06-15T13:57:43.127 DEBUG:teuthology.task.console_log:Killing console logger for smithi096 2020-06-15T13:57:43.128 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2020-06-15T13:57:43.129 DEBUG:teuthology.task.console_log:Killing console logger for smithi109 2020-06-15T13:57:43.129 DEBUG:teuthology.task.console_log:Killing console logger for smithi096 2020-06-15T13:57:43.129 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2020-06-15T13:57:43.130 DEBUG:teuthology.task.console_log:Killing console logger for smithi109 2020-06-15T13:57:43.130 DEBUG:teuthology.exit:Finished running handlers