2021-08-05T17:01:12.459 INFO:root:teuthology version: 1.1.0-04c2febe 2021-08-05T17:01:12.579 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi006.front.sepia.ceph.com' reimaging 2021-08-05T17:01:12.581 INFO:teuthology.lock.ops:Updating [ubuntu@smithi006.front.sepia.ceph.com]: reset os type and version on server 2021-08-05T17:01:12.581 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2021-08-05T17:01:12.631 INFO:teuthology.lock.ops:Node 'ubuntu@smithi006.front.sepia.ceph.com' reimaging is complete 2021-08-05T17:01:12.632 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging 2021-08-05T17:01:12.632 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server 2021-08-05T17:01:12.633 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-08-05T17:01:12.702 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete 2021-08-05T17:01:12.843 INFO:teuthology.provision.fog.smithi006:Scheduling deploy of centos 8.3 2021-08-05T17:01:12.911 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of centos 8.3 2021-08-05T17:01:13.184 INFO:teuthology.orchestra.console:Power off smithi006 2021-08-05T17:01:13.245 INFO:teuthology.orchestra.console:Power off smithi187 2021-08-05T17:01:25.672 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2021-08-05T17:01:25.718 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-08-05T17:01:25.774 INFO:teuthology.orchestra.console:Power on smithi006 2021-08-05T17:01:25.820 INFO:teuthology.orchestra.console:Power on smithi187 2021-08-05T17:01:30.044 INFO:teuthology.orchestra.console:Power on for smithi006 completed 2021-08-05T17:01:30.146 INFO:teuthology.provision.fog.smithi006:Waiting for deploy to finish 2021-08-05T17:01:34.184 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2021-08-05T17:01:34.285 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2021-08-05T17:03:46.362 ERROR:teuthology.orchestra.connection:Error authenticating with smithi006.front.sepia.ceph.com: Authentication failed. 2021-08-05T17:04:05.562 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed. 2021-08-05T17:05:20.810 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-05T17:05:31.276 INFO:teuthology.provision.fog.smithi006:Node is ready 2021-08-05T17:05:31.300 INFO:teuthology.orchestra.run.smithi006.stdout:smithi012 2021-08-05T17:05:31.363 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.12 smithi012.front.sepia.ceph.com smithi012 2021-08-05T17:05:31.744 INFO:teuthology.provision.fog.smithi006:Deploy complete! 2021-08-05T17:05:44.400 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-05T17:06:04.765 INFO:teuthology.provision.fog.smithi187:Node is ready 2021-08-05T17:06:04.786 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2021-08-05T17:06:04.844 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2021-08-05T17:06:05.190 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2021-08-05T17:06:05.336 INFO:teuthology.lock.ops:Checking smithi006.front.sepia.ceph.com 2021-08-05T17:06:05.338 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-05T17:06:05.358 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com 2021-08-05T17:06:05.358 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-05T17:06:05.372 INFO:teuthology.lock.ops:Updating [smithi006.front.sepia.ceph.com]: set os type and version on server 2021-08-05T17:06:05.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-05T17:06:05.673 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2021-08-05T17:06:05.728 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Linux" 2021-08-05T17:06:05.728 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="8" 2021-08-05T17:06:05.729 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2021-08-05T17:06:05.729 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2021-08-05T17:06:05.729 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="8" 2021-08-05T17:06:05.729 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el8" 2021-08-05T17:06:05.730 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Linux 8" 2021-08-05T17:06:05.731 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2021-08-05T17:06:05.731 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-05T17:06:05.731 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2021-08-05T17:06:05.732 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-05T17:06:05.732 INFO:teuthology.orchestra.run.smithi006.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-05T17:06:05.732 INFO:teuthology.orchestra.run.smithi006.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-05T17:06:05.737 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2021-08-05T17:06:05.754 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server 2021-08-05T17:06:05.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-05T17:06:06.075 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2021-08-05T17:06:06.129 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Linux" 2021-08-05T17:06:06.131 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8" 2021-08-05T17:06:06.131 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2021-08-05T17:06:06.131 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2021-08-05T17:06:06.131 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2021-08-05T17:06:06.132 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2021-08-05T17:06:06.132 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Linux 8" 2021-08-05T17:06:06.132 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2021-08-05T17:06:06.132 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-05T17:06:06.132 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2021-08-05T17:06:06.133 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-05T17:06:06.133 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-05T17:06:06.133 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-05T17:06:06.134 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-08-05T17:06:06.229 INFO:teuthology.dispatcher.supervisor:Running job 6313327 2021-08-05T17:06:06.230 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/virtualenv/bin/teuthology -v --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2021-08-05_16:59:53-rgw-wip-cbodley-testing-distro-basic-smithi/6313327 --name cbodley-2021-08-05_16:59:53-rgw-wip-cbodley-testing-distro-basic-smithi --description rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} -- /home/teuthworker/archive/cbodley-2021-08-05_16:59:53-rgw-wip-cbodley-testing-distro-basic-smithi/6313327/orig.config.yaml 2021-08-05T17:06:06.237 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/cbodley-2021-08-05_16:59:53-rgw-wip-cbodley-testing-distro-basic-smithi/6313327 2021-08-05T17:06:06.239 INFO:teuthology.dispatcher.supervisor:Job PID: 27245 2021-08-05T17:06:06.239 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-06T05:06:24.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-06T05:06:24.316 INFO:teuthology.kill:Killing Pids: {27245} 2021-08-06T05:06:25.572 INFO:teuthology.kill:Nuking machines: ['smithi006', 'smithi187'] 2021-08-06T05:06:26.604 INFO:teuthology.kill:2021-08-06 05:06:26,603.603 INFO:teuthology.nuke:targets: 2021-08-06T05:06:26.605 INFO:teuthology.kill: smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbhc3yvqDSWOTH09ein70fQdOVRoqeKGJk7BXg1HT/pZEejmQqWYpU9f9S6R4f3vQRSq6KKuJ4pK1p2zRTDCLCXTaByfG2+Eq4ef9ZNGPNt8XatCPnSt0qKz9OnLlcMJaFxhgr1VzSI4w2XiWxPlyxcSVUavff0xIdo3eHwoI+12ybkGJFDw7Q8otj2ItZlwbXTjQ6/EpEoLgb2QZCQklwAVKvD2FoBjlNWNAcsAqV4oBKN/9TCNiO7eANfaoIysRG7IUFLVYyqcikgejDm0Vms7/TWYporvYOoguZQLVlWk9KGWuqnIhHtKaCL01B88PtUKh1BCZx4x6PQdpBGASMrTqleFfzkAYvGvv2aBpAEB5TFDMohQskfPBMgb7oqKBvOvh95vms6LW2Ts511GIMtjVqQImbX8T2p3rF2U/XayA8wnjx/C1j8brpkeFn20IrGIHw5BzqWFNa8PtmwAX44HGBVU2GG7YKljID0iqfze72E7ZeBgksPqCblCFvqG0= 2021-08-06T05:06:26.605 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ6hZ7mzLINHAYWtQWxeJJq7hyP729DsB1/MvF52j/mVqojpD3JHVTaNDSgT337nQejtCoEts2QlSjilImVeeiTjWFflv18RP/h4wyOTyNkncq3fVnvTa89lXI8UPkKJx6PDWnXmF0wn/SU05NO5I8dUxwdROjxmgBR5x2rMiSWETNGQ+QeCj5izGAdgaxTDUo7KLPUKfxxxEl19uavlgw/GebA04Bua/uSRTFI8R5Nl6hqJUeY5jWUJTHIntrGZGT+xg2GBo74kuxAYiPtoGI6FethjM/xIgTspUBAzpjlZ+meveTzzn8b/BrD7QLQ8X3tkYF4en3psjjr7a+zTgUihpc7XeTEUTr6Qo4AEwSr+uRW4IhZIiAQALF+lvRNYoKbjtEoUv4fNfR6BSBZFBP4XoYo+rIaeJNdcr+wctCjA5CiHX5fRP+Mbwxy8XfBr7v94zqVtdr1LvRWS6d7HZSxTAZX8u4HQBlFY8hhsxDxyMaG49JGsFnh/9Z1t5GEs8= 2021-08-06T05:06:26.605 INFO:teuthology.kill:2021-08-06 05:06:26,604.604 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-06T05:06:26.607 INFO:teuthology.kill:2021-08-06 05:06:26,606.606 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-06T05:06:26.667 INFO:teuthology.kill:2021-08-06 05:06:26,667.667 INFO:teuthology.orchestra.console:Power off smithi187 2021-08-06T05:06:26.684 INFO:teuthology.kill:2021-08-06 05:06:26,684.684 INFO:teuthology.orchestra.console:Power off smithi006 2021-08-06T05:06:35.047 INFO:teuthology.kill:2021-08-06 05:06:35,046.046 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-08-06T05:06:35.053 INFO:teuthology.kill:2021-08-06 05:06:35,053.053 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2021-08-06T05:06:35.311 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-08-06T05:06:35.312 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-08-06T05:06:35.313 INFO:teuthology.misc:Compressing logs... 2021-08-06T05:06:35.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-06T05:07:09.899 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-06T05:07:09.953 INFO:teuthology.kill:No teuthology processes running 2021-08-06T05:07:11.108 INFO:teuthology.kill:Nuking machines: ['smithi006', 'smithi187'] 2021-08-06T05:07:12.104 INFO:teuthology.kill:2021-08-06 05:07:12,103.103 INFO:teuthology.nuke:targets: 2021-08-06T05:07:12.105 INFO:teuthology.kill: smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbhc3yvqDSWOTH09ein70fQdOVRoqeKGJk7BXg1HT/pZEejmQqWYpU9f9S6R4f3vQRSq6KKuJ4pK1p2zRTDCLCXTaByfG2+Eq4ef9ZNGPNt8XatCPnSt0qKz9OnLlcMJaFxhgr1VzSI4w2XiWxPlyxcSVUavff0xIdo3eHwoI+12ybkGJFDw7Q8otj2ItZlwbXTjQ6/EpEoLgb2QZCQklwAVKvD2FoBjlNWNAcsAqV4oBKN/9TCNiO7eANfaoIysRG7IUFLVYyqcikgejDm0Vms7/TWYporvYOoguZQLVlWk9KGWuqnIhHtKaCL01B88PtUKh1BCZx4x6PQdpBGASMrTqleFfzkAYvGvv2aBpAEB5TFDMohQskfPBMgb7oqKBvOvh95vms6LW2Ts511GIMtjVqQImbX8T2p3rF2U/XayA8wnjx/C1j8brpkeFn20IrGIHw5BzqWFNa8PtmwAX44HGBVU2GG7YKljID0iqfze72E7ZeBgksPqCblCFvqG0= 2021-08-06T05:07:12.105 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ6hZ7mzLINHAYWtQWxeJJq7hyP729DsB1/MvF52j/mVqojpD3JHVTaNDSgT337nQejtCoEts2QlSjilImVeeiTjWFflv18RP/h4wyOTyNkncq3fVnvTa89lXI8UPkKJx6PDWnXmF0wn/SU05NO5I8dUxwdROjxmgBR5x2rMiSWETNGQ+QeCj5izGAdgaxTDUo7KLPUKfxxxEl19uavlgw/GebA04Bua/uSRTFI8R5Nl6hqJUeY5jWUJTHIntrGZGT+xg2GBo74kuxAYiPtoGI6FethjM/xIgTspUBAzpjlZ+meveTzzn8b/BrD7QLQ8X3tkYF4en3psjjr7a+zTgUihpc7XeTEUTr6Qo4AEwSr+uRW4IhZIiAQALF+lvRNYoKbjtEoUv4fNfR6BSBZFBP4XoYo+rIaeJNdcr+wctCjA5CiHX5fRP+Mbwxy8XfBr7v94zqVtdr1LvRWS6d7HZSxTAZX8u4HQBlFY8hhsxDxyMaG49JGsFnh/9Z1t5GEs8= 2021-08-06T05:07:12.125 INFO:teuthology.kill:2021-08-06 05:07:12,125.125 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-06T05:07:12.128 INFO:teuthology.kill:2021-08-06 05:07:12,128.128 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-06T05:07:12.186 INFO:teuthology.kill:2021-08-06 05:07:12,185.185 INFO:teuthology.orchestra.console:Power off smithi006 2021-08-06T05:07:12.199 INFO:teuthology.kill:2021-08-06 05:07:12,199.199 INFO:teuthology.orchestra.console:Power off smithi187 2021-08-06T05:07:12.328 INFO:teuthology.kill:2021-08-06 05:07:12,328.328 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2021-08-06T05:07:12.341 INFO:teuthology.kill:2021-08-06 05:07:12,341.341 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-08-06T05:07:12.472 INFO:teuthology.kill:2021-08-06 05:07:12,472.472 INFO:teuthology.lock.ops:unlocked smithi006.front.sepia.ceph.com 2021-08-06T05:07:12.481 INFO:teuthology.kill:2021-08-06 05:07:12,481.481 INFO:teuthology.lock.ops:unlocked smithi187.front.sepia.ceph.com 2021-08-06T05:09:12.689 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-06T05:09:12.775 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-06T05:09:12.776 INFO:teuthology.nuke:Checking targets against current locks 2021-08-06T05:09:12.862 INFO:teuthology.nuke:Not nuking smithi187.front.sepia.ceph.com because description doesn't match 2021-08-06T05:09:12.862 INFO:teuthology.nuke:Not nuking smithi006.front.sepia.ceph.com because description doesn't match