2021-06-04T11:56:49.178 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-04T11:56:49.330 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging 2021-06-04T11:56:49.331 INFO:teuthology.lock.ops:Updating [ubuntu@smithi041.front.sepia.ceph.com]: reset os type and version on server 2021-06-04T11:56:49.331 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-06-04T11:56:49.346 INFO:teuthology.lock.ops:Node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging is complete 2021-06-04T11:56:49.346 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging 2021-06-04T11:56:49.346 INFO:teuthology.lock.ops:Updating [ubuntu@smithi062.front.sepia.ceph.com]: reset os type and version on server 2021-06-04T11:56:49.347 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-06-04T11:56:49.361 INFO:teuthology.lock.ops:Node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging is complete 2021-06-04T11:56:49.362 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi076.front.sepia.ceph.com' reimaging 2021-06-04T11:56:49.362 INFO:teuthology.lock.ops:Updating [ubuntu@smithi076.front.sepia.ceph.com]: reset os type and version on server 2021-06-04T11:56:49.362 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2021-06-04T11:56:49.376 INFO:teuthology.lock.ops:Node 'ubuntu@smithi076.front.sepia.ceph.com' reimaging is complete 2021-06-04T11:56:49.376 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi144.front.sepia.ceph.com' reimaging 2021-06-04T11:56:49.376 INFO:teuthology.lock.ops:Updating [ubuntu@smithi144.front.sepia.ceph.com]: reset os type and version on server 2021-06-04T11:56:49.377 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2021-06-04T11:56:49.390 INFO:teuthology.lock.ops:Node 'ubuntu@smithi144.front.sepia.ceph.com' reimaging is complete 2021-06-04T11:56:49.554 INFO:teuthology.provision.fog.smithi041:Scheduling deploy of centos 8.3 2021-06-04T11:56:49.580 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of centos 8.3 2021-06-04T11:56:49.583 INFO:teuthology.provision.fog.smithi076:Scheduling deploy of centos 8.3 2021-06-04T11:56:49.609 INFO:teuthology.provision.fog.smithi144:Scheduling deploy of centos 8.3 2021-06-04T11:56:49.907 INFO:teuthology.orchestra.console:Power off smithi041 2021-06-04T11:56:49.948 INFO:teuthology.orchestra.console:Power off smithi076 2021-06-04T11:56:49.984 INFO:teuthology.orchestra.console:Power off smithi144 2021-06-04T11:56:50.001 INFO:teuthology.orchestra.console:Power off smithi062 2021-06-04T11:56:50.111 INFO:teuthology.orchestra.console:Power off for smithi076 completed 2021-06-04T11:56:50.213 INFO:teuthology.orchestra.console:Power on smithi076 2021-06-04T11:56:54.464 INFO:teuthology.orchestra.console:Power on for smithi076 completed 2021-06-04T11:56:54.565 INFO:teuthology.provision.fog.smithi076:Waiting for deploy to finish 2021-06-04T11:56:58.358 INFO:teuthology.orchestra.console:Power off for smithi144 completed 2021-06-04T11:56:58.368 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-06-04T11:56:58.461 INFO:teuthology.orchestra.console:Power on smithi144 2021-06-04T11:56:58.471 INFO:teuthology.orchestra.console:Power on smithi062 2021-06-04T11:57:02.399 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-06-04T11:57:02.501 INFO:teuthology.orchestra.console:Power on smithi041 2021-06-04T11:57:06.757 INFO:teuthology.orchestra.console:Power on for smithi041 completed 2021-06-04T11:57:06.830 INFO:teuthology.orchestra.console:Power on for smithi144 completed 2021-06-04T11:57:06.837 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2021-06-04T11:57:06.858 INFO:teuthology.provision.fog.smithi041:Waiting for deploy to finish 2021-06-04T11:57:06.931 INFO:teuthology.provision.fog.smithi144:Waiting for deploy to finish 2021-06-04T11:57:06.938 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2021-06-04T12:01:29.905 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-04T12:01:30.302 INFO:teuthology.provision.fog.smithi041:Node is ready 2021-06-04T12:01:30.319 INFO:teuthology.orchestra.run.smithi041.stdout:smithi041.front.sepia.ceph.com 2021-06-04T12:01:30.375 INFO:teuthology.orchestra.run.smithi041.stdout:172.21.15.41 smithi041.front.sepia.ceph.com smithi041 2021-06-04T12:01:30.694 INFO:teuthology.provision.fog.smithi041:Deploy complete! 2021-06-04T12:01:38.911 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-04T12:01:39.213 INFO:teuthology.provision.fog.smithi144:Node is ready 2021-06-04T12:01:39.227 INFO:teuthology.orchestra.run.smithi144.stdout:smithi144.front.sepia.ceph.com 2021-06-04T12:01:39.282 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144 smithi144.front.sepia.ceph.com smithi144 2021-06-04T12:01:39.602 INFO:teuthology.provision.fog.smithi144:Deploy complete! 2021-06-04T12:01:47.864 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-04T12:01:48.172 INFO:teuthology.provision.fog.smithi062:Node is ready 2021-06-04T12:01:48.187 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2021-06-04T12:01:48.247 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2021-06-04T12:01:48.576 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2021-06-04T12:01:59.528 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-04T12:01:59.868 INFO:teuthology.provision.fog.smithi076:Node is ready 2021-06-04T12:01:59.884 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076.front.sepia.ceph.com 2021-06-04T12:01:59.939 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76 smithi076.front.sepia.ceph.com smithi076 2021-06-04T12:02:00.255 INFO:teuthology.provision.fog.smithi076:Deploy complete! 2021-06-04T12:02:00.437 INFO:teuthology.lock.ops:Checking smithi041.front.sepia.ceph.com 2021-06-04T12:02:00.438 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-04T12:02:00.455 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2021-06-04T12:02:00.455 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-04T12:02:00.471 INFO:teuthology.lock.ops:Checking smithi076.front.sepia.ceph.com 2021-06-04T12:02:00.472 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-04T12:02:00.485 INFO:teuthology.lock.ops:Checking smithi144.front.sepia.ceph.com 2021-06-04T12:02:00.486 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-04T12:02:00.499 INFO:teuthology.lock.ops:Updating [smithi041.front.sepia.ceph.com]: set os type and version on server 2021-06-04T12:02:00.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T12:02:00.793 INFO:teuthology.orchestra.run.smithi041.stdout:x86_64 2021-06-04T12:02:00.848 INFO:teuthology.orchestra.run.smithi041.stdout:NAME="CentOS Linux" 2021-06-04T12:02:00.848 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION="8" 2021-06-04T12:02:00.848 INFO:teuthology.orchestra.run.smithi041.stdout:ID="centos" 2021-06-04T12:02:00.849 INFO:teuthology.orchestra.run.smithi041.stdout:ID_LIKE="rhel fedora" 2021-06-04T12:02:00.849 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION_ID="8" 2021-06-04T12:02:00.849 INFO:teuthology.orchestra.run.smithi041.stdout:PLATFORM_ID="platform:el8" 2021-06-04T12:02:00.849 INFO:teuthology.orchestra.run.smithi041.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-04T12:02:00.850 INFO:teuthology.orchestra.run.smithi041.stdout:ANSI_COLOR="0;31" 2021-06-04T12:02:00.850 INFO:teuthology.orchestra.run.smithi041.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-04T12:02:00.850 INFO:teuthology.orchestra.run.smithi041.stdout:HOME_URL="https://centos.org/" 2021-06-04T12:02:00.850 INFO:teuthology.orchestra.run.smithi041.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-04T12:02:00.851 INFO:teuthology.orchestra.run.smithi041.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-04T12:02:00.851 INFO:teuthology.orchestra.run.smithi041.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-04T12:02:00.852 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-06-04T12:02:00.868 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2021-06-04T12:02:00.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T12:02:01.089 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2021-06-04T12:02:01.146 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Linux" 2021-06-04T12:02:01.147 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8" 2021-06-04T12:02:01.147 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2021-06-04T12:02:01.147 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2021-06-04T12:02:01.147 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8" 2021-06-04T12:02:01.148 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2021-06-04T12:02:01.148 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-04T12:02:01.148 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2021-06-04T12:02:01.148 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-04T12:02:01.148 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2021-06-04T12:02:01.149 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-04T12:02:01.149 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-04T12:02:01.149 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-04T12:02:01.150 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-06-04T12:02:01.171 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: set os type and version on server 2021-06-04T12:02:01.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T12:02:01.463 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2021-06-04T12:02:01.517 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Linux" 2021-06-04T12:02:01.518 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="8" 2021-06-04T12:02:01.518 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2021-06-04T12:02:01.518 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2021-06-04T12:02:01.518 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="8" 2021-06-04T12:02:01.519 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el8" 2021-06-04T12:02:01.519 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-04T12:02:01.519 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2021-06-04T12:02:01.520 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-04T12:02:01.520 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2021-06-04T12:02:01.520 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-04T12:02:01.520 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-04T12:02:01.521 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-04T12:02:01.521 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2021-06-04T12:02:01.581 INFO:teuthology.lock.ops:Updating [smithi144.front.sepia.ceph.com]: set os type and version on server 2021-06-04T12:02:01.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T12:02:01.867 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2021-06-04T12:02:01.921 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Linux" 2021-06-04T12:02:01.921 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="8" 2021-06-04T12:02:01.921 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2021-06-04T12:02:01.922 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2021-06-04T12:02:01.922 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="8" 2021-06-04T12:02:01.922 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el8" 2021-06-04T12:02:01.922 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-04T12:02:01.923 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2021-06-04T12:02:01.923 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-04T12:02:01.923 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2021-06-04T12:02:01.923 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-04T12:02:01.923 INFO:teuthology.orchestra.run.smithi144.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-04T12:02:01.924 INFO:teuthology.orchestra.run.smithi144.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-04T12:02:01.924 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2021-06-04T12:02:02.027 INFO:teuthology.dispatcher.supervisor:Running job 6149433 2021-06-04T12:02:02.028 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-06-03_14:47:27-rados-final_gbenhano_ncb-distro-basic-smithi/6149433 --name benhanokh-2021-06-03_14:47:27-rados-final_gbenhano_ncb-distro-basic-smithi --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} -- /home/teuthworker/archive/benhanokh-2021-06-03_14:47:27-rados-final_gbenhano_ncb-distro-basic-smithi/6149433/orig.config.yaml 2021-06-04T12:02:02.032 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-06-03_14:47:27-rados-final_gbenhano_ncb-distro-basic-smithi/6149433 2021-06-04T12:02:02.032 INFO:teuthology.dispatcher.supervisor:Job PID: 3056 2021-06-04T12:02:02.033 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-04T18:16:28.960 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-04T18:16:29.151 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-04T18:16:29.152 INFO:teuthology.nuke:Checking targets against current locks 2021-06-04T18:16:29.223 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable