2021-10-15T15:27:06.517 INFO:root:teuthology version: 1.1.0-e4202151 2021-10-15T15:27:06.624 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging 2021-10-15T15:27:06.625 INFO:teuthology.lock.ops:Updating [ubuntu@gibba030.front.sepia.ceph.com]: reset os type and version on server 2021-10-15T15:27:06.625 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server 2021-10-15T15:27:06.643 INFO:teuthology.lock.ops:Node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging is complete 2021-10-15T15:27:06.643 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging 2021-10-15T15:27:06.643 INFO:teuthology.lock.ops:Updating [ubuntu@gibba042.front.sepia.ceph.com]: reset os type and version on server 2021-10-15T15:27:06.644 INFO:teuthology.lock.ops:Updating gibba042.front.sepia.ceph.com on lock server 2021-10-15T15:27:06.660 INFO:teuthology.lock.ops:Node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging is complete 2021-10-15T15:27:06.843 INFO:teuthology.provision.fog.gibba030:Scheduling deploy of centos 8.3 2021-10-15T15:27:06.858 INFO:teuthology.provision.fog.gibba042:Scheduling deploy of centos 8.3 2021-10-15T15:27:07.148 INFO:teuthology.orchestra.console:Power off gibba030 2021-10-15T15:27:07.161 INFO:teuthology.orchestra.console:Power off gibba042 2021-10-15T15:27:15.531 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-10-15T15:27:15.537 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-10-15T15:27:15.633 INFO:teuthology.orchestra.console:Power on gibba030 2021-10-15T15:27:15.640 INFO:teuthology.orchestra.console:Power on gibba042 2021-10-15T15:27:24.009 INFO:teuthology.orchestra.console:Power on for gibba030 completed 2021-10-15T15:27:24.015 INFO:teuthology.orchestra.console:Power on for gibba042 completed 2021-10-15T15:27:24.110 INFO:teuthology.provision.fog.gibba030:Waiting for deploy to finish 2021-10-15T15:27:24.116 INFO:teuthology.provision.fog.gibba042:Waiting for deploy to finish 2021-10-15T15:31:21.985 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-10-15T15:31:30.922 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-10-15T15:31:32.473 INFO:teuthology.provision.fog.gibba030:Node is ready 2021-10-15T15:31:32.486 INFO:teuthology.orchestra.run.gibba030.stdout:gibba032 2021-10-15T15:31:32.538 INFO:teuthology.orchestra.run.gibba030.stdout:172.21.2.132 gibba032.front.sepia.ceph.com gibba032 2021-10-15T15:31:32.848 INFO:teuthology.provision.fog.gibba030:Deploy complete! 2021-10-15T15:31:46.611 INFO:teuthology.provision.fog.gibba042:Node is ready 2021-10-15T15:31:46.624 INFO:teuthology.orchestra.run.gibba042.stdout:gibba042.front.sepia.ceph.com 2021-10-15T15:31:46.675 INFO:teuthology.orchestra.run.gibba042.stdout:172.21.2.142 gibba042.front.sepia.ceph.com gibba042 2021-10-15T15:31:46.977 INFO:teuthology.provision.fog.gibba042:Deploy complete! 2021-10-15T15:31:47.106 INFO:teuthology.lock.ops:Checking gibba030.front.sepia.ceph.com 2021-10-15T15:31:47.107 INFO:teuthology.lock.ops:New key found. Updating... 2021-10-15T15:31:47.124 INFO:teuthology.lock.ops:Checking gibba042.front.sepia.ceph.com 2021-10-15T15:31:47.124 INFO:teuthology.lock.ops:New key found. Updating... 2021-10-15T15:31:47.140 INFO:teuthology.lock.ops:Updating [gibba030.front.sepia.ceph.com]: set os type and version on server 2021-10-15T15:31:47.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba030.front.sepia.ceph.com' 2021-10-15T15:31:47.369 INFO:teuthology.orchestra.run.gibba030.stdout:x86_64 2021-10-15T15:31:47.421 INFO:teuthology.orchestra.run.gibba030.stdout:NAME="CentOS Linux" 2021-10-15T15:31:47.422 INFO:teuthology.orchestra.run.gibba030.stdout:VERSION="8" 2021-10-15T15:31:47.422 INFO:teuthology.orchestra.run.gibba030.stdout:ID="centos" 2021-10-15T15:31:47.422 INFO:teuthology.orchestra.run.gibba030.stdout:ID_LIKE="rhel fedora" 2021-10-15T15:31:47.422 INFO:teuthology.orchestra.run.gibba030.stdout:VERSION_ID="8" 2021-10-15T15:31:47.423 INFO:teuthology.orchestra.run.gibba030.stdout:PLATFORM_ID="platform:el8" 2021-10-15T15:31:47.423 INFO:teuthology.orchestra.run.gibba030.stdout:PRETTY_NAME="CentOS Linux 8" 2021-10-15T15:31:47.423 INFO:teuthology.orchestra.run.gibba030.stdout:ANSI_COLOR="0;31" 2021-10-15T15:31:47.423 INFO:teuthology.orchestra.run.gibba030.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-15T15:31:47.423 INFO:teuthology.orchestra.run.gibba030.stdout:HOME_URL="https://centos.org/" 2021-10-15T15:31:47.424 INFO:teuthology.orchestra.run.gibba030.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-15T15:31:47.424 INFO:teuthology.orchestra.run.gibba030.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-15T15:31:47.424 INFO:teuthology.orchestra.run.gibba030.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-15T15:31:47.425 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server 2021-10-15T15:31:47.441 INFO:teuthology.lock.ops:Updating [gibba042.front.sepia.ceph.com]: set os type and version on server 2021-10-15T15:31:47.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba042.front.sepia.ceph.com' 2021-10-15T15:31:47.666 INFO:teuthology.orchestra.run.gibba042.stdout:x86_64 2021-10-15T15:31:47.717 INFO:teuthology.orchestra.run.gibba042.stdout:NAME="CentOS Linux" 2021-10-15T15:31:47.718 INFO:teuthology.orchestra.run.gibba042.stdout:VERSION="8" 2021-10-15T15:31:47.718 INFO:teuthology.orchestra.run.gibba042.stdout:ID="centos" 2021-10-15T15:31:47.718 INFO:teuthology.orchestra.run.gibba042.stdout:ID_LIKE="rhel fedora" 2021-10-15T15:31:47.718 INFO:teuthology.orchestra.run.gibba042.stdout:VERSION_ID="8" 2021-10-15T15:31:47.719 INFO:teuthology.orchestra.run.gibba042.stdout:PLATFORM_ID="platform:el8" 2021-10-15T15:31:47.719 INFO:teuthology.orchestra.run.gibba042.stdout:PRETTY_NAME="CentOS Linux 8" 2021-10-15T15:31:47.719 INFO:teuthology.orchestra.run.gibba042.stdout:ANSI_COLOR="0;31" 2021-10-15T15:31:47.719 INFO:teuthology.orchestra.run.gibba042.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-15T15:31:47.720 INFO:teuthology.orchestra.run.gibba042.stdout:HOME_URL="https://centos.org/" 2021-10-15T15:31:47.720 INFO:teuthology.orchestra.run.gibba042.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-15T15:31:47.720 INFO:teuthology.orchestra.run.gibba042.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-15T15:31:47.720 INFO:teuthology.orchestra.run.gibba042.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-15T15:31:47.721 INFO:teuthology.lock.ops:Updating gibba042.front.sepia.ceph.com on lock server 2021-10-15T15:31:47.802 INFO:teuthology.dispatcher.supervisor:Running job 6442584 2021-10-15T15:31:47.802 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-10-14_06:00:04-rbd-octopus-distro-default-gibba/6442584 --name teuthology-2021-10-14_06:00:04-rbd-octopus-distro-default-gibba --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} -- /home/teuthworker/archive/teuthology-2021-10-14_06:00:04-rbd-octopus-distro-default-gibba/6442584/orig.config.yaml 2021-10-15T15:31:47.805 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-10-14_06:00:04-rbd-octopus-distro-default-gibba/6442584 2021-10-15T15:31:47.806 INFO:teuthology.dispatcher.supervisor:Job PID: 19632 2021-10-15T15:31:47.806 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-10-16T03:32:39.383 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-10-16T03:32:39.631 INFO:teuthology.kill:Killing Pids: {19632} 2021-10-16T03:32:41.977 INFO:teuthology.task.internal:roles: ubuntu@gibba030.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-10-16T03:32:41.980 INFO:teuthology.task.internal:roles: ubuntu@gibba042.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-10-16T03:32:41.981 INFO:teuthology.misc:Compressing logs... 2021-10-16T03:32:41.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba030.front.sepia.ceph.com' 2021-10-16T03:32:42.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@gibba042.front.sepia.ceph.com' 2021-10-16T03:33:40.035 INFO:teuthology.kill:No teuthology processes running 2021-10-16T03:33:41.302 INFO:teuthology.kill:Nuking machines: ['gibba030', 'gibba042'] 2021-10-16T03:33:42.425 INFO:teuthology.kill:2021-10-16 03:33:42,424.424 INFO:teuthology.nuke:targets: 2021-10-16T03:33:42.426 INFO:teuthology.kill: gibba030.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlbHlYAAUyAb5QVbxBRZgzh3bUvmgDzMS3ptAuP15r5fvndkH4OmQ8k/LkPk6mnsnES6RdB95A7EyI8kQR/IkRhoNJpZvpREdcRW6oDme4CF54+6rSV8XuM8vMm1McecVTZ0LbvEFxE8h6+ltFfs7cy0G3L1JgnIeenTX0HAYMjV25qTSJRrPpETcrovjtZcyGvas+eS1+XjOIIItGILjylTpq21Q8uffEzctfwzlkFqPhS0jOKhRVgxagkXKB0zuKJzet96emQGQFGi+zPeCX7UIfct2NHV/FHlWF8HWyH6bDghQpOAbhwzyv3VXJn/k2O9wmR7HS00h5wAe2a0j2NeWCNsjna9INGoLz/zXpyZ25ySVCTk2FC2rAykMVd1G887tVJDyxgioNs3dQz0ypztsn7wLARaMlLeyZpxe7GeNY+0ophN4DIypSlqRtL74vJCrVG7JDvUoaGefX+ybzrV6d+aIcCwIBjq3awwYOADZBDFHA40vYGgT7q1AySAE= 2021-10-16T03:33:42.426 INFO:teuthology.kill: gibba042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD99qE3C5li68M7IyAGBe1k9RaJHr2zu4JYeaoipEGpsaC/U62GVlnv6yA0PPPC8zRJGeWWylAnKlB0i8QB2slusjroME3CS/PaZsUjuVNLCpW6k9Eh3wP2vYD6830yxYm9dwq2gZlkQI42DN64XCDddcLJhHubYHXkyhHTcc1yAQIU3zw3AxcDxgJF33+MGEt+e8KmU00cIaNpXXcp3enzSTETiKShKOsT4rezIG82odWTMdGR9APW/2mWJEjfcQMqQ1J5/Lwq81ZMNmHIFYPhd6xnwofWZgd3lH0Rsgzd9Z1IlafnQs7jG+HP60scc502QzjVkA5qq5Lc7WK8Q0m1Ya/3qx/R5XYl4oWpJ0TQXsUsJ6BrpDYTuncMgjl4DZxajB0ThpKxYPmXlU085G1fXuy+Yx3desh9F17/UQYLhCpfqIBW4I5ah+ZDF6arZm3isD6zzxx6zI3lh3cZYCmexHvneyHB1kHHqk1nwsqEiMNqHPNJH7u1EjIWYwvBySk= 2021-10-16T03:33:42.450 INFO:teuthology.kill:2021-10-16 03:33:42,450.450 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-16T03:33:42.454 INFO:teuthology.kill:2021-10-16 03:33:42,454.454 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-16T03:33:42.525 INFO:teuthology.kill:2021-10-16 03:33:42,524.524 INFO:teuthology.orchestra.console:Power off gibba030 2021-10-16T03:33:42.538 INFO:teuthology.kill:2021-10-16 03:33:42,538.538 INFO:teuthology.orchestra.console:Power off gibba042 2021-10-16T03:33:50.904 INFO:teuthology.kill:2021-10-16 03:33:50,904.904 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-10-16T03:33:50.911 INFO:teuthology.kill:2021-10-16 03:33:50,911.911 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-10-16T03:33:51.049 INFO:teuthology.kill:2021-10-16 03:33:51,048.048 INFO:teuthology.lock.ops:unlocked: gibba030.front.sepia.ceph.com 2021-10-16T03:33:51.052 INFO:teuthology.kill:2021-10-16 03:33:51,052.052 INFO:teuthology.lock.ops:unlocked: gibba042.front.sepia.ceph.com 2021-10-16T03:35:51.367 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-10-16T03:35:51.489 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-10-16T03:35:51.490 INFO:teuthology.nuke:Checking targets against current locks 2021-10-16T03:35:51.603 INFO:teuthology.nuke:Not nuking gibba042.front.sepia.ceph.com because description doesn't match 2021-10-16T03:35:51.604 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e4202151f92bf096cded41ad9b054cfeb0e4b2c0/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable