2021-09-15T22:09:30.446 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-15T22:09:30.545 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi019.front.sepia.ceph.com' reimaging 2021-09-15T22:09:30.546 INFO:teuthology.lock.ops:Updating [ubuntu@smithi019.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T22:09:30.546 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2021-09-15T22:09:30.564 INFO:teuthology.lock.ops:Node 'ubuntu@smithi019.front.sepia.ceph.com' reimaging is complete 2021-09-15T22:09:30.564 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi026.front.sepia.ceph.com' reimaging 2021-09-15T22:09:30.564 INFO:teuthology.lock.ops:Updating [ubuntu@smithi026.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T22:09:30.565 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2021-09-15T22:09:30.582 INFO:teuthology.lock.ops:Node 'ubuntu@smithi026.front.sepia.ceph.com' reimaging is complete 2021-09-15T22:09:30.778 INFO:teuthology.provision.fog.smithi019:Scheduling deploy of centos 8.2 2021-09-15T22:09:30.786 INFO:teuthology.provision.fog.smithi026:Scheduling deploy of centos 8.2 2021-09-15T22:09:31.072 INFO:teuthology.orchestra.console:Power off smithi019 2021-09-15T22:09:31.106 INFO:teuthology.orchestra.console:Power off smithi026 2021-09-15T22:09:39.437 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2021-09-15T22:09:39.466 INFO:teuthology.orchestra.console:Power off for smithi026 completed 2021-09-15T22:09:39.538 INFO:teuthology.orchestra.console:Power on smithi019 2021-09-15T22:09:39.567 INFO:teuthology.orchestra.console:Power on smithi026 2021-09-15T22:09:47.907 INFO:teuthology.orchestra.console:Power on for smithi019 completed 2021-09-15T22:09:47.930 INFO:teuthology.orchestra.console:Power on for smithi026 completed 2021-09-15T22:09:48.010 INFO:teuthology.provision.fog.smithi019:Waiting for deploy to finish 2021-09-15T22:09:48.031 INFO:teuthology.provision.fog.smithi026:Waiting for deploy to finish 2021-09-15T22:13:55.654 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T22:13:55.958 INFO:teuthology.provision.fog.smithi026:Node is ready 2021-09-15T22:13:56.014 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026.front.sepia.ceph.com 2021-09-15T22:13:56.070 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26 smithi026.front.sepia.ceph.com smithi026 2021-09-15T22:13:56.402 INFO:teuthology.provision.fog.smithi026:Deploy complete! 2021-09-15T22:14:04.728 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T22:14:05.041 INFO:teuthology.provision.fog.smithi019:Node is ready 2021-09-15T22:14:05.057 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com 2021-09-15T22:14:05.113 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19 smithi019.front.sepia.ceph.com smithi019 2021-09-15T22:14:05.445 INFO:teuthology.provision.fog.smithi019:Deploy complete! 2021-09-15T22:14:05.620 INFO:teuthology.lock.ops:Checking smithi019.front.sepia.ceph.com 2021-09-15T22:14:05.621 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-15T22:14:05.639 INFO:teuthology.lock.ops:Checking smithi026.front.sepia.ceph.com 2021-09-15T22:14:05.639 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-15T22:14:05.656 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: set os type and version on server 2021-09-15T22:14:05.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-15T22:14:05.920 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2021-09-15T22:14:05.975 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Linux" 2021-09-15T22:14:05.975 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8 (Core)" 2021-09-15T22:14:05.976 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2021-09-15T22:14:05.976 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2021-09-15T22:14:05.976 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8" 2021-09-15T22:14:05.976 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8" 2021-09-15T22:14:05.977 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-09-15T22:14:05.977 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2021-09-15T22:14:05.977 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-15T22:14:05.977 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.centos.org/" 2021-09-15T22:14:05.978 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-15T22:14:05.978 INFO:teuthology.orchestra.run.smithi019.stdout: 2021-09-15T22:14:05.978 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-15T22:14:05.978 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-15T22:14:05.978 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-09-15T22:14:05.979 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-09-15T22:14:05.979 INFO:teuthology.orchestra.run.smithi019.stdout: 2021-09-15T22:14:05.980 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2021-09-15T22:14:06.002 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: set os type and version on server 2021-09-15T22:14:06.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-15T22:14:06.287 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2021-09-15T22:14:06.343 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Linux" 2021-09-15T22:14:06.343 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8 (Core)" 2021-09-15T22:14:06.343 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2021-09-15T22:14:06.343 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2021-09-15T22:14:06.344 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8" 2021-09-15T22:14:06.344 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2021-09-15T22:14:06.344 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-09-15T22:14:06.344 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2021-09-15T22:14:06.345 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-15T22:14:06.345 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://www.centos.org/" 2021-09-15T22:14:06.345 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-15T22:14:06.345 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-09-15T22:14:06.345 INFO:teuthology.orchestra.run.smithi026.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-15T22:14:06.346 INFO:teuthology.orchestra.run.smithi026.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-15T22:14:06.346 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-09-15T22:14:06.346 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-09-15T22:14:06.346 INFO:teuthology.orchestra.run.smithi026.stdout: 2021-09-15T22:14:06.347 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2021-09-15T22:14:06.443 INFO:teuthology.dispatcher.supervisor:Running job 6391775 2021-09-15T22:14:06.444 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-09-15_21:37:43-upgrade:octopus-x-master-distro-basic-smithi/6391775 --name teuthology-2021-09-15_21:37:43-upgrade:octopus-x-master-distro-basic-smithi --description upgrade:octopus-x/stress-split/{0-distro/centos_8.2_container_tools_3.0 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} -- /home/teuthworker/archive/teuthology-2021-09-15_21:37:43-upgrade:octopus-x-master-distro-basic-smithi/6391775/orig.config.yaml 2021-09-15T22:14:06.448 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-09-15_21:37:43-upgrade:octopus-x-master-distro-basic-smithi/6391775 2021-09-15T22:14:06.449 INFO:teuthology.dispatcher.supervisor:Job PID: 22862 2021-09-15T22:14:06.449 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-09-16T10:14:54.804 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-09-16T10:14:54.984 INFO:teuthology.kill:Killing Pids: {22862} 2021-09-16T10:14:56.119 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi026'] 2021-09-16T10:14:57.116 INFO:teuthology.kill:2021-09-16 10:14:57,115.115 INFO:teuthology.nuke:targets: 2021-09-16T10:14:57.116 INFO:teuthology.kill: smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaBGHE1KtBnTPzrJ5Qezhso86ZCyOozfUYqozkKiI9WWkbrtj9l3eAcOEnUkh9L+e+CQXJHD3Clt1hqNhP8QJ8ppa9llNeEF6nTVmzhA0aVqFptJ1S0B5tVjbCkNrPdvd00QX8N5awFrKdOqxp4PgSQCm9CKkaZrZjq6eMcXRYDlnaVWDk1oe0DPJPYa0Ra/jzJhUFaDV/6lxef5z/yPZYonp5cK1DWT5UxlCxcMBDc4RLWKGUD94Cfp+oRQTRrQgUX08zVf4vpoN1eAAUdONncKtYW6wuTPozlLJd6s9L9HhVt7TC82VxqQK3K15sPBvpU1vuBaGtnQTMsq/TTvluKsKeKwIUbqdSE6S0+z0R1xUlGYleoepihZ9Y3YlMZgOWFPSlZlSxdEhdy15CtEGhrF+X9iBhegPznuMXq7ZxSp9MVVwCHLqVAnwGHwUgrD1c80/3PA2zMlx9SUMJzZkS+k5XqgXvC85ypId/Ov8dgenqMe4swggm2yBr81KfJlk= 2021-09-16T10:14:57.117 INFO:teuthology.kill: smithi026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLv26cq9BYOMJ8nEUYYcDdRB5odGr6MDwNegp5lp9Pg4k/fDIA+YPP8WGu5MFvIyAC0POeBQ6pKFlRUQWwc1qJw8fhNy24iidsmXpBZqFjN6dXyQypAgY1y+cZqM6v6tLp7fGntC2gr/Y/XIrW0K4vTUKz3lpNjupcex6Pb7c9Wh4mfKgUey6WiGQWDtExk5idZTEimu1PU+X5qAWU8O1toQDNBdJ2I8sfyBgypiitPZ1RkaejqtK0xf9pYxP1XEwvFCo3IF+9GBvHo58nLXq0tuYvu1klKt6cz1FA4s9fh1xTA7HaXKQwqINh3weQ4TIqEd4eWf56E5tokW4V1KLSPXK27fibN38AeZBBkZV4jC9n9IV3RNho6bCCrQJXJzSNkwTywiEKQq3pp9L1uYG2x56KiLyu/ZoWN3/YUHibAOtXSsSiD1rcUqzEPt4xdmvKXQffLuNKE5xlwJX0tmCxSTGx+lNs46rxVtaMH7ltbUou1GPw9d9GNWB43N63p7E= 2021-09-16T10:14:57.117 INFO:teuthology.kill:2021-09-16 10:14:57,116.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T10:14:57.118 INFO:teuthology.kill:2021-09-16 10:14:57,118.118 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T10:14:57.181 INFO:teuthology.kill:2021-09-16 10:14:57,181.181 INFO:teuthology.orchestra.console:Power off smithi026 2021-09-16T10:14:57.196 INFO:teuthology.kill:2021-09-16 10:14:57,195.195 INFO:teuthology.orchestra.console:Power off smithi019 2021-09-16T10:15:05.556 INFO:teuthology.kill:2021-09-16 10:15:05,555.555 INFO:teuthology.orchestra.console:Power off for smithi026 completed 2021-09-16T10:15:09.670 INFO:teuthology.kill:2021-09-16 10:15:09,669.669 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2021-09-16T10:15:09.921 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-09-16T10:15:09.921 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-09-16T10:15:09.922 INFO:teuthology.misc:Compressing logs... 2021-09-16T10:15:09.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-16T10:15:44.587 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-09-16T10:15:44.684 INFO:teuthology.kill:No teuthology processes running 2021-09-16T10:15:45.748 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi026'] 2021-09-16T10:15:46.770 INFO:teuthology.kill:2021-09-16 10:15:46,769.769 INFO:teuthology.nuke:targets: 2021-09-16T10:15:46.770 INFO:teuthology.kill: smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaBGHE1KtBnTPzrJ5Qezhso86ZCyOozfUYqozkKiI9WWkbrtj9l3eAcOEnUkh9L+e+CQXJHD3Clt1hqNhP8QJ8ppa9llNeEF6nTVmzhA0aVqFptJ1S0B5tVjbCkNrPdvd00QX8N5awFrKdOqxp4PgSQCm9CKkaZrZjq6eMcXRYDlnaVWDk1oe0DPJPYa0Ra/jzJhUFaDV/6lxef5z/yPZYonp5cK1DWT5UxlCxcMBDc4RLWKGUD94Cfp+oRQTRrQgUX08zVf4vpoN1eAAUdONncKtYW6wuTPozlLJd6s9L9HhVt7TC82VxqQK3K15sPBvpU1vuBaGtnQTMsq/TTvluKsKeKwIUbqdSE6S0+z0R1xUlGYleoepihZ9Y3YlMZgOWFPSlZlSxdEhdy15CtEGhrF+X9iBhegPznuMXq7ZxSp9MVVwCHLqVAnwGHwUgrD1c80/3PA2zMlx9SUMJzZkS+k5XqgXvC85ypId/Ov8dgenqMe4swggm2yBr81KfJlk= 2021-09-16T10:15:46.771 INFO:teuthology.kill: smithi026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLv26cq9BYOMJ8nEUYYcDdRB5odGr6MDwNegp5lp9Pg4k/fDIA+YPP8WGu5MFvIyAC0POeBQ6pKFlRUQWwc1qJw8fhNy24iidsmXpBZqFjN6dXyQypAgY1y+cZqM6v6tLp7fGntC2gr/Y/XIrW0K4vTUKz3lpNjupcex6Pb7c9Wh4mfKgUey6WiGQWDtExk5idZTEimu1PU+X5qAWU8O1toQDNBdJ2I8sfyBgypiitPZ1RkaejqtK0xf9pYxP1XEwvFCo3IF+9GBvHo58nLXq0tuYvu1klKt6cz1FA4s9fh1xTA7HaXKQwqINh3weQ4TIqEd4eWf56E5tokW4V1KLSPXK27fibN38AeZBBkZV4jC9n9IV3RNho6bCCrQJXJzSNkwTywiEKQq3pp9L1uYG2x56KiLyu/ZoWN3/YUHibAOtXSsSiD1rcUqzEPt4xdmvKXQffLuNKE5xlwJX0tmCxSTGx+lNs46rxVtaMH7ltbUou1GPw9d9GNWB43N63p7E= 2021-09-16T10:15:46.795 INFO:teuthology.kill:2021-09-16 10:15:46,795.795 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T10:15:46.798 INFO:teuthology.kill:2021-09-16 10:15:46,798.798 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T10:15:46.856 INFO:teuthology.kill:2021-09-16 10:15:46,855.855 INFO:teuthology.orchestra.console:Power off smithi019 2021-09-16T10:15:46.869 INFO:teuthology.kill:2021-09-16 10:15:46,868.868 INFO:teuthology.orchestra.console:Power off smithi026 2021-09-16T10:15:47.000 INFO:teuthology.kill:2021-09-16 10:15:47,000.000 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2021-09-16T10:15:47.014 INFO:teuthology.kill:2021-09-16 10:15:47,013.013 INFO:teuthology.orchestra.console:Power off for smithi026 completed 2021-09-16T10:15:47.144 INFO:teuthology.kill:2021-09-16 10:15:47,143.143 INFO:teuthology.lock.ops:unlocked smithi019.front.sepia.ceph.com 2021-09-16T10:15:47.155 INFO:teuthology.kill:2021-09-16 10:15:47,155.155 INFO:teuthology.lock.ops:unlocked smithi026.front.sepia.ceph.com 2021-09-16T10:17:47.457 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-09-16T10:17:47.588 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-09-16T10:17:47.588 INFO:teuthology.nuke:Checking targets against current locks 2021-09-16T10:17:47.673 INFO:teuthology.nuke:Not nuking smithi019.front.sepia.ceph.com because description doesn't match 2021-09-16T10:17:47.673 INFO:teuthology.nuke:Not nuking smithi026.front.sepia.ceph.com because description doesn't match