2021-03-03T02:17:35.771 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-03T02:17:35.864 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging 2021-03-03T02:17:35.865 INFO:teuthology.lock.ops:Updating [ubuntu@smithi162.front.sepia.ceph.com]: reset os type and version on server 2021-03-03T02:17:35.866 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2021-03-03T02:17:35.880 INFO:teuthology.lock.ops:Node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging is complete 2021-03-03T02:17:35.880 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging 2021-03-03T02:17:35.881 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server 2021-03-03T02:17:35.881 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-03-03T02:17:35.896 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete 2021-03-03T02:17:36.094 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of rhel 8.3 2021-03-03T02:17:36.103 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of rhel 8.3 2021-03-03T02:17:36.424 INFO:teuthology.orchestra.console:Power off smithi162 2021-03-03T02:17:36.460 INFO:teuthology.orchestra.console:Power off smithi187 2021-03-03T02:17:44.818 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-03-03T02:17:44.919 INFO:teuthology.orchestra.console:Power on smithi187 2021-03-03T02:17:48.907 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2021-03-03T02:17:49.009 INFO:teuthology.orchestra.console:Power on smithi162 2021-03-03T02:17:53.286 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2021-03-03T02:17:53.387 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2021-03-03T02:17:57.375 INFO:teuthology.orchestra.console:Power on for smithi162 completed 2021-03-03T02:17:57.476 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish 2021-03-03T02:22:25.076 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-03T02:22:25.349 INFO:teuthology.provision.fog.smithi187:Node is ready 2021-03-03T02:22:25.364 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2021-03-03T02:22:25.418 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2021-03-03T02:22:25.738 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2021-03-03T02:22:53.103 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-03T02:22:53.429 INFO:teuthology.provision.fog.smithi162:Node is ready 2021-03-03T02:22:53.444 INFO:teuthology.orchestra.run.smithi162.stdout:smithi162.front.sepia.ceph.com 2021-03-03T02:22:53.498 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.15.162 smithi162.front.sepia.ceph.com smithi162 2021-03-03T02:22:53.816 INFO:teuthology.provision.fog.smithi162:Deploy complete! 2021-03-03T02:22:53.936 INFO:teuthology.lock.ops:Checking smithi162.front.sepia.ceph.com 2021-03-03T02:22:53.937 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-03T02:22:53.955 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com 2021-03-03T02:22:53.956 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-03T02:22:53.970 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: set os type and version on server 2021-03-03T02:22:53.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-03T02:22:54.253 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2021-03-03T02:22:54.307 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Red Hat Enterprise Linux" 2021-03-03T02:22:54.308 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8.3 (Ootpa)" 2021-03-03T02:22:54.308 INFO:teuthology.orchestra.run.smithi162.stdout:ID="rhel" 2021-03-03T02:22:54.308 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="fedora" 2021-03-03T02:22:54.309 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8.3" 2021-03-03T02:22:54.309 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2021-03-03T02:22:54.309 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-03T02:22:54.309 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2021-03-03T02:22:54.309 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-03T02:22:54.310 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.redhat.com/" 2021-03-03T02:22:54.310 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-03T02:22:54.310 INFO:teuthology.orchestra.run.smithi162.stdout: 2021-03-03T02:22:54.310 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-03T02:22:54.311 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-03T02:22:54.311 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-03T02:22:54.311 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-03T02:22:54.312 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2021-03-03T02:22:54.329 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server 2021-03-03T02:22:54.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-03T02:22:54.615 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2021-03-03T02:22:54.669 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Red Hat Enterprise Linux" 2021-03-03T02:22:54.670 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8.3 (Ootpa)" 2021-03-03T02:22:54.670 INFO:teuthology.orchestra.run.smithi187.stdout:ID="rhel" 2021-03-03T02:22:54.670 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="fedora" 2021-03-03T02:22:54.671 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8.3" 2021-03-03T02:22:54.671 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2021-03-03T02:22:54.671 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-03T02:22:54.671 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2021-03-03T02:22:54.671 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-03T02:22:54.672 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.redhat.com/" 2021-03-03T02:22:54.672 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-03T02:22:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-03-03T02:22:54.672 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-03T02:22:54.673 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-03T02:22:54.673 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-03T02:22:54.673 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-03T02:22:54.674 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-03-03T02:22:54.750 INFO:teuthology.dispatcher.supervisor:Running job 5927782 2021-03-03T02:22:54.750 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology -v --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi/5927782 --name rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/sync workloads/pool-create-delete} -- /home/teuthworker/archive/rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi/5927782/orig.config.yaml 2021-03-03T02:22:54.753 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rzarzynski-2021-03-02_18:30:36-rados-master-distro-basic-smithi/5927782 2021-03-03T02:22:54.754 INFO:teuthology.dispatcher.supervisor:Job PID: 13938 2021-03-03T02:22:54.755 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-03T14:23:46.012 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-03T14:23:50.623 INFO:teuthology.kill:Killing Pids: {13938} 2021-03-03T14:23:51.823 INFO:teuthology.kill:Nuking machines: ['smithi162', 'smithi187'] 2021-03-03T14:23:52.840 INFO:teuthology.kill:2021-03-03 14:23:52,839.839 INFO:teuthology.nuke:targets: 2021-03-03T14:23:52.841 INFO:teuthology.kill: smithi162.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPwuC99uZde/a3Kzv4VJlGQSfKtbJtqQNp8evoy0Rb94ncteX3P6/cto1JGt9O1TBHz+DXQFdGKeMVIQzc4lObrPJzoNsLYDOsFg4MU+PeDYaya+zKmlYc0YXWCjvy8r/Fkc69BZr8VPt2H2/MBpNVf9V9THh5aPiZOacU8wKhyWOMRuo3PiGLHfWChxFOiy5JmQE85kN+bB2XzoX+8giPgabc2YArQvRQgIYWixIFnGb0LQXJ7OooHbwsfGxVbgJhil4aChaqNkr/tvLXF172wGwT3N4WGqVout1yUUJGINomLGekuti68N4BnEhjfI5kzAW9hJxROjqj4RsuFJI5l1DpodnRWwufhyywgvTRelAKoeXQAdUk1B59RHQ1ti5y9CbruN43LBjyU9uRuP1qt/EeyCySQF837wwzx/J2l2fgthB76xkOCEAJFkj5rWPz2om5MNURy1GZGXwgyXvj8XQ7tYrydXEtVUG4iV7Iga7DpLmKTESJGbWnBJA/Ao0= 2021-03-03T14:23:52.841 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ8Sh8ZuSmSdG/4MdmCU1f9ehK9neNcucY0hLT3qWg/VTCi9m/wbZi2zD8giitgvIUXS/dn1lH9BkzXrUw8DN+StxRcCVAJ+NoMprqzQDPVRGy9AXQOJYVR++XfSyUZMXxNwCv5v0mFTPwOjEXU3Wn+fjiv1ICh8YxHAzyPaSnPaD4TWngNLsRhBv6G6xJbPzAAKd6m/rbpFTku2m8x4Wa3vUWtM0dX0W53I2UyWdV/VB1aAA3MKd5+EZc3zHP7bq6ub9Bb6SfAGR4icR28FuNR+vOBfK0rjIU0RlAufGoAXUkf5y1lyrtDSwdDkzmkxaueb3N8BxGOGSNjk9S1FxkQI890gDNlEZaRl20vdEQW7LLFyiTaxEE8bYUa6Qg5X0JrtiuIuC7hf6G47FLfsX1GpBJEG6uZbaKy7zjtsfMdXUKZfQsiRTOjDxzIBBEGZCs4XmVmqx6+PrevRNE1OzHRYuKmv+kuEfUWF5Ie8PH5Fx+oETuld6ydTkgH4G/FOE= 2021-03-03T14:23:52.842 INFO:teuthology.kill:2021-03-03 14:23:52,840.840 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:23:52.842 INFO:teuthology.kill:2021-03-03 14:23:52,842.842 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:23:52.972 INFO:teuthology.kill:2021-03-03 14:23:52,972.972 INFO:teuthology.orchestra.console:Power off smithi187 2021-03-03T14:23:52.989 INFO:teuthology.kill:2021-03-03 14:23:52,989.989 INFO:teuthology.orchestra.console:Power off smithi162 2021-03-03T14:24:01.395 INFO:teuthology.kill:2021-03-03 14:24:01,395.395 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2021-03-03T14:24:05.529 INFO:teuthology.kill:2021-03-03 14:24:05,529.529 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-03-03T14:24:05.793 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2021-03-03T14:24:05.795 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2021-03-03T14:24:05.795 INFO:teuthology.misc:Compressing logs... 2021-03-03T14:24:05.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-03T14:25:05.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-03T14:25:05.931 INFO:teuthology.kill:No teuthology processes running 2021-03-03T14:25:07.105 INFO:teuthology.kill:Nuking machines: ['smithi162', 'smithi187'] 2021-03-03T14:25:08.011 INFO:teuthology.kill:2021-03-03 14:25:08,011.011 INFO:teuthology.nuke:targets: 2021-03-03T14:25:08.012 INFO:teuthology.kill: smithi162.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPwuC99uZde/a3Kzv4VJlGQSfKtbJtqQNp8evoy0Rb94ncteX3P6/cto1JGt9O1TBHz+DXQFdGKeMVIQzc4lObrPJzoNsLYDOsFg4MU+PeDYaya+zKmlYc0YXWCjvy8r/Fkc69BZr8VPt2H2/MBpNVf9V9THh5aPiZOacU8wKhyWOMRuo3PiGLHfWChxFOiy5JmQE85kN+bB2XzoX+8giPgabc2YArQvRQgIYWixIFnGb0LQXJ7OooHbwsfGxVbgJhil4aChaqNkr/tvLXF172wGwT3N4WGqVout1yUUJGINomLGekuti68N4BnEhjfI5kzAW9hJxROjqj4RsuFJI5l1DpodnRWwufhyywgvTRelAKoeXQAdUk1B59RHQ1ti5y9CbruN43LBjyU9uRuP1qt/EeyCySQF837wwzx/J2l2fgthB76xkOCEAJFkj5rWPz2om5MNURy1GZGXwgyXvj8XQ7tYrydXEtVUG4iV7Iga7DpLmKTESJGbWnBJA/Ao0= 2021-03-03T14:25:08.013 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ8Sh8ZuSmSdG/4MdmCU1f9ehK9neNcucY0hLT3qWg/VTCi9m/wbZi2zD8giitgvIUXS/dn1lH9BkzXrUw8DN+StxRcCVAJ+NoMprqzQDPVRGy9AXQOJYVR++XfSyUZMXxNwCv5v0mFTPwOjEXU3Wn+fjiv1ICh8YxHAzyPaSnPaD4TWngNLsRhBv6G6xJbPzAAKd6m/rbpFTku2m8x4Wa3vUWtM0dX0W53I2UyWdV/VB1aAA3MKd5+EZc3zHP7bq6ub9Bb6SfAGR4icR28FuNR+vOBfK0rjIU0RlAufGoAXUkf5y1lyrtDSwdDkzmkxaueb3N8BxGOGSNjk9S1FxkQI890gDNlEZaRl20vdEQW7LLFyiTaxEE8bYUa6Qg5X0JrtiuIuC7hf6G47FLfsX1GpBJEG6uZbaKy7zjtsfMdXUKZfQsiRTOjDxzIBBEGZCs4XmVmqx6+PrevRNE1OzHRYuKmv+kuEfUWF5Ie8PH5Fx+oETuld6ydTkgH4G/FOE= 2021-03-03T14:25:08.032 INFO:teuthology.kill:2021-03-03 14:25:08,032.032 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:25:08.035 INFO:teuthology.kill:2021-03-03 14:25:08,035.035 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-03T14:25:08.100 INFO:teuthology.kill:2021-03-03 14:25:08,100.100 INFO:teuthology.orchestra.console:Power off smithi162 2021-03-03T14:25:08.113 INFO:teuthology.kill:2021-03-03 14:25:08,113.113 INFO:teuthology.orchestra.console:Power off smithi187 2021-03-03T14:25:08.245 INFO:teuthology.kill:2021-03-03 14:25:08,245.245 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2021-03-03T14:25:08.257 INFO:teuthology.kill:2021-03-03 14:25:08,257.257 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-03-03T14:25:08.386 INFO:teuthology.kill:2021-03-03 14:25:08,385.385 INFO:teuthology.lock.ops:unlocked smithi162.front.sepia.ceph.com 2021-03-03T14:25:08.399 INFO:teuthology.kill:2021-03-03 14:25:08,399.399 INFO:teuthology.lock.ops:unlocked smithi187.front.sepia.ceph.com 2021-03-03T14:27:08.687 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-03T14:27:08.794 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-03T14:27:08.794 INFO:teuthology.nuke:Checking targets against current locks 2021-03-03T14:27:08.898 INFO:teuthology.nuke:Not nuking smithi187.front.sepia.ceph.com because description doesn't match 2021-03-03T14:27:08.898 INFO:teuthology.nuke:Not nuking smithi162.front.sepia.ceph.com because description doesn't match