2021-04-01T18:16:36.970 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-01T18:16:37.075 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging 2021-04-01T18:16:37.076 INFO:teuthology.lock.ops:Updating [ubuntu@smithi001.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T18:16:37.076 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2021-04-01T18:16:37.092 INFO:teuthology.lock.ops:Node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging is complete 2021-04-01T18:16:37.092 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging 2021-04-01T18:16:37.092 INFO:teuthology.lock.ops:Updating [ubuntu@smithi141.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T18:16:37.093 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2021-04-01T18:16:37.118 INFO:teuthology.lock.ops:Node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging is complete 2021-04-01T18:16:37.289 INFO:teuthology.provision.fog.smithi001:Scheduling deploy of centos 8.3 2021-04-01T18:16:37.319 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of centos 8.3 2021-04-01T18:16:37.717 INFO:teuthology.orchestra.console:Power off smithi001 2021-04-01T18:16:37.756 INFO:teuthology.orchestra.console:Power off smithi141 2021-04-01T18:16:50.230 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-04-01T18:16:50.235 INFO:teuthology.orchestra.console:Power off for smithi141 completed 2021-04-01T18:16:50.332 INFO:teuthology.orchestra.console:Power on smithi001 2021-04-01T18:16:50.341 INFO:teuthology.orchestra.console:Power on smithi141 2021-04-01T18:16:54.598 INFO:teuthology.orchestra.console:Power on for smithi001 completed 2021-04-01T18:16:54.602 INFO:teuthology.orchestra.console:Power on for smithi141 completed 2021-04-01T18:16:54.700 INFO:teuthology.provision.fog.smithi001:Waiting for deploy to finish 2021-04-01T18:16:54.703 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish 2021-04-01T18:21:11.318 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T18:21:11.657 INFO:teuthology.provision.fog.smithi001:Node is ready 2021-04-01T18:21:11.676 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001.front.sepia.ceph.com 2021-04-01T18:21:11.737 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1 smithi001.front.sepia.ceph.com smithi001 2021-04-01T18:21:12.072 INFO:teuthology.provision.fog.smithi001:Deploy complete! 2021-04-01T18:21:41.441 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T18:21:41.758 INFO:teuthology.provision.fog.smithi141:Node is ready 2021-04-01T18:21:41.773 INFO:teuthology.orchestra.run.smithi141.stdout:smithi141.front.sepia.ceph.com 2021-04-01T18:21:41.827 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141 2021-04-01T18:21:42.145 INFO:teuthology.provision.fog.smithi141:Deploy complete! 2021-04-01T18:21:42.386 INFO:teuthology.lock.ops:Checking smithi001.front.sepia.ceph.com 2021-04-01T18:21:42.393 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-01T18:21:42.411 INFO:teuthology.lock.ops:Checking smithi141.front.sepia.ceph.com 2021-04-01T18:21:42.412 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-01T18:21:42.432 INFO:teuthology.lock.ops:Updating [smithi001.front.sepia.ceph.com]: set os type and version on server 2021-04-01T18:21:42.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-01T18:21:42.736 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2021-04-01T18:21:42.792 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Linux" 2021-04-01T18:21:42.792 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="8" 2021-04-01T18:21:42.793 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2021-04-01T18:21:42.793 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2021-04-01T18:21:42.793 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="8" 2021-04-01T18:21:42.794 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el8" 2021-04-01T18:21:42.794 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Linux 8" 2021-04-01T18:21:42.795 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2021-04-01T18:21:42.806 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-01T18:21:42.806 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2021-04-01T18:21:42.806 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-01T18:21:42.807 INFO:teuthology.orchestra.run.smithi001.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-01T18:21:42.807 INFO:teuthology.orchestra.run.smithi001.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-01T18:21:42.816 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2021-04-01T18:21:42.843 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: set os type and version on server 2021-04-01T18:21:42.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-01T18:21:43.140 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2021-04-01T18:21:43.196 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="CentOS Linux" 2021-04-01T18:21:43.197 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="8" 2021-04-01T18:21:43.197 INFO:teuthology.orchestra.run.smithi141.stdout:ID="centos" 2021-04-01T18:21:43.197 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE="rhel fedora" 2021-04-01T18:21:43.198 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="8" 2021-04-01T18:21:43.198 INFO:teuthology.orchestra.run.smithi141.stdout:PLATFORM_ID="platform:el8" 2021-04-01T18:21:43.199 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="CentOS Linux 8" 2021-04-01T18:21:43.199 INFO:teuthology.orchestra.run.smithi141.stdout:ANSI_COLOR="0;31" 2021-04-01T18:21:43.200 INFO:teuthology.orchestra.run.smithi141.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-01T18:21:43.200 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://centos.org/" 2021-04-01T18:21:43.200 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-01T18:21:43.201 INFO:teuthology.orchestra.run.smithi141.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-01T18:21:43.201 INFO:teuthology.orchestra.run.smithi141.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-01T18:21:43.202 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2021-04-01T18:21:43.351 INFO:teuthology.dispatcher.supervisor:Running job 6015341 2021-04-01T18:21:43.352 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-04-01_17:51:51-rados-wip-ronenf-cscrub-class-distro-basic-smithi/6015341 --name rfriedma-2021-04-01_17:51:51-rados-wip-ronenf-cscrub-class-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rados_api_tests} -- /home/teuthworker/archive/rfriedma-2021-04-01_17:51:51-rados-wip-ronenf-cscrub-class-distro-basic-smithi/6015341/orig.config.yaml 2021-04-01T18:21:43.376 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-04-01_17:51:51-rados-wip-ronenf-cscrub-class-distro-basic-smithi/6015341 2021-04-01T18:21:43.377 INFO:teuthology.dispatcher.supervisor:Job PID: 31395 2021-04-01T18:21:43.377 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-02T06:22:57.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-02T06:22:57.961 INFO:teuthology.kill:Killing Pids: {31395} 2021-04-02T06:22:59.208 INFO:teuthology.kill:Nuking machines: ['smithi001', 'smithi141'] 2021-04-02T06:23:00.479 INFO:teuthology.kill:2021-04-02 06:23:00,478.478 INFO:teuthology.nuke:targets: 2021-04-02T06:23:00.480 INFO:teuthology.kill: smithi001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5Hd/qT0beMpImuKbJU2xi97O/Z1pGVPLu0EoZ96m5VuFbig3xsVDL3e7BHbAxzsGI9IHJd3O/S5n8/65adJ6VZA0drqh2+1f9TGu1QOSkQpn3Rj7DwIVLNuhR6RSS/a5/B3eMJo/8vXvXWZ1tK3RLhRNUFgc4/BicBGALu+u/E8VtYde15jhv3EqyusX1eED4e7vZeV+5ITMbgko3tKQDirnvSoKbWL8JUAVOCx8X5Cr2xCqIoMwc6f+a+ejKqdbx0OfxzoDSX8McPXHAPZmOTb3FZ7ZMchbOyqz6AF2GErhQEv9sSmFMaBH9AWFXNvTXNpqXoLm3T0hywY4lgeEGhsAVWETmf40Tq6drXyNpvQDrELMY+umJdjDj91ZRIDiZ63CJ4eZvyOCldXLlO8/OauzT92PvhYRmsLcHKscIn3cE3Lw0A9lWm0Uyw2U9zEOHYYKmgbTmxgwXZSvJ3VI27tO0RbfK1CW/HzYhkS/7DvrDGotELqRMr9yAyWjQ07E= 2021-04-02T06:23:00.481 INFO:teuthology.kill: smithi141.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdY0F2Xl3sM8nhCRbGeY88unPNNr3YoulgiFU0TNzZ2X4/CqbjB+Rio73syXeyfQLREF0wT/j+Ki+WNQhw0zbCKxDji2zBj0PKq3CKFSdxKZMkxK77XwZVmCytZOcyTru+Rpj0ZCddrsrpiw+c7KGiS+pRY///cTq2loCeIalLoXTPVbiCGcNS2UWdG29DiJEuSY+DBV9DccH3qJDesgcViZyV54/mNxxmrYjFDpjANEeRUEhQl2OOoXIABjM6udBwgX9FmQpmKlFCcaQWPibYCfTMxW3KDcRYuHP5bnJCsuv/CGQ37kw9f6H4yxr9IHqVE9idhIFd9vVD4NYxsuzkIwSgJfZ/4H33l2IjZMgTS4l+4YEDmiL6JlDk3R1FX86iI8o/xTuNMpDVNk3Ck3MnzdoR+P3btfM6qdxJc9DjjmPdlDmV/1mgYAsVlFHzndXaVq5LSc+4GGns9q1FumsMs9weKI/kaQ7j//Eu94hE/Ds2U3hV8JPLFackuSTYh8U= 2021-04-02T06:23:00.481 INFO:teuthology.kill:2021-04-02 06:23:00,479.479 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T06:23:00.483 INFO:teuthology.kill:2021-04-02 06:23:00,482.482 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T06:23:00.582 INFO:teuthology.kill:2021-04-02 06:23:00,582.582 INFO:teuthology.orchestra.console:Power off smithi141 2021-04-02T06:23:00.841 INFO:teuthology.kill:2021-04-02 06:23:00,601.601 INFO:teuthology.orchestra.console:Power off smithi001 2021-04-02T06:23:09.049 INFO:teuthology.kill:2021-04-02 06:23:09,048.048 INFO:teuthology.orchestra.console:Power off for smithi141 completed 2021-04-02T06:23:09.054 INFO:teuthology.kill:2021-04-02 06:23:09,054.054 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-04-02T06:23:09.291 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-04-02T06:23:09.294 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-04-02T06:23:09.295 INFO:teuthology.misc:Compressing logs... 2021-04-02T06:23:09.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-02T06:24:09.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-02T06:24:09.416 INFO:teuthology.kill:No teuthology processes running 2021-04-02T06:24:11.287 INFO:teuthology.kill:Nuking machines: ['smithi001', 'smithi141'] 2021-04-02T06:24:12.314 INFO:teuthology.kill:2021-04-02 06:24:12,313.313 INFO:teuthology.nuke:targets: 2021-04-02T06:24:12.315 INFO:teuthology.kill: smithi001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5Hd/qT0beMpImuKbJU2xi97O/Z1pGVPLu0EoZ96m5VuFbig3xsVDL3e7BHbAxzsGI9IHJd3O/S5n8/65adJ6VZA0drqh2+1f9TGu1QOSkQpn3Rj7DwIVLNuhR6RSS/a5/B3eMJo/8vXvXWZ1tK3RLhRNUFgc4/BicBGALu+u/E8VtYde15jhv3EqyusX1eED4e7vZeV+5ITMbgko3tKQDirnvSoKbWL8JUAVOCx8X5Cr2xCqIoMwc6f+a+ejKqdbx0OfxzoDSX8McPXHAPZmOTb3FZ7ZMchbOyqz6AF2GErhQEv9sSmFMaBH9AWFXNvTXNpqXoLm3T0hywY4lgeEGhsAVWETmf40Tq6drXyNpvQDrELMY+umJdjDj91ZRIDiZ63CJ4eZvyOCldXLlO8/OauzT92PvhYRmsLcHKscIn3cE3Lw0A9lWm0Uyw2U9zEOHYYKmgbTmxgwXZSvJ3VI27tO0RbfK1CW/HzYhkS/7DvrDGotELqRMr9yAyWjQ07E= 2021-04-02T06:24:12.315 INFO:teuthology.kill: smithi141.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdY0F2Xl3sM8nhCRbGeY88unPNNr3YoulgiFU0TNzZ2X4/CqbjB+Rio73syXeyfQLREF0wT/j+Ki+WNQhw0zbCKxDji2zBj0PKq3CKFSdxKZMkxK77XwZVmCytZOcyTru+Rpj0ZCddrsrpiw+c7KGiS+pRY///cTq2loCeIalLoXTPVbiCGcNS2UWdG29DiJEuSY+DBV9DccH3qJDesgcViZyV54/mNxxmrYjFDpjANEeRUEhQl2OOoXIABjM6udBwgX9FmQpmKlFCcaQWPibYCfTMxW3KDcRYuHP5bnJCsuv/CGQ37kw9f6H4yxr9IHqVE9idhIFd9vVD4NYxsuzkIwSgJfZ/4H33l2IjZMgTS4l+4YEDmiL6JlDk3R1FX86iI8o/xTuNMpDVNk3Ck3MnzdoR+P3btfM6qdxJc9DjjmPdlDmV/1mgYAsVlFHzndXaVq5LSc+4GGns9q1FumsMs9weKI/kaQ7j//Eu94hE/Ds2U3hV8JPLFackuSTYh8U= 2021-04-02T06:24:12.338 INFO:teuthology.kill:2021-04-02 06:24:12,338.338 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T06:24:12.351 INFO:teuthology.kill:2021-04-02 06:24:12,351.351 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T06:24:12.460 INFO:teuthology.kill:2021-04-02 06:24:12,459.459 INFO:teuthology.orchestra.console:Power off smithi001 2021-04-02T06:24:12.483 INFO:teuthology.kill:2021-04-02 06:24:12,483.483 INFO:teuthology.orchestra.console:Power off smithi141 2021-04-02T06:24:12.732 INFO:teuthology.kill:2021-04-02 06:24:12,732.732 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-04-02T06:24:12.734 INFO:teuthology.kill:2021-04-02 06:24:12,734.734 INFO:teuthology.orchestra.console:Power off for smithi141 completed 2021-04-02T06:24:12.911 INFO:teuthology.kill:2021-04-02 06:24:12,910.910 INFO:teuthology.lock.ops:unlocked smithi001.front.sepia.ceph.com 2021-04-02T06:24:12.927 INFO:teuthology.kill:2021-04-02 06:24:12,926.926 INFO:teuthology.lock.ops:unlocked smithi141.front.sepia.ceph.com 2021-04-02T06:26:13.269 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-02T06:26:13.377 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-02T06:26:13.378 INFO:teuthology.nuke:Checking targets against current locks 2021-04-02T06:26:13.456 INFO:teuthology.nuke:Not nuking smithi001.front.sepia.ceph.com because description doesn't match 2021-04-02T06:26:13.456 INFO:teuthology.nuke:Not nuking smithi141.front.sepia.ceph.com because description doesn't match