2021-04-06T17:38:09.633 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-06T17:38:09.747 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging 2021-04-06T17:38:09.748 INFO:teuthology.lock.ops:Updating [ubuntu@smithi032.front.sepia.ceph.com]: reset os type and version on server 2021-04-06T17:38:09.749 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-04-06T17:38:09.767 INFO:teuthology.lock.ops:Node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging is complete 2021-04-06T17:38:09.767 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi075.front.sepia.ceph.com' reimaging 2021-04-06T17:38:09.768 INFO:teuthology.lock.ops:Updating [ubuntu@smithi075.front.sepia.ceph.com]: reset os type and version on server 2021-04-06T17:38:09.768 INFO:teuthology.lock.ops:Updating smithi075.front.sepia.ceph.com on lock server 2021-04-06T17:38:09.784 INFO:teuthology.lock.ops:Node 'ubuntu@smithi075.front.sepia.ceph.com' reimaging is complete 2021-04-06T17:38:10.001 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of centos 8.3 2021-04-06T17:38:10.035 INFO:teuthology.provision.fog.smithi075:Scheduling deploy of centos 8.3 2021-04-06T17:38:10.418 INFO:teuthology.orchestra.console:Power off smithi032 2021-04-06T17:38:10.460 INFO:teuthology.orchestra.console:Power off smithi075 2021-04-06T17:38:18.823 INFO:teuthology.orchestra.console:Power off for smithi075 completed 2021-04-06T17:38:18.924 INFO:teuthology.orchestra.console:Power on smithi075 2021-04-06T17:38:22.908 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-04-06T17:38:23.010 INFO:teuthology.orchestra.console:Power on smithi032 2021-04-06T17:38:27.265 INFO:teuthology.orchestra.console:Power on for smithi032 completed 2021-04-06T17:38:27.286 INFO:teuthology.orchestra.console:Power on for smithi075 completed 2021-04-06T17:38:27.366 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish 2021-04-06T17:38:27.387 INFO:teuthology.provision.fog.smithi075:Waiting for deploy to finish 2021-04-06T17:42:39.581 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-06T17:42:39.904 INFO:teuthology.provision.fog.smithi032:Node is ready 2021-04-06T17:42:39.920 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com 2021-04-06T17:42:39.978 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032 2021-04-06T17:42:40.301 INFO:teuthology.provision.fog.smithi032:Deploy complete! 2021-04-06T17:47:32.483 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-06T17:47:32.781 INFO:teuthology.provision.fog.smithi075:Node is ready 2021-04-06T17:47:32.796 INFO:teuthology.orchestra.run.smithi075.stdout:smithi075.front.sepia.ceph.com 2021-04-06T17:47:32.852 INFO:teuthology.orchestra.run.smithi075.stdout:172.21.15.75 smithi075.front.sepia.ceph.com smithi075 2021-04-06T17:47:33.175 INFO:teuthology.provision.fog.smithi075:Deploy complete! 2021-04-06T17:48:01.923 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com 2021-04-06T17:48:01.924 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-06T17:48:03.078 INFO:teuthology.lock.ops:Checking smithi075.front.sepia.ceph.com 2021-04-06T17:48:03.079 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-06T17:48:03.550 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server 2021-04-06T17:48:03.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-06T17:48:03.776 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2021-04-06T17:48:03.834 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Linux" 2021-04-06T17:48:03.834 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8" 2021-04-06T17:48:03.835 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2021-04-06T17:48:03.835 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2021-04-06T17:48:03.835 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8" 2021-04-06T17:48:03.835 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2021-04-06T17:48:03.836 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Linux 8" 2021-04-06T17:48:03.836 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2021-04-06T17:48:03.836 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-06T17:48:03.836 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2021-04-06T17:48:03.837 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-06T17:48:03.837 INFO:teuthology.orchestra.run.smithi032.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-06T17:48:03.837 INFO:teuthology.orchestra.run.smithi032.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-06T17:48:03.838 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-04-06T17:48:03.938 INFO:teuthology.lock.ops:Updating [smithi075.front.sepia.ceph.com]: set os type and version on server 2021-04-06T17:48:03.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-06T17:48:04.237 INFO:teuthology.orchestra.run.smithi075.stdout:x86_64 2021-04-06T17:48:04.292 INFO:teuthology.orchestra.run.smithi075.stdout:NAME="CentOS Linux" 2021-04-06T17:48:04.292 INFO:teuthology.orchestra.run.smithi075.stdout:VERSION="8" 2021-04-06T17:48:04.293 INFO:teuthology.orchestra.run.smithi075.stdout:ID="centos" 2021-04-06T17:48:04.293 INFO:teuthology.orchestra.run.smithi075.stdout:ID_LIKE="rhel fedora" 2021-04-06T17:48:04.293 INFO:teuthology.orchestra.run.smithi075.stdout:VERSION_ID="8" 2021-04-06T17:48:04.293 INFO:teuthology.orchestra.run.smithi075.stdout:PLATFORM_ID="platform:el8" 2021-04-06T17:48:04.293 INFO:teuthology.orchestra.run.smithi075.stdout:PRETTY_NAME="CentOS Linux 8" 2021-04-06T17:48:04.294 INFO:teuthology.orchestra.run.smithi075.stdout:ANSI_COLOR="0;31" 2021-04-06T17:48:04.294 INFO:teuthology.orchestra.run.smithi075.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-06T17:48:04.294 INFO:teuthology.orchestra.run.smithi075.stdout:HOME_URL="https://centos.org/" 2021-04-06T17:48:04.294 INFO:teuthology.orchestra.run.smithi075.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-06T17:48:04.295 INFO:teuthology.orchestra.run.smithi075.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-06T17:48:04.295 INFO:teuthology.orchestra.run.smithi075.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-06T17:48:04.296 INFO:teuthology.lock.ops:Updating smithi075.front.sepia.ceph.com on lock server 2021-04-06T17:48:04.669 INFO:teuthology.dispatcher.supervisor:Running job 6024797 2021-04-06T17:48:04.670 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-04-06_17:31:13-rados-wip-ronenf-cscrub-class-distro-basic-smithi/6024797 --name rfriedma-2021-04-06_17:31:13-rados-wip-ronenf-cscrub-class-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big} -- /home/teuthworker/archive/rfriedma-2021-04-06_17:31:13-rados-wip-ronenf-cscrub-class-distro-basic-smithi/6024797/orig.config.yaml 2021-04-06T17:48:04.677 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-04-06_17:31:13-rados-wip-ronenf-cscrub-class-distro-basic-smithi/6024797 2021-04-06T17:48:04.678 INFO:teuthology.dispatcher.supervisor:Job PID: 26329 2021-04-06T17:48:04.679 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-06T18:42:53.598 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/rfriedma-2021-04-06_17:31:13-rados-wip-ronenf-cscrub-class-distro-basic-smithi/jobs/ failed with status 504: 504 Gateway Time-out

504 Gateway Time-out

nginx/1.10.3 (Ubuntu)
2021-04-07T05:49:43.692 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-07T05:49:44.481 INFO:teuthology.kill:Killing Pids: {26329} 2021-04-07T05:49:48.217 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi075'] 2021-04-07T05:49:49.726 INFO:teuthology.kill:2021-04-07 05:49:49,725.725 INFO:teuthology.nuke:targets: 2021-04-07T05:49:49.727 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCITY3u4YLB1lwrnaAxFQ2PwT5drYrkmIYCtlFhk3WK3ic256/sYCsun5KPUUYeqduCy9mWp+w87B6feby9YifuDmNx+CbnivoM1SvcYFn+mOFRBTmJMfS41CH65JDQDPxkBssiobp5z8ZThRYwmzKTbykIpdZRVnQ2s1Ntvb48E+pdQlyNkNJ130qF4RZyEA+6+NhC2hkRiXrCAk+jvwBMOjtZo45SCduropkmYEFjM1H+yYzDFN+TXZLWZ305P+FyKYoo3yxj+2wjVipGVouvEYmhH0Wg0ChpLE7Q2TaKvI/q3zSJJIUyYfAd3dEFHT/dYBy76udPswnOPjo7UYoL0ELUayOAh8/JpMR8kez96hmPDOu6VffWTwpwQK0jttEWInvcoIDCrIbIK9pd4WUUZ+ptNf1dBZQRp60DA8RNatjjcZ75kVS51HPOAFaOEPxDzPWmDha4qMzvgnFL/KLEkJx7R7XVFWUWtkxOJPqHZDN1s4fdBfsVx1BLVQa7Ak= 2021-04-07T05:49:49.728 INFO:teuthology.kill: smithi075.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyRxFGsu0KTPYbPlFcls4Z55E1/v5tfWfORr4EUr7+MGKl3sNkSBrhw5ZoS+F4EbhZ4VG/X3FtvDydRgj5w2DRv1o6mOr5BZ4Pa63zUXJ/XmHds6xyp+42xuhdDMoX8FtUY3vsxI8xmoYynczFf/vgVStLqNMhN7XYsEeztGpfSW3MeQmthAFBTrBG7QoejLR6ftNzC+tu6Sanbe9jHJmdyAAfyNIScbldNN5BF/B5ZHaCTtGJJQGX/D41+SdacRzCiZKfCEdhT1nOFA2BHmytl1zZ58D5f3KwXd9h3+YmCsybiLsyL7TNZDHj50XF8cDIZyPc/FszXXqda3KpMlCaI+dHZdNTu40zo7S5JTb7eQFhR8SWTQQlmNNv3CuoeKgp2iFn8+pgWE2eFQsMTTa0pDlki7N5koEr3x8k/w4wYj8d0vMHIEdC+NVKxHnzChH3QtGMdeIU2bCimwPyoWzNRyoVNZxI6bi5YPyZaJSIA048A7DrF4+M64uGeXH+6nE= 2021-04-07T05:49:49.729 INFO:teuthology.kill:2021-04-07 05:49:49,726.726 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-07T05:49:49.731 INFO:teuthology.kill:2021-04-07 05:49:49,730.730 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-07T05:49:49.828 INFO:teuthology.kill:2021-04-07 05:49:49,828.828 INFO:teuthology.orchestra.console:Power off smithi032 2021-04-07T05:49:49.847 INFO:teuthology.kill:2021-04-07 05:49:49,847.847 INFO:teuthology.orchestra.console:Power off smithi075 2021-04-07T05:49:58.221 INFO:teuthology.kill:2021-04-07 05:49:58,220.220 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-04-07T05:49:58.243 INFO:teuthology.kill:2021-04-07 05:49:58,243.243 INFO:teuthology.orchestra.console:Power off for smithi075 completed 2021-04-07T05:49:58.556 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-04-07T05:49:58.560 INFO:teuthology.task.internal:roles: ubuntu@smithi075.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-04-07T05:49:58.565 INFO:teuthology.misc:Compressing logs... 2021-04-07T05:49:58.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-07T05:50:58.736 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-07T05:50:58.854 INFO:teuthology.kill:No teuthology processes running 2021-04-07T05:51:01.028 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi075'] 2021-04-07T05:51:02.414 INFO:teuthology.kill:2021-04-07 05:51:02,413.413 INFO:teuthology.nuke:targets: 2021-04-07T05:51:02.414 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCITY3u4YLB1lwrnaAxFQ2PwT5drYrkmIYCtlFhk3WK3ic256/sYCsun5KPUUYeqduCy9mWp+w87B6feby9YifuDmNx+CbnivoM1SvcYFn+mOFRBTmJMfS41CH65JDQDPxkBssiobp5z8ZThRYwmzKTbykIpdZRVnQ2s1Ntvb48E+pdQlyNkNJ130qF4RZyEA+6+NhC2hkRiXrCAk+jvwBMOjtZo45SCduropkmYEFjM1H+yYzDFN+TXZLWZ305P+FyKYoo3yxj+2wjVipGVouvEYmhH0Wg0ChpLE7Q2TaKvI/q3zSJJIUyYfAd3dEFHT/dYBy76udPswnOPjo7UYoL0ELUayOAh8/JpMR8kez96hmPDOu6VffWTwpwQK0jttEWInvcoIDCrIbIK9pd4WUUZ+ptNf1dBZQRp60DA8RNatjjcZ75kVS51HPOAFaOEPxDzPWmDha4qMzvgnFL/KLEkJx7R7XVFWUWtkxOJPqHZDN1s4fdBfsVx1BLVQa7Ak= 2021-04-07T05:51:02.415 INFO:teuthology.kill: smithi075.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyRxFGsu0KTPYbPlFcls4Z55E1/v5tfWfORr4EUr7+MGKl3sNkSBrhw5ZoS+F4EbhZ4VG/X3FtvDydRgj5w2DRv1o6mOr5BZ4Pa63zUXJ/XmHds6xyp+42xuhdDMoX8FtUY3vsxI8xmoYynczFf/vgVStLqNMhN7XYsEeztGpfSW3MeQmthAFBTrBG7QoejLR6ftNzC+tu6Sanbe9jHJmdyAAfyNIScbldNN5BF/B5ZHaCTtGJJQGX/D41+SdacRzCiZKfCEdhT1nOFA2BHmytl1zZ58D5f3KwXd9h3+YmCsybiLsyL7TNZDHj50XF8cDIZyPc/FszXXqda3KpMlCaI+dHZdNTu40zo7S5JTb7eQFhR8SWTQQlmNNv3CuoeKgp2iFn8+pgWE2eFQsMTTa0pDlki7N5koEr3x8k/w4wYj8d0vMHIEdC+NVKxHnzChH3QtGMdeIU2bCimwPyoWzNRyoVNZxI6bi5YPyZaJSIA048A7DrF4+M64uGeXH+6nE= 2021-04-07T05:51:02.436 INFO:teuthology.kill:2021-04-07 05:51:02,436.436 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-07T05:51:02.453 INFO:teuthology.kill:2021-04-07 05:51:02,452.452 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-07T05:51:02.518 INFO:teuthology.kill:2021-04-07 05:51:02,518.518 INFO:teuthology.orchestra.console:Power off smithi032 2021-04-07T05:51:02.532 INFO:teuthology.kill:2021-04-07 05:51:02,532.532 INFO:teuthology.orchestra.console:Power off smithi075 2021-04-07T05:51:02.661 INFO:teuthology.kill:2021-04-07 05:51:02,660.660 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-04-07T05:51:02.672 INFO:teuthology.kill:2021-04-07 05:51:02,672.672 INFO:teuthology.orchestra.console:Power off for smithi075 completed 2021-04-07T05:51:02.816 INFO:teuthology.kill:2021-04-07 05:51:02,816.816 INFO:teuthology.lock.ops:unlocked smithi032.front.sepia.ceph.com 2021-04-07T05:51:02.833 INFO:teuthology.kill:2021-04-07 05:51:02,832.832 INFO:teuthology.lock.ops:unlocked smithi075.front.sepia.ceph.com 2021-04-07T05:53:03.158 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-07T05:53:03.339 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-07T05:53:03.344 INFO:teuthology.nuke:Checking targets against current locks 2021-04-07T05:53:03.548 INFO:teuthology.nuke:Not nuking smithi032.front.sepia.ceph.com because description doesn't match 2021-04-07T05:53:03.549 INFO:teuthology.nuke:Not nuking smithi075.front.sepia.ceph.com because description doesn't match