2021-12-07T14:06:19.287 INFO:root:teuthology version: 1.1.0-844d1377 2021-12-07T14:06:19.414 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging 2021-12-07T14:06:19.415 INFO:teuthology.lock.ops:Updating [ubuntu@smithi183.front.sepia.ceph.com]: reset os type and version on server 2021-12-07T14:06:19.415 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2021-12-07T14:06:19.434 INFO:teuthology.lock.ops:Node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging is complete 2021-12-07T14:06:19.434 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi190.front.sepia.ceph.com' reimaging 2021-12-07T14:06:19.434 INFO:teuthology.lock.ops:Updating [ubuntu@smithi190.front.sepia.ceph.com]: reset os type and version on server 2021-12-07T14:06:19.435 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2021-12-07T14:06:19.455 INFO:teuthology.lock.ops:Node 'ubuntu@smithi190.front.sepia.ceph.com' reimaging is complete 2021-12-07T14:06:19.671 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of centos 8.stream 2021-12-07T14:06:19.692 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of centos 8.stream 2021-12-07T14:06:20.051 INFO:teuthology.orchestra.console:Power off smithi190 2021-12-07T14:06:20.067 INFO:teuthology.orchestra.console:Power off smithi183 2021-12-07T14:06:28.431 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2021-12-07T14:06:28.532 INFO:teuthology.orchestra.console:Power on smithi183 2021-12-07T14:06:32.536 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2021-12-07T14:06:32.638 INFO:teuthology.orchestra.console:Power on smithi190 2021-12-07T14:06:36.895 INFO:teuthology.orchestra.console:Power on for smithi190 completed 2021-12-07T14:06:36.907 INFO:teuthology.orchestra.console:Power on for smithi183 completed 2021-12-07T14:06:36.996 INFO:teuthology.provision.fog.smithi190:Waiting for deploy to finish 2021-12-07T14:06:37.008 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish 2021-12-07T14:11:26.980 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-07T14:11:27.102 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-07T14:11:27.239 INFO:teuthology.provision.fog.smithi190:Node is ready 2021-12-07T14:11:27.257 INFO:teuthology.orchestra.run.smithi190.stdout:smithi190.front.sepia.ceph.com 2021-12-07T14:11:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.15.190 smithi190.front.sepia.ceph.com smithi190 2021-12-07T14:11:27.394 INFO:teuthology.provision.fog.smithi183:Node is ready 2021-12-07T14:11:27.412 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183.front.sepia.ceph.com 2021-12-07T14:11:27.469 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.183 smithi183.front.sepia.ceph.com smithi183 2021-12-07T14:11:27.649 INFO:teuthology.provision.fog.smithi190:Deploy complete! 2021-12-07T14:11:27.795 INFO:teuthology.provision.fog.smithi183:Deploy complete! 2021-12-07T14:11:27.989 INFO:teuthology.lock.ops:Checking smithi183.front.sepia.ceph.com 2021-12-07T14:11:27.990 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-07T14:11:28.010 INFO:teuthology.lock.ops:Checking smithi190.front.sepia.ceph.com 2021-12-07T14:11:28.010 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-07T14:11:28.033 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: set os type and version on server 2021-12-07T14:11:28.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2021-12-07T14:11:28.225 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2021-12-07T14:11:28.281 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2021-12-07T14:11:28.281 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="8" 2021-12-07T14:11:28.281 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2021-12-07T14:11:28.282 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2021-12-07T14:11:28.282 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="8" 2021-12-07T14:11:28.282 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el8" 2021-12-07T14:11:28.283 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-07T14:11:28.283 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2021-12-07T14:11:28.283 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-07T14:11:28.283 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2021-12-07T14:11:28.284 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-07T14:11:28.284 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-07T14:11:28.284 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-07T14:11:28.285 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2021-12-07T14:11:28.304 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: set os type and version on server 2021-12-07T14:11:28.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2021-12-07T14:11:28.502 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2021-12-07T14:11:28.560 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2021-12-07T14:11:28.560 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="8" 2021-12-07T14:11:28.560 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2021-12-07T14:11:28.561 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2021-12-07T14:11:28.561 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="8" 2021-12-07T14:11:28.561 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el8" 2021-12-07T14:11:28.561 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-07T14:11:28.562 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2021-12-07T14:11:28.562 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-07T14:11:28.563 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2021-12-07T14:11:28.563 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-07T14:11:28.563 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-07T14:11:28.563 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-07T14:11:28.564 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2021-12-07T14:11:28.685 INFO:teuthology.dispatcher.supervisor:Running job 6548042 2021-12-07T14:11:28.685 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-12-06_16:41:40-rados-WIP_GBH_safe_fast_shutdown_2-distro-basic-smithi/6548042 --name benhanokh-2021-12-06_16:41:40-rados-WIP_GBH_safe_fast_shutdown_2-distro-basic-smithi --description rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} -- /home/teuthworker/archive/benhanokh-2021-12-06_16:41:40-rados-WIP_GBH_safe_fast_shutdown_2-distro-basic-smithi/6548042/orig.config.yaml 2021-12-07T14:11:28.690 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-12-06_16:41:40-rados-WIP_GBH_safe_fast_shutdown_2-distro-basic-smithi/6548042 2021-12-07T14:11:28.691 INFO:teuthology.dispatcher.supervisor:Job PID: 8937 2021-12-07T14:11:28.692 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-08T02:12:23.439 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-08T02:12:23.638 INFO:teuthology.kill:Killing Pids: {8937} 2021-12-08T02:12:24.805 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-12-08T02:12:24.806 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-12-08T02:12:24.806 INFO:teuthology.misc:Compressing logs... 2021-12-08T02:12:24.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2021-12-08T02:12:24.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2021-12-08T02:12:25.294 INFO:teuthology.misc:Compressing logs... 2021-12-08T02:12:26.106 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-08T02:12:26.322 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-08T02:12:34.182 INFO:teuthology.misc:Compressing logs... 2021-12-08T02:13:19.387 INFO:teuthology.kill:No teuthology processes running 2021-12-08T02:13:20.579 INFO:teuthology.kill:Nuking machines: ['smithi183', 'smithi190'] 2021-12-08T02:13:21.558 INFO:teuthology.kill:2021-12-08 02:13:21,558.558 INFO:teuthology.nuke:targets: 2021-12-08T02:13:21.559 INFO:teuthology.kill: smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDUGev4zgH/NU9WjDveWWyRu0WgkCxVpIwxLcoWe6QHQU2OXPsglsvkciuK6TDOifWghG7FEYwSejN0lTAEBMkkc0tWnxivm5IYQTiY68/6amQU+7ps8uqBkfw4oDkU4dkEWEuss2+vYQkvM2A8VI6KMfBjTzHGoIFDUggjEBdPe8DF9c+cm3A/94Kov1ew7gPF+s3FyoDwOhpGs4t9+thhc8LHKT910vOmo2pWSbS1WveHmULs9MgpZvW/4pg0vrUrsTwmb1NewX0jZGRHDDz6VwnCbd4Vg+8ix7MTHz1o6paPuELeV9Z0BPu9YqE25t+lKQ8+PbmBQ+QXqR/mR9HGz3V8NnPIWKRTe3bdKdQmMWRgN/YkJi882FMEB2kLl3eia8mNH9ZgVs6X2KSJhpZfhgG/1Ok9t+YCGhuoJ2Jvhw48UkL6TBkZTdtQtChdakgcBdBsXxl5/iTJDMWHSGMiKqdVatzH8Uncs4ETwzAyraQwRjtUuXRkhVrqlTJ/bE= 2021-12-08T02:13:21.560 INFO:teuthology.kill: smithi190.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlUf7M+pCZNCt+nK0NFf/Nf4dcXbzISP604OLyICUwrGDsrlruM2Dtpf5H6lSjn8mhGFFwLy6JgVjzQvCA+FVBm452OXYlUKPMlYZhkRlLj5Lvt71EqQJMl4qXk2MOWH1DC3Qpc7hxu8MljFk8wHUqj1wPo5ibme6g19ADFPNdHWkWd9UtbsmyW0YtOyU5j71wWYjwe6raYce5JgnJhZhS+JSSo+/VJ5s0N5znMLwNtplXbGQBLRgqBOtXMuIBnniIqHgAyyS5xoQxHxbjIuAUU/9K4711sTiYCBBM9J66UQ6qD8Y70IQMxANq83DsaMkRFl4UndFmmKNVUM1NZ1BGxpEnSyedPoCp2CPkpTP4wOk/sOuFA7ksrqDai2FYJ+mAL5ypM4UmX0WAa3MckMSHWr7Gj8n1zCN0IqJ83IM4pqdQZZymdoPVQtlIIeM25vWxcK409OlxWvPDamb8HNmGlDq0a2jxMOdRk580Mj5eFTSjM65jbFWcpx4Jpn1mdzc= 2021-12-08T02:13:21.586 INFO:teuthology.kill:2021-12-08 02:13:21,585.585 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-08T02:13:21.590 INFO:teuthology.kill:2021-12-08 02:13:21,590.590 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-08T02:13:21.669 INFO:teuthology.kill:2021-12-08 02:13:21,668.668 INFO:teuthology.orchestra.console:Power off smithi183 2021-12-08T02:13:21.683 INFO:teuthology.kill:2021-12-08 02:13:21,683.683 INFO:teuthology.orchestra.console:Power off smithi190 2021-12-08T02:13:30.047 INFO:teuthology.kill:2021-12-08 02:13:30,046.046 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2021-12-08T02:13:30.052 INFO:teuthology.kill:2021-12-08 02:13:30,052.052 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2021-12-08T02:13:30.208 INFO:teuthology.kill:2021-12-08 02:13:30,207.207 INFO:teuthology.lock.ops:unlocked: smithi190.front.sepia.ceph.com 2021-12-08T02:13:30.210 INFO:teuthology.kill:2021-12-08 02:13:30,210.210 INFO:teuthology.lock.ops:unlocked: smithi183.front.sepia.ceph.com 2021-12-08T02:15:30.553 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-12-08T02:15:30.662 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi183 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-12-07_16:02:55-rados-wip-yuri11-testing-2021-12-06-1619-distro-default-smithi/6551050 2021-12-08T02:15:30.663 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi190 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-12-07_16:02:55-rados-wip-yuri11-testing-2021-12-06-1619-distro-default-smithi/6551050