2023-01-23T17:59:28.344 INFO:root:teuthology version: 0.0.1.dev20+g5abecab
2023-01-23T17:59:28.462 INFO:teuthology.lock.ops:Start node 'smithi083.front.sepia.ceph.com' reimaging
2023-01-23T17:59:28.463 INFO:teuthology.lock.ops:Updating [smithi083.front.sepia.ceph.com]: reset os type and version on server
2023-01-23T17:59:28.463 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server
2023-01-23T17:59:28.478 INFO:teuthology.lock.ops:Node 'smithi083.front.sepia.ceph.com' reimaging is complete
2023-01-23T17:59:28.479 INFO:teuthology.lock.ops:Start node 'smithi088.front.sepia.ceph.com' reimaging
2023-01-23T17:59:28.479 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: reset os type and version on server
2023-01-23T17:59:28.480 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server
2023-01-23T17:59:28.495 INFO:teuthology.lock.ops:Node 'smithi088.front.sepia.ceph.com' reimaging is complete
2023-01-23T17:59:28.622 INFO:teuthology.provision.fog.smithi083:Scheduling deploy of centos 8.stream
2023-01-23T17:59:28.648 INFO:teuthology.provision.fog.smithi088:Scheduling deploy of centos 8.stream
2023-01-23T17:59:28.874 INFO:teuthology.orchestra.console:Power off smithi083
2023-01-23T17:59:28.888 INFO:teuthology.orchestra.console:Power off smithi088
2023-01-23T17:59:37.249 INFO:teuthology.orchestra.console:Power off for smithi088 completed
2023-01-23T17:59:37.350 INFO:teuthology.orchestra.console:Power on smithi088
2023-01-23T17:59:41.353 INFO:teuthology.orchestra.console:Power off for smithi083 completed
2023-01-23T17:59:41.454 INFO:teuthology.orchestra.console:Power on smithi083
2023-01-23T17:59:45.716 INFO:teuthology.orchestra.console:Power on for smithi083 completed
2023-01-23T17:59:45.737 INFO:teuthology.orchestra.console:Power on for smithi088 completed
2023-01-23T17:59:45.818 INFO:teuthology.provision.fog.smithi083:Waiting for deploy to finish
2023-01-23T17:59:45.839 INFO:teuthology.provision.fog.smithi088:Waiting for deploy to finish
2023-01-23T18:02:50.348 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-01-23T18:02:59.432 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-01-23T18:03:05.452 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:03:08.492 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-01-23T18:03:14.536 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:03:17.576 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-01-23T18:03:23.596 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:03:32.680 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:03:41.740 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:03:50.825 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:04:15.336 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:04:23.584 WARNING:teuthology.provision.fog:timed out
2023-01-23T18:04:24.396 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-01-23T18:04:29.945 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-23T18:04:30.200 INFO:teuthology.provision.fog.smithi083:Node is ready
2023-01-23T18:04:30.259 INFO:teuthology.orchestra.run.smithi083.stdout:smithi083.front.sepia.ceph.com
2023-01-23T18:04:30.318 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83 smithi083.front.sepia.ceph.com smithi083
2023-01-23T18:04:30.668 INFO:teuthology.provision.fog.smithi083:Deploy complete!
2023-01-23T18:04:30.835 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-23T18:04:51.079 INFO:teuthology.provision.fog.smithi088:Node is ready
2023-01-23T18:04:51.101 INFO:teuthology.orchestra.run.smithi088.stdout:smithi001
2023-01-23T18:04:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.1 smithi001 smithi001
2023-01-23T18:04:51.535 INFO:teuthology.provision.fog.smithi088:Deploy complete!
2023-01-23T18:04:51.735 INFO:teuthology.lock.ops:Checking smithi083.front.sepia.ceph.com
2023-01-23T18:04:51.736 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-23T18:04:51.757 INFO:teuthology.lock.ops:Checking smithi088.front.sepia.ceph.com
2023-01-23T18:04:51.758 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-23T18:04:51.773 INFO:teuthology.lock.ops:Updating [smithi083.front.sepia.ceph.com]: set os type and version on server
2023-01-23T18:04:51.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2023-01-23T18:04:52.033 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64
2023-01-23T18:04:52.090 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream"
2023-01-23T18:04:52.091 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="8"
2023-01-23T18:04:52.091 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos"
2023-01-23T18:04:52.091 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora"
2023-01-23T18:04:52.092 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="8"
2023-01-23T18:04:52.092 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el8"
2023-01-23T18:04:52.092 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 8"
2023-01-23T18:04:52.092 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31"
2023-01-23T18:04:52.092 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-01-23T18:04:52.093 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/"
2023-01-23T18:04:52.093 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-23T18:04:52.093 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-23T18:04:52.093 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-01-23T18:04:52.094 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server
2023-01-23T18:04:52.112 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: set os type and version on server
2023-01-23T18:04:52.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com'
2023-01-23T18:04:52.308 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64
2023-01-23T18:04:52.365 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream"
2023-01-23T18:04:52.365 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8"
2023-01-23T18:04:52.366 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos"
2023-01-23T18:04:52.366 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora"
2023-01-23T18:04:52.366 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8"
2023-01-23T18:04:52.366 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8"
2023-01-23T18:04:52.366 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8"
2023-01-23T18:04:52.367 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31"
2023-01-23T18:04:52.367 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-01-23T18:04:52.367 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/"
2023-01-23T18:04:52.368 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-23T18:04:52.368 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-23T18:04:52.368 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-01-23T18:04:52.369 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server
2023-01-23T18:04:52.457 INFO:teuthology.dispatcher.supervisor:Running job 7133820
2023-01-23T18:04:52.458 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-23_17:16:25-rados-wip-yuri6-testing-2023-01-22-0750-distro-default-smithi/7133820 --name yuriw-2023-01-23_17:16:25-rados-wip-yuri6-testing-2023-01-22-0750-distro-default-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} -- /home/teuthworker/archive/yuriw-2023-01-23_17:16:25-rados-wip-yuri6-testing-2023-01-22-0750-distro-default-smithi/7133820/orig.config.yaml
2023-01-23T18:04:52.461 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-01-23_17:16:25-rados-wip-yuri6-testing-2023-01-22-0750-distro-default-smithi/7133820
2023-01-23T18:04:52.462 INFO:teuthology.dispatcher.supervisor:Job PID: 1453
2023-01-23T18:04:52.462 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-01-24T06:05:45.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-24T06:05:45.764 INFO:teuthology.kill:Killing Pids: {1453}
2023-01-24T06:05:46.626 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2023-01-24T06:05:46.627 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2023-01-24T06:05:46.628 INFO:teuthology.misc:Compressing logs...
2023-01-24T06:05:46.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2023-01-24T06:05:46.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com'
2023-01-24T06:05:47.102 INFO:teuthology.misc:Compressing logs...
2023-01-24T06:05:47.613 INFO:teuthology.orchestra.run.smithi088.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-01-24T06:05:48.897 INFO:teuthology.orchestra.run.smithi083.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-01-24T06:05:49.555 INFO:teuthology.misc:Compressing logs...
2023-01-24T06:06:09.978 INFO:teuthology.kill:No teuthology processes running
2023-01-24T06:06:10.691 INFO:teuthology.kill:Nuking machines: ['smithi083', 'smithi088']
2023-01-24T06:06:11.301 INFO:teuthology.kill:2023-01-24 06:06:11,301.301 INFO:teuthology.nuke:targets:
2023-01-24T06:06:11.302 INFO:teuthology.kill:  smithi083.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/ThQ8KnCiGZql47KGFOSDZoePn/WVZWItpihzYnAqcbTlFiiDK2Py9HlpHMmnIvVFkG8AfIUBj40f9vNcQeiywdLyfWijSKx65R6HPvvQZ69C/yZ1RZkWX3U15yVdMC0XV3Xu/L4BMM2e8+OCOlfWov50w7HJ/eCFnS/VSzrmBAurSC2epHu1LtNWmdsfm/TjhFhVDauFs30FhQ3dIKfjmZbFOhmReukg5SX5LyrBPhQl16KRi2b/StVb8K0bTWTTBACprK2YXa+eevha8vcERj9yDcOUpiiGjvenSnIzzcbNUo/D70nSv9ekiKOWXkKmbQQ+67u0TToRijPpKPPI262bVdXBesKh693DG4EPsLEpV04RpN83FcoiSLWsIPiwG+WQt25fWd1ed27eQitySDzZ2wm4Bk8BiGIxSa/Up12ImmJvKYiWrr+kfgi7BVrqkD6xrVHwg2fKBc9vGRxq82nG4361eTXPrBcwCGQpkUdgAdYqT1OWp+8aCIa3+Uc=
2023-01-24T06:06:11.302 INFO:teuthology.kill:  smithi088.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCYO16YaRRWhGc4QdfxQyL7HBAtRn++Q/ZgkW/Wgd6T02nX2WKwyRgUhsmqd41YrfkIrpfsZ6YDDSuNw+3kpxq+bzhNOFi5BNK2RrKXUqy1TJVX/5gyz32fK16N3SlYTK59EWl4NoYb+dKg6TYi8RkGSgM00C1jSluhVukXv1CO3xOev+Z45qnAIBFJNUrFJ15bJQPPaGVUiBNRhq1YLrp0F2/5lKtBWhjDJHfhy3JBXClDhhHcFzFiJkise4GE7xyR3lC7zsaUctSFw/pKr+5Nd3FnZJJVIXnKWdhEiw3vJLKKW7perwQSQxrgCBqQUxsi8EgSMxXNYKypaDzknUfyCSjp1mVP4fVJ2SxZNHMHvTU2IBoIqxni702Fz6SQWJvL/EKY67eiYdmZWAE1ppluyPD0CPvVRJDtsaYkNVnr8SGsDJplSPycAstSAjYnIBnp+5D25XV0pphb7aeEvjw9I+2Eqk9d2+Oj8XdhSmeEQ22culwu0y8b0cmWHeqz0y0=
2023-01-24T06:06:11.303 INFO:teuthology.kill:2023-01-24 06:06:11,301.301 INFO:teuthology.nuke:Checking targets against current locks
2023-01-24T06:06:11.348 INFO:teuthology.kill:2023-01-24 06:06:11,347.347 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-24T06:06:11.374 INFO:teuthology.kill:2023-01-24 06:06:11,374.374 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-24T06:06:11.419 INFO:teuthology.kill:2023-01-24 06:06:11,418.418 INFO:teuthology.orchestra.console:Power off smithi083
2023-01-24T06:06:11.452 INFO:teuthology.kill:2023-01-24 06:06:11,451.451 INFO:teuthology.orchestra.console:Power off smithi088
2023-01-24T06:06:19.786 INFO:teuthology.kill:2023-01-24 06:06:19,785.785 INFO:teuthology.orchestra.console:Power off for smithi083 completed
2023-01-24T06:06:19.942 INFO:teuthology.kill:2023-01-24 06:06:19,942.942 INFO:teuthology.lock.ops:unlocked: smithi083.front.sepia.ceph.com
2023-01-24T06:06:23.921 INFO:teuthology.kill:2023-01-24 06:06:23,920.920 INFO:teuthology.orchestra.console:Power off for smithi088 completed
2023-01-24T06:06:24.079 INFO:teuthology.kill:2023-01-24 06:06:24,079.079 INFO:teuthology.lock.ops:unlocked: smithi088.front.sepia.ceph.com
2023-01-24T06:08:24.344 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-01-24T06:08:24.458 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi083 but it was locked by another job: /home/teuthworker/archive/lflores-2023-01-23_23:00:46-upgrade-main-distro-default-smithi/7134676
2023-01-24T06:08:24.459 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi088 but it was locked by another job: /home/teuthworker/archive/lflores-2023-01-23_23:00:46-upgrade-main-distro-default-smithi/7134676