2021-10-30T04:37:17.005 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-30T04:37:17.127 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging 2021-10-30T04:37:17.128 INFO:teuthology.lock.ops:Updating [ubuntu@smithi097.front.sepia.ceph.com]: reset os type and version on server 2021-10-30T04:37:17.129 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2021-10-30T04:37:17.147 INFO:teuthology.lock.ops:Node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging is complete 2021-10-30T04:37:17.148 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging 2021-10-30T04:37:17.148 INFO:teuthology.lock.ops:Updating [ubuntu@smithi122.front.sepia.ceph.com]: reset os type and version on server 2021-10-30T04:37:17.149 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2021-10-30T04:37:17.168 INFO:teuthology.lock.ops:Node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging is complete 2021-10-30T04:37:17.368 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of centos 8.3 2021-10-30T04:37:17.377 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.3 2021-10-30T04:37:17.714 INFO:teuthology.orchestra.console:Power off smithi122 2021-10-30T04:37:17.753 INFO:teuthology.orchestra.console:Power off smithi097 2021-10-30T04:37:17.864 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2021-10-30T04:37:17.896 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-10-30T04:37:17.966 INFO:teuthology.orchestra.console:Power on smithi122 2021-10-30T04:37:17.997 INFO:teuthology.orchestra.console:Power on smithi097 2021-10-30T04:37:22.221 INFO:teuthology.orchestra.console:Power on for smithi122 completed 2021-10-30T04:37:22.322 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish 2021-10-30T04:37:26.354 INFO:teuthology.orchestra.console:Power on for smithi097 completed 2021-10-30T04:37:26.455 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish 2021-10-30T04:42:12.080 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-10-30T04:42:12.422 INFO:teuthology.provision.fog.smithi122:Node is ready 2021-10-30T04:42:12.438 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122.front.sepia.ceph.com 2021-10-30T04:42:12.494 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122 2021-10-30T04:42:12.813 INFO:teuthology.provision.fog.smithi122:Deploy complete! 2021-10-30T04:42:15.931 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-10-30T04:42:16.211 INFO:teuthology.provision.fog.smithi097:Node is ready 2021-10-30T04:42:16.226 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com 2021-10-30T04:42:16.282 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097 2021-10-30T04:42:16.606 INFO:teuthology.provision.fog.smithi097:Deploy complete! 2021-10-30T04:42:16.758 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com 2021-10-30T04:42:16.759 INFO:teuthology.lock.ops:New key found. Updating... 2021-10-30T04:42:16.781 INFO:teuthology.lock.ops:Checking smithi122.front.sepia.ceph.com 2021-10-30T04:42:16.781 INFO:teuthology.lock.ops:New key found. Updating... 2021-10-30T04:42:16.802 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server 2021-10-30T04:42:16.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2021-10-30T04:42:17.059 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2021-10-30T04:42:17.114 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Linux" 2021-10-30T04:42:17.115 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8" 2021-10-30T04:42:17.116 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2021-10-30T04:42:17.117 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2021-10-30T04:42:17.118 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2021-10-30T04:42:17.118 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2021-10-30T04:42:17.119 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Linux 8" 2021-10-30T04:42:17.119 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2021-10-30T04:42:17.120 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-30T04:42:17.120 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2021-10-30T04:42:17.120 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-30T04:42:17.121 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-30T04:42:17.121 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-30T04:42:17.123 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2021-10-30T04:42:17.141 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: set os type and version on server 2021-10-30T04:42:17.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2021-10-30T04:42:17.411 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2021-10-30T04:42:17.466 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Linux" 2021-10-30T04:42:17.466 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8" 2021-10-30T04:42:17.467 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2021-10-30T04:42:17.467 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2021-10-30T04:42:17.467 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8" 2021-10-30T04:42:17.468 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2021-10-30T04:42:17.468 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Linux 8" 2021-10-30T04:42:17.469 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2021-10-30T04:42:17.469 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-30T04:42:17.469 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2021-10-30T04:42:17.470 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-10-30T04:42:17.470 INFO:teuthology.orchestra.run.smithi122.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-10-30T04:42:17.471 INFO:teuthology.orchestra.run.smithi122.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-10-30T04:42:17.472 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2021-10-30T04:42:17.576 INFO:teuthology.dispatcher.supervisor:Running job 6467649 2021-10-30T04:42:17.577 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467649 --name yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467649/orig.config.yaml 2021-10-30T04:42:17.580 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467649 2021-10-30T04:42:17.581 INFO:teuthology.dispatcher.supervisor:Job PID: 27313 2021-10-30T04:42:17.581 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-10-30T16:43:12.932 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-10-30T16:43:13.112 INFO:teuthology.kill:Killing Pids: {27313} 2021-10-30T16:43:14.251 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-10-30T16:43:14.252 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-10-30T16:43:14.252 INFO:teuthology.misc:Compressing logs... 2021-10-30T16:43:14.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2021-10-30T16:43:14.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2021-10-30T16:43:14.761 INFO:teuthology.misc:Compressing logs... 2021-10-30T16:43:15.233 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-10-30T16:43:15.476 INFO:teuthology.misc:Compressing logs... 2021-10-30T16:43:36.894 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-10-30T16:43:43.099 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-mon.smithi122.log: file size changed while zipping 2021-10-30T16:43:44.540 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-mon.smithi097.log: file size changed while zipping 2021-10-30T16:44:07.072 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-10-30T16:44:26.110 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-10-30T16:44:42.608 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi097.ebblyr.log: file size changed while zipping 2021-10-30T16:45:02.468 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-10-30T16:45:23.270 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-10-30T16:46:06.850 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi097.nbzvna.log: file size changed while zipping 2021-10-30T16:46:15.199 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /var/log/ceph/7fbcb586-393d-11ec-8c28-001a4aab830c/ceph-mgr.smithi097.muowyc.log: file size changed while zipping 2021-10-30T16:47:08.505 INFO:teuthology.kill:No teuthology processes running 2021-10-30T16:47:09.695 INFO:teuthology.kill:Nuking machines: ['smithi097', 'smithi122'] 2021-10-30T16:47:10.688 INFO:teuthology.kill:2021-10-30 16:47:10,687.687 INFO:teuthology.nuke:targets: 2021-10-30T16:47:10.689 INFO:teuthology.kill: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGGpvG4uHIyg/8hnAevjIWVqU193q4mYl2HbGvSs6Rmg5yhQjFKZWq28LAfWQG8Lst/66aZueWXX6/iDq55wX6eBOjtgR9O9fmdISNUKjV42YrFM+Oq+w+wK5jTSZNVHy95pcZjI26Vel7JlzNvOp+KXqj1VU/46rgb3Aose1dUZv+f5NXzehN3xWlXRjhypgAO06BcRqonacTOb0xqpLoobqIoEfapFo5xu5+oCWiGF7pR+gG4/TygTxTmhC0ddPm0YVKx+1FRK1689x/X4sLH3K+xslo9vi9xfa/MSht5rOUKry74m9vrVVuAVf8morilLWnkwvJFqJI7bRyqpvKDTrrPuOk6Ews5CMTHdsI0LQbOmyBU55S972SupUj6ZM2BE/K4Mm5EaPn+mzuhE92Tim+rOyzKVOPerYfCRgWvVnuabqG0wIR4c719R9o+J+e4NhfB2Wg63n5cndnnMcrw2qAjP5EvnhargpHxjcTl3LqCQj6dFxANlPjqJzPk9k= 2021-10-30T16:47:10.689 INFO:teuthology.kill: smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRsvX8XuQ3jDE+RXmVadA08TRmYTsIYqlsp60jLHvaKSDlUrHDeH6BSqghkZreoyyKaZ2E4k7wUkd8LrpXaYjeauE2nweA78hX76I51e9Y24skeIKqCdYuNHprZNxKA5lZIno3Y53dXTM9n+zJU40qi3Q4j9n9RCwss1mzmaZKXkxJsJJcL6rIsKx/6CkhU8cKDgSyTNUWRdvX9eT8AHUGapYuxpPINBW++pc7U493wsmz6hX4ZTGDabkjN6MkTf2jkL6QSmnSd0LbgExvtLV/Oqw+Q6Ed8fIwbFhbHQ6JK4ta7welRF83bwqR2hWdKZMfshsfmOeFwcIRzQi8l8djrwP2KALbls1eYruDwoml2Xr9Lxdm8HXE/ovmGTH857os72uPdk0lBY4umEpV3j9DOIo/AhvkYGDHW41tSmIJ8lXomj0GVNjNVWDLAR20P+GfDBOGvzf0nXCDFfZFY0iqlsiBJLFSI1iNHyDediOOUVh6zSnyws+R+7lzHKb/VYM= 2021-10-30T16:47:10.717 INFO:teuthology.kill:2021-10-30 16:47:10,716.716 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-30T16:47:10.720 INFO:teuthology.kill:2021-10-30 16:47:10,720.720 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-30T16:47:10.808 INFO:teuthology.kill:2021-10-30 16:47:10,808.808 INFO:teuthology.orchestra.console:Power off smithi122 2021-10-30T16:47:10.826 INFO:teuthology.kill:2021-10-30 16:47:10,825.825 INFO:teuthology.orchestra.console:Power off smithi097 2021-10-30T16:47:19.194 INFO:teuthology.kill:2021-10-30 16:47:19,193.193 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-10-30T16:47:19.336 INFO:teuthology.kill:2021-10-30 16:47:19,335.335 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com 2021-10-30T16:47:23.296 INFO:teuthology.kill:2021-10-30 16:47:23,296.296 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2021-10-30T16:47:23.456 INFO:teuthology.kill:2021-10-30 16:47:23,456.456 INFO:teuthology.lock.ops:unlocked: smithi122.front.sepia.ceph.com 2021-10-30T16:49:23.773 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-10-30T16:49:23.916 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi097 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-10-29_05:17:03-krbd-pacific-testing-default-smithi/6466908 2021-10-30T16:49:23.917 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi122 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-10-29_05:17:03-krbd-pacific-testing-default-smithi/6466908