2021-11-05T15:59:34.767 INFO:root:teuthology version: 1.1.0-27954452 2021-11-05T15:59:34.898 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging 2021-11-05T15:59:34.899 INFO:teuthology.lock.ops:Updating [ubuntu@smithi013.front.sepia.ceph.com]: reset os type and version on server 2021-11-05T15:59:34.900 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server 2021-11-05T15:59:34.918 INFO:teuthology.lock.ops:Node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging is complete 2021-11-05T15:59:34.919 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging 2021-11-05T15:59:34.919 INFO:teuthology.lock.ops:Updating [ubuntu@smithi183.front.sepia.ceph.com]: reset os type and version on server 2021-11-05T15:59:34.919 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2021-11-05T15:59:34.937 INFO:teuthology.lock.ops:Node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging is complete 2021-11-05T15:59:35.132 INFO:teuthology.provision.fog.smithi013:Scheduling deploy of centos 8.3 2021-11-05T15:59:35.139 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of centos 8.3 2021-11-05T15:59:35.441 INFO:teuthology.orchestra.console:Power off smithi013 2021-11-05T15:59:35.454 INFO:teuthology.orchestra.console:Power off smithi183 2021-11-05T15:59:43.807 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-11-05T15:59:43.811 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2021-11-05T15:59:43.909 INFO:teuthology.orchestra.console:Power on smithi013 2021-11-05T15:59:43.916 INFO:teuthology.orchestra.console:Power on smithi183 2021-11-05T15:59:52.271 INFO:teuthology.orchestra.console:Power on for smithi013 completed 2021-11-05T15:59:52.276 INFO:teuthology.orchestra.console:Power on for smithi183 completed 2021-11-05T15:59:52.373 INFO:teuthology.provision.fog.smithi013:Waiting for deploy to finish 2021-11-05T15:59:52.377 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish 2021-11-05T16:03:33.205 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-05T16:03:48.571 INFO:teuthology.provision.fog.smithi013:Node is ready 2021-11-05T16:03:48.634 INFO:teuthology.orchestra.run.smithi013.stdout:smithi013.front.sepia.ceph.com 2021-11-05T16:03:48.694 INFO:teuthology.orchestra.run.smithi013.stdout:172.21.15.13 smithi013.front.sepia.ceph.com smithi013 2021-11-05T16:03:49.050 INFO:teuthology.provision.fog.smithi013:Deploy complete! 2021-11-05T16:03:55.448 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-05T16:04:15.670 INFO:teuthology.provision.fog.smithi183:Node is ready 2021-11-05T16:04:15.689 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183.front.sepia.ceph.com 2021-11-05T16:04:15.745 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.183 smithi183.front.sepia.ceph.com smithi183 2021-11-05T16:04:16.096 INFO:teuthology.provision.fog.smithi183:Deploy complete! 2021-11-05T16:04:16.244 INFO:teuthology.lock.ops:Checking smithi013.front.sepia.ceph.com 2021-11-05T16:04:16.245 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-05T16:04:16.263 INFO:teuthology.lock.ops:Checking smithi183.front.sepia.ceph.com 2021-11-05T16:04:16.263 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-05T16:04:16.288 INFO:teuthology.lock.ops:Updating [smithi013.front.sepia.ceph.com]: set os type and version on server 2021-11-05T16:04:16.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi013.front.sepia.ceph.com' 2021-11-05T16:04:16.567 INFO:teuthology.orchestra.run.smithi013.stdout:x86_64 2021-11-05T16:04:16.623 INFO:teuthology.orchestra.run.smithi013.stdout:NAME="CentOS Linux" 2021-11-05T16:04:16.623 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION="8" 2021-11-05T16:04:16.623 INFO:teuthology.orchestra.run.smithi013.stdout:ID="centos" 2021-11-05T16:04:16.623 INFO:teuthology.orchestra.run.smithi013.stdout:ID_LIKE="rhel fedora" 2021-11-05T16:04:16.623 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION_ID="8" 2021-11-05T16:04:16.624 INFO:teuthology.orchestra.run.smithi013.stdout:PLATFORM_ID="platform:el8" 2021-11-05T16:04:16.624 INFO:teuthology.orchestra.run.smithi013.stdout:PRETTY_NAME="CentOS Linux 8" 2021-11-05T16:04:16.624 INFO:teuthology.orchestra.run.smithi013.stdout:ANSI_COLOR="0;31" 2021-11-05T16:04:16.624 INFO:teuthology.orchestra.run.smithi013.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-05T16:04:16.625 INFO:teuthology.orchestra.run.smithi013.stdout:HOME_URL="https://centos.org/" 2021-11-05T16:04:16.625 INFO:teuthology.orchestra.run.smithi013.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-11-05T16:04:16.625 INFO:teuthology.orchestra.run.smithi013.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-11-05T16:04:16.626 INFO:teuthology.orchestra.run.smithi013.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-11-05T16:04:16.627 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server 2021-11-05T16:04:16.648 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: set os type and version on server 2021-11-05T16:04:16.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2021-11-05T16:04:16.851 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2021-11-05T16:04:16.905 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Linux" 2021-11-05T16:04:16.905 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="8" 2021-11-05T16:04:16.906 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2021-11-05T16:04:16.906 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2021-11-05T16:04:16.906 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="8" 2021-11-05T16:04:16.906 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el8" 2021-11-05T16:04:16.907 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Linux 8" 2021-11-05T16:04:16.907 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2021-11-05T16:04:16.907 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-05T16:04:16.907 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2021-11-05T16:04:16.907 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-11-05T16:04:16.908 INFO:teuthology.orchestra.run.smithi183.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-11-05T16:04:16.908 INFO:teuthology.orchestra.run.smithi183.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-11-05T16:04:16.908 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2021-11-05T16:04:17.005 INFO:teuthology.dispatcher.supervisor:Running job 6487031 2021-11-05T16:04:17.005 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-11-05_14:29:40-orch:cephadm-wip-sage-testing-2021-11-04-1303-distro-basic-smithi/6487031 --name sage-2021-11-05_14:29:40-orch:cephadm-wip-sage-testing-2021-11-04-1303-distro-basic-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/sage-2021-11-05_14:29:40-orch:cephadm-wip-sage-testing-2021-11-04-1303-distro-basic-smithi/6487031/orig.config.yaml 2021-11-05T16:04:17.009 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-11-05_14:29:40-orch:cephadm-wip-sage-testing-2021-11-04-1303-distro-basic-smithi/6487031 2021-11-05T16:04:17.010 INFO:teuthology.dispatcher.supervisor:Job PID: 2496 2021-11-05T16:04:17.010 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-06T04:05:08.315 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-06T04:05:08.518 INFO:teuthology.kill:Killing Pids: {2496} 2021-11-06T04:05:09.630 INFO:teuthology.task.internal:roles: ubuntu@smithi013.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-06T04:05:09.631 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-06T04:05:09.632 INFO:teuthology.misc:Compressing logs... 2021-11-06T04:05:09.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi013.front.sepia.ceph.com' 2021-11-06T04:05:09.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2021-11-06T04:05:10.161 INFO:teuthology.misc:Compressing logs... 2021-11-06T04:05:10.449 INFO:teuthology.misc:Compressing logs... 2021-11-06T04:05:10.963 INFO:teuthology.orchestra.run.smithi013.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-mgr.smithi013.dwsyzd.log: file size changed while zipping 2021-11-06T04:05:11.117 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2021-11-06T04:05:11.684 INFO:teuthology.orchestra.run.smithi013.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2021-11-06T04:05:11.890 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2021-11-06T04:05:12.480 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2021-11-06T04:05:12.802 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-mon.smithi183.log: file size changed while zipping 2021-11-06T04:05:12.831 INFO:teuthology.orchestra.run.smithi013.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-mon.smithi013.log: file size changed while zipping 2021-11-06T04:05:13.485 INFO:teuthology.orchestra.run.smithi013.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2021-11-06T04:05:14.254 INFO:teuthology.orchestra.run.smithi013.stderr:gzip: /var/log/ceph/b45c6c26-3e53-11ec-8c28-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2021-11-06T04:05:49.762 INFO:teuthology.kill:No teuthology processes running 2021-11-06T04:05:50.819 INFO:teuthology.kill:Nuking machines: ['smithi013', 'smithi183'] 2021-11-06T04:05:51.742 INFO:teuthology.kill:2021-11-06 04:05:51,741.741 INFO:teuthology.nuke:targets: 2021-11-06T04:05:51.743 INFO:teuthology.kill: smithi013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAU/zIEl1jUxfKLTPKp86hGPKG8365iz0mf3Vd7Su6RdmblGPWmtw+9sVnS7sf10U7+em1vcB9izGFhIu5KxW6EPRmALmwHAm5VWpupE6zF6O3sjbe0V+UyjKV4nypPgrfn9lS6USR/AEY4f8m+i7+sc2u282qjBX8H91Ea2ynl5U68svXMpYdg02xiqUs/7d0nLk+yvMbETdqNLzy3UCoJkqKK/x+AM/Rf8nJQo/yXS40M5PwgIbmQaCT1gKJzhlZ5mcM4bP64brPvsMe4CX4eJH3X8HUCAokA5LpJ3NnN7QcJ/FHv+GPvpvgxbk7NJhvfRiVEGcGVFGtcOFLqEjKX4Z9/qcGj6ShauKheIK5VmQo8Hk+oWmsZvSTMtntgfR5nNqHwvZjyv0PlLAhFbka1L3Gu6wErBZHAEujuB//JVo+NDE9sz2h1/JEjg5ZUPX2Ye9r4Y+qCIxustVgXOl+vk53dsqeBRTXzOn71TOeGvZllFOtkqEY9UJ0GqonC6s= 2021-11-06T04:05:51.743 INFO:teuthology.kill: smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCToteVga2H8Xbj+tMO4H1Wx2GN84tdV8UnqPbG/5qdAiFmDJJneOzZkNTkGgoRVN8lqVefqje4SAO++AUlIDIcKzIBl1WuhIh9ERD/ZxzzTaQkTmSpafFM4xvQvF4gunJkY94T4gIjWvj46TaUmH9DXQxGTlXAZLhhinDhto2OCbdaWCorWliIYMEpazpYDGRB6RomagVDv9JNPjGSKW1h12Nqry/rAdDyhGngR0gdKpJr20B18XgzulipN1CdSnWITUPU0NGNGzksZUDnEODPk8aoa9O4YKlzgRK5+LnBGR3YtRUdAESR2k2q1NGnGcZ6+wsKZbZPzL7SRaVisZ/4g6oOURe+hUXCBS5IZ6O93YrKd5qLPbD279iKdtG9UVAKRgLzaOrggiBy/l+kqQH4euj/NmCVSboZMdaA2+FKUzwu4NETxMhF56+xAXVciw7Grm7Z+8u95TTG7YuZG6RkmUeYC6gKNRLwGPaC3wmTdM8kzTC2im8BI0FwF0fifE= 2021-11-06T04:05:51.768 INFO:teuthology.kill:2021-11-06 04:05:51,768.768 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-06T04:05:51.772 INFO:teuthology.kill:2021-11-06 04:05:51,772.772 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-06T04:05:51.835 INFO:teuthology.kill:2021-11-06 04:05:51,834.834 INFO:teuthology.orchestra.console:Power off smithi013 2021-11-06T04:05:51.848 INFO:teuthology.kill:2021-11-06 04:05:51,847.847 INFO:teuthology.orchestra.console:Power off smithi183 2021-11-06T04:06:00.203 INFO:teuthology.kill:2021-11-06 04:06:00,203.203 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-11-06T04:06:00.207 INFO:teuthology.kill:2021-11-06 04:06:00,207.207 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2021-11-06T04:06:00.354 INFO:teuthology.kill:2021-11-06 04:06:00,353.353 INFO:teuthology.lock.ops:unlocked: smithi013.front.sepia.ceph.com 2021-11-06T04:06:00.356 INFO:teuthology.kill:2021-11-06 04:06:00,355.355 INFO:teuthology.lock.ops:unlocked: smithi183.front.sepia.ceph.com 2021-11-06T04:08:00.643 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-06T04:08:00.756 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi013 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-04_20:25:46-rados-wip-yuri-testing-2021-11-04-0731-distro-basic-smithi/6485666 2021-11-06T04:08:00.756 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi183 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-04_20:25:46-rados-wip-yuri-testing-2021-11-04-0731-distro-basic-smithi/6485668