2022-07-04T18:05:32.050 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-04T18:05:32.194 INFO:teuthology.lock.ops:Start node 'smithi057.front.sepia.ceph.com' reimaging 2022-07-04T18:05:32.195 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: reset os type and version on server 2022-07-04T18:05:32.196 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2022-07-04T18:05:32.218 INFO:teuthology.lock.ops:Node 'smithi057.front.sepia.ceph.com' reimaging is complete 2022-07-04T18:05:32.219 INFO:teuthology.lock.ops:Start node 'smithi156.front.sepia.ceph.com' reimaging 2022-07-04T18:05:32.219 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: reset os type and version on server 2022-07-04T18:05:32.219 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2022-07-04T18:05:32.243 INFO:teuthology.lock.ops:Node 'smithi156.front.sepia.ceph.com' reimaging is complete 2022-07-04T18:05:32.428 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of centos 8.stream 2022-07-04T18:05:32.456 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of centos 8.stream 2022-07-04T18:05:32.774 INFO:teuthology.orchestra.console:Power off smithi057 2022-07-04T18:05:32.821 INFO:teuthology.orchestra.console:Power off smithi156 2022-07-04T18:05:41.153 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2022-07-04T18:05:41.184 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2022-07-04T18:05:41.255 INFO:teuthology.orchestra.console:Power on smithi057 2022-07-04T18:05:41.292 INFO:teuthology.orchestra.console:Power on smithi156 2022-07-04T18:05:49.638 INFO:teuthology.orchestra.console:Power on for smithi057 completed 2022-07-04T18:05:49.654 INFO:teuthology.orchestra.console:Power on for smithi156 completed 2022-07-04T18:05:49.740 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish 2022-07-04T18:05:49.755 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish 2022-07-04T18:09:56.269 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-04T18:10:11.342 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-04T18:10:16.980 INFO:teuthology.provision.fog.smithi057:Node is ready 2022-07-04T18:10:17.005 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com 2022-07-04T18:10:17.067 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057 2022-07-04T18:10:17.438 INFO:teuthology.provision.fog.smithi057:Deploy complete! 2022-07-04T18:10:36.481 INFO:teuthology.provision.fog.smithi156:Node is ready 2022-07-04T18:10:36.505 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156.front.sepia.ceph.com 2022-07-04T18:10:36.566 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.15.156 smithi156.front.sepia.ceph.com smithi156 2022-07-04T18:10:36.945 INFO:teuthology.provision.fog.smithi156:Deploy complete! 2022-07-04T18:10:37.128 INFO:teuthology.lock.ops:Checking smithi057.front.sepia.ceph.com 2022-07-04T18:10:37.129 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-04T18:10:37.155 INFO:teuthology.lock.ops:Checking smithi156.front.sepia.ceph.com 2022-07-04T18:10:37.155 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-04T18:10:37.177 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: set os type and version on server 2022-07-04T18:10:37.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-07-04T18:10:37.445 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2022-07-04T18:10:37.504 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2022-07-04T18:10:37.505 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8" 2022-07-04T18:10:37.505 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2022-07-04T18:10:37.505 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2022-07-04T18:10:37.506 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8" 2022-07-04T18:10:37.506 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8" 2022-07-04T18:10:37.506 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-04T18:10:37.507 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2022-07-04T18:10:37.507 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-04T18:10:37.507 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2022-07-04T18:10:37.507 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-04T18:10:37.507 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-04T18:10:37.508 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-04T18:10:37.509 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2022-07-04T18:10:37.532 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: set os type and version on server 2022-07-04T18:10:37.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2022-07-04T18:10:37.739 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2022-07-04T18:10:37.797 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2022-07-04T18:10:37.797 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8" 2022-07-04T18:10:37.798 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2022-07-04T18:10:37.798 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2022-07-04T18:10:37.798 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8" 2022-07-04T18:10:37.798 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8" 2022-07-04T18:10:37.799 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-04T18:10:37.799 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2022-07-04T18:10:37.799 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-04T18:10:37.799 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2022-07-04T18:10:37.799 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-04T18:10:37.800 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-04T18:10:37.800 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-04T18:10:37.800 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2022-07-04T18:10:37.923 INFO:teuthology.dispatcher.supervisor:Running job 6912676 2022-07-04T18:10:37.924 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/bin/teuthology -v --owner scheduled_dparmar@teuthology --archive /home/teuthworker/archive/dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi/6912676 --name dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi --description fs:upgrade/upgrade_without_reducing_max_mds/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi/6912676/orig.config.yaml 2022-07-04T18:10:37.928 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi/6912676 2022-07-04T18:10:37.929 INFO:teuthology.dispatcher.supervisor:Job PID: 18315 2022-07-04T18:10:37.930 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-07-05T06:11:35.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-07-05T06:11:36.056 INFO:teuthology.kill:Killing Pids: {18315} 2022-07-05T06:11:37.702 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-07-05T06:11:37.703 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-07-05T06:11:37.704 INFO:teuthology.misc:Compressing logs... 2022-07-05T06:11:37.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2022-07-05T06:11:37.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2022-07-05T06:11:38.253 INFO:teuthology.misc:Compressing logs... 2022-07-05T06:11:38.622 INFO:teuthology.misc:Compressing logs... 2022-07-05T06:11:41.532 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-07-05T06:11:43.512 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-mon.smithi057.log: file size changed while zipping 2022-07-05T06:11:44.296 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-07-05T06:11:46.297 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-mon.smithi156.log: file size changed while zipping 2022-07-05T06:11:46.357 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-07-05T06:11:48.923 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-07-05T06:11:49.949 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-07-05T06:11:51.891 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-07-05T06:11:52.991 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/8d6b1f92-fbc5-11ec-842c-001a4aab830c/ceph-mgr.smithi057.tlxyla.log: file size changed while zipping 2022-07-05T06:12:44.432 INFO:teuthology.kill:No teuthology processes running 2022-07-05T06:12:46.087 INFO:teuthology.kill:Nuking machines: ['smithi057', 'smithi156'] 2022-07-05T06:12:47.507 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-07-05T06:12:47.509 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-07-05T06:12:47.509 INFO:teuthology.kill:2022-07-05 06:12:47,507.507 INFO:teuthology.nuke:targets: 2022-07-05T06:12:47.510 INFO:teuthology.kill: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3XRBxkZMusT/GyZdCgMpjAa+HNm6SDzd8CogIqrbmWlv/h89LV+QsSO4n4qZzKRB0y8zRUwXk69c/ANWs3mOJdE4Wk0GnEJnSe2oBSFe8Izm+FMO2kBuHw25293R13uQlVhXwmdVY4vBMpTq7AvciOus3zgjKscfeR87fTzHEk3EJdo17z5nzZp5M8+TrN8krChYWjlgu6/gB5vRmvL+puZyDltgtvYMvKNRtm/NL8jV5+5A9GluMfdrzA/n/ImIJztI12uc0TsLYOua7S2FJLfOQULBd1eO7TRzlTGfmKYhUU0EududN60KRd9EZoJKHhLXOctUiPzATPuvi/j/kBopeVKHer3cT4yNg/wh77qjyE21bcr5y1zDsw75/LqDDNNekySyabE0VlybK1jysdLi2OjtaKcDzU3TsLvVI6JO49lgasB+aFKG5ClCR9Tull+W+hQKPMnZG1oCORpzRE7Yf1WhiymaMjm0uJvrTf3t4esvk+64VbDTGGEDeI4E= 2022-07-05T06:12:47.510 INFO:teuthology.kill: smithi156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC13j5xflBJaLyYIfD0zGEepJlGjFhx5kFpwuu9lRsjOrJn+6erXT3uozXnaEPlji1H+Mv69bYvCe1ZrelVTUrywtYOoe4+YnEO+g5Yii+SkQZPA9HQl0FiBPt3XfBW/fagQwthNLI9+Ii0nJ6hwWwnNlcjSYNX6lLesSAfrQjlOXsWx4LNIdFX3bBszObRj3bHnRm/2dXqxyV+51e5vNl7OzPmP16ltqbzB8siFLLVL2jl8yS4vVUMAOyXHBHyJZTzg74LiI27R0LaOSRJUcalxZ7J9/7Pf5VgcdqBeA8og4W0tYDPkJeHdA0B/OycqLO5b+63Io69jJTeTpdL36CMGF04/x8ICszRH/PLDzL1WCk/WQvk4o3oPJMCCYmjI8Djg2KgRDVVnvCQpvcWOZm/rGGvYsbgRL+qnDPtgmx7sZgDLMIOwq2fP/yEEmJ4jicwXIMIn8wr5slgYaghdTduUukxz40m2SSwn7ncUKj41b5ubMJj5nYGsiRWKokd4fc= 2022-07-05T06:12:47.511 INFO:teuthology.kill:2022-07-05 06:12:47,507.507 INFO:teuthology.nuke:Checking targets against current locks 2022-07-05T06:12:47.573 INFO:teuthology.kill:2022-07-05 06:12:47,572.572 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-05T06:12:47.604 INFO:teuthology.kill:2022-07-05 06:12:47,604.604 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-05T06:12:47.654 INFO:teuthology.kill:2022-07-05 06:12:47,653.653 INFO:teuthology.orchestra.console:Power off smithi057 2022-07-05T06:12:47.701 INFO:teuthology.kill:2022-07-05 06:12:47,700.700 INFO:teuthology.orchestra.console:Power off smithi156 2022-07-05T06:12:56.037 INFO:teuthology.kill:2022-07-05 06:12:56,037.037 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2022-07-05T06:12:56.196 INFO:teuthology.kill:2022-07-05 06:12:56,195.195 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com 2022-07-05T06:13:00.179 INFO:teuthology.kill:2022-07-05 06:13:00,178.178 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2022-07-05T06:13:00.344 INFO:teuthology.kill:2022-07-05 06:13:00,344.344 INFO:teuthology.lock.ops:unlocked: smithi156.front.sepia.ceph.com 2022-07-05T06:15:00.734 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-07-05T06:15:00.863 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi057 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2022-07-05_05:48:04-rgw-wip-skoduri-dbstore-lc-distro-default-smithi/6913830 2022-07-05T06:15:00.864 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi156 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2022-07-05_05:48:04-rgw-wip-skoduri-dbstore-lc-distro-default-smithi/6913830