2022-02-26T06:29:22.871 INFO:root:teuthology version: 1.1.0-5825e8d0
2022-02-26T06:29:22.996 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging
2022-02-26T06:29:22.997 INFO:teuthology.lock.ops:Updating [ubuntu@smithi162.front.sepia.ceph.com]: reset os type and version on server
2022-02-26T06:29:22.997 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server
2022-02-26T06:29:23.015 INFO:teuthology.lock.ops:Node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging is complete
2022-02-26T06:29:23.015 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi181.front.sepia.ceph.com' reimaging
2022-02-26T06:29:23.016 INFO:teuthology.lock.ops:Updating [ubuntu@smithi181.front.sepia.ceph.com]: reset os type and version on server
2022-02-26T06:29:23.016 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2022-02-26T06:29:23.034 INFO:teuthology.lock.ops:Node 'ubuntu@smithi181.front.sepia.ceph.com' reimaging is complete
2022-02-26T06:29:23.230 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of rhel 8.4
2022-02-26T06:29:23.245 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of rhel 8.4
2022-02-26T06:29:23.546 INFO:teuthology.orchestra.console:Power off smithi181
2022-02-26T06:29:23.561 INFO:teuthology.orchestra.console:Power off smithi162
2022-02-26T06:29:31.922 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2022-02-26T06:29:31.928 INFO:teuthology.orchestra.console:Power off for smithi162 completed
2022-02-26T06:29:32.024 INFO:teuthology.orchestra.console:Power on smithi181
2022-02-26T06:29:32.037 INFO:teuthology.orchestra.console:Power on smithi162
2022-02-26T06:29:40.506 INFO:teuthology.orchestra.console:Power on for smithi181 completed
2022-02-26T06:29:40.511 INFO:teuthology.orchestra.console:Power on for smithi162 completed
2022-02-26T06:29:40.608 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish
2022-02-26T06:29:40.612 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish
2022-02-26T06:33:48.703 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-26T06:34:09.016 INFO:teuthology.provision.fog.smithi181:Node is ready
2022-02-26T06:34:09.037 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com
2022-02-26T06:34:09.093 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181
2022-02-26T06:34:09.437 INFO:teuthology.provision.fog.smithi181:Deploy complete!
2022-02-26T06:34:27.293 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-26T06:34:27.532 INFO:teuthology.provision.fog.smithi162:Node is ready
2022-02-26T06:34:27.548 INFO:teuthology.orchestra.run.smithi162.stdout:smithi162.front.sepia.ceph.com
2022-02-26T06:34:27.604 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.15.162 smithi162.front.sepia.ceph.com smithi162
2022-02-26T06:34:27.923 INFO:teuthology.provision.fog.smithi162:Deploy complete!
2022-02-26T06:34:28.122 INFO:teuthology.lock.ops:Checking smithi162.front.sepia.ceph.com
2022-02-26T06:34:28.123 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-26T06:34:28.141 INFO:teuthology.lock.ops:Checking smithi181.front.sepia.ceph.com
2022-02-26T06:34:28.142 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-26T06:34:28.160 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: set os type and version on server
2022-02-26T06:34:28.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com'
2022-02-26T06:34:28.378 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64
2022-02-26T06:34:28.432 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Red Hat Enterprise Linux"
2022-02-26T06:34:28.433 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8.4 (Ootpa)"
2022-02-26T06:34:28.433 INFO:teuthology.orchestra.run.smithi162.stdout:ID="rhel"
2022-02-26T06:34:28.433 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="fedora"
2022-02-26T06:34:28.433 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8.4"
2022-02-26T06:34:28.434 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8"
2022-02-26T06:34:28.434 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-02-26T06:34:28.434 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31"
2022-02-26T06:34:28.434 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-02-26T06:34:28.435 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.redhat.com/"
2022-02-26T06:34:28.435 INFO:teuthology.orchestra.run.smithi162.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-02-26T06:34:28.435 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-26T06:34:28.435 INFO:teuthology.orchestra.run.smithi162.stdout:
2022-02-26T06:34:28.436 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-26T06:34:28.436 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-02-26T06:34:28.436 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-02-26T06:34:28.436 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-02-26T06:34:28.438 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server
2022-02-26T06:34:28.461 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: set os type and version on server
2022-02-26T06:34:28.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2022-02-26T06:34:28.701 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64
2022-02-26T06:34:28.756 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Red Hat Enterprise Linux"
2022-02-26T06:34:28.756 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8.4 (Ootpa)"
2022-02-26T06:34:28.756 INFO:teuthology.orchestra.run.smithi181.stdout:ID="rhel"
2022-02-26T06:34:28.757 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="fedora"
2022-02-26T06:34:28.757 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8.4"
2022-02-26T06:34:28.757 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8"
2022-02-26T06:34:28.757 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-02-26T06:34:28.758 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31"
2022-02-26T06:34:28.758 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-02-26T06:34:28.758 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.redhat.com/"
2022-02-26T06:34:28.758 INFO:teuthology.orchestra.run.smithi181.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-02-26T06:34:28.758 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-26T06:34:28.759 INFO:teuthology.orchestra.run.smithi181.stdout:
2022-02-26T06:34:28.759 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-26T06:34:28.759 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-02-26T06:34:28.759 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-02-26T06:34:28.760 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-02-26T06:34:28.760 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2022-02-26T06:34:28.874 INFO:teuthology.dispatcher.supervisor:Running job 6706954
2022-02-26T06:34:28.874 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_5825e8d094b6aff4351d26f1ceb5efc598c156a0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-02-25_18:58:00-upgrade-master-distro-default-smithi/6706954 --name yuriw-2022-02-25_18:58:00-upgrade-master-distro-default-smithi --description upgrade/octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} -- /home/teuthworker/archive/yuriw-2022-02-25_18:58:00-upgrade-master-distro-default-smithi/6706954/orig.config.yaml
2022-02-26T06:34:28.877 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-02-25_18:58:00-upgrade-master-distro-default-smithi/6706954
2022-02-26T06:34:28.878 INFO:teuthology.dispatcher.supervisor:Job PID: 14268
2022-02-26T06:34:28.879 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-26T13:04:57.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-26T13:04:58.075 INFO:teuthology.kill:Killing Pids: {14268}
2022-02-26T13:04:59.399 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2022-02-26T13:04:59.400 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2022-02-26T13:04:59.401 INFO:teuthology.misc:Compressing logs...
2022-02-26T13:04:59.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com'
2022-02-26T13:04:59.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2022-02-26T13:04:59.977 INFO:teuthology.misc:Compressing logs...
2022-02-26T13:05:00.285 INFO:teuthology.misc:Compressing logs...
2022-02-26T13:05:15.505 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-02-26T13:05:18.863 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-02-26T13:05:19.979 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2022-02-26T13:05:32.595 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2022-02-26T13:05:36.558 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-02-26T13:05:39.554 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2022-02-26T13:05:50.311 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-02-26T13:05:55.683 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-02-26T13:06:05.568 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-02-26T13:06:10.162 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-02-26T13:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-02-26T13:06:25.543 INFO:teuthology.orchestra.run.smithi162.stderr:gzip: /var/log/ceph/f21d17b0-96d0-11ec-8c35-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2022-02-26T13:06:56.592 INFO:teuthology.kill:No teuthology processes running
2022-02-26T13:06:57.766 INFO:teuthology.kill:Nuking machines: ['smithi162', 'smithi181']
2022-02-26T13:06:58.783 INFO:teuthology.kill:2022-02-26 13:06:58,783.783 INFO:teuthology.nuke:targets:
2022-02-26T13:06:58.784 INFO:teuthology.kill:  smithi162.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8oEzOem7OLAEScmxw9oFA9AhheUjLINLLUX2yMPyc2XcbKb5/QIEilWiNL/Yg1dPVHD7eaVNv+hgbrGR2Py7g2lQaRY+EVRGOoCDE65DWpwUlRngDKUMfBGi5gaug2nXUf4UqcnM+i1DzRlx7jMYtY/yyuLlWgQ+C9O7MkmWD5R/GIyCV5UTdD6vBMqyHpNJdr2SzhqOzKtyTMta6sl+RllOclOBAxHrpTpP+tok6FOL5iPy9t9RP+koJFALC4SemudFs1XjI5nSjBLJbquX/NehwA+ymqazO6VWcWv9GyoEto0i5z+kUBOxxXdCDroNfj4B7mddr3rIW4YbGz6nDixDXsLVV8g8oJO+cx8+87pQc8eioJEhqGDSnUiOfMtKN6ZvLnsaexMJHJuCkGYBxAwdCHKG8wzIpKrGIJtDZz1fS4/leE4FHfD4flwbXVuzTmxrzdKTMWwXjfZf8HMWJ5QznUZ7wFyDFhuCPoBJX2/20aL/b56lTYg0n7G8/cCM=
2022-02-26T13:06:58.784 INFO:teuthology.kill:  smithi181.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPt74SOJ6WUNgkPPeeaCORefchnQLt/U09EVew/pJAexDuM8XGrGWNF2+I6oXxCwIddUWDyQ5QuAnGi7yNXEHmDWWVqNxmS94v/ycrFN9OJfROgxedUyKCXSnDFM/IpqcKmrjJs8FvdHXpTVZMdUI1zqCUklem14SFnMRAjW1yN7EyDhpZZYGldGwU7Xp63Qm1XA998QCtzv0MT5/iLupS0rDh4LxjtzfK9bftV8WDu9eBHrrX1IlhpR6rM4UP1VDzoQFYjLpixzxSfvPHbORTco8bqLqgTRI4m7G+yVYlaUHAedofif8m/SscRow9w3RgCP42/0JhLCtsbMijP3yguwR+X++uFHleok7o0pC0ib4p1vJojxc5WsNO2HQlyjrPW+KkvyA+3ucKJ/RwIYoVMGAN7JHxEj5XeNuNGb3tu4wGtSoBZAs3CQBNSrgPJeG9MSGUzo7r/vsij++gOf3NKh4xLQy30SPwz71AoA4QcGIks0DV6uSmVj5ziAtvJOM=
2022-02-26T13:06:58.810 INFO:teuthology.kill:2022-02-26 13:06:58,810.810 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-26T13:06:58.813 INFO:teuthology.kill:2022-02-26 13:06:58,813.813 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-26T13:06:58.890 INFO:teuthology.kill:2022-02-26 13:06:58,890.890 INFO:teuthology.orchestra.console:Power off smithi162
2022-02-26T13:06:58.907 INFO:teuthology.kill:2022-02-26 13:06:58,907.907 INFO:teuthology.orchestra.console:Power off smithi181
2022-02-26T13:07:11.376 INFO:teuthology.kill:2022-02-26 13:07:11,376.376 INFO:teuthology.orchestra.console:Power off for smithi162 completed
2022-02-26T13:07:11.383 INFO:teuthology.kill:2022-02-26 13:07:11,383.383 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2022-02-26T13:07:11.526 INFO:teuthology.kill:2022-02-26 13:07:11,525.525 INFO:teuthology.lock.ops:unlocked: smithi162.front.sepia.ceph.com
2022-02-26T13:07:11.528 INFO:teuthology.kill:2022-02-26 13:07:11,528.528 INFO:teuthology.lock.ops:unlocked: smithi181.front.sepia.ceph.com
2022-02-26T13:09:11.863 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-26T13:09:11.979 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi162 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-25_19:09:45-fs-master-distro-default-smithi/6707429
2022-02-26T13:09:11.980 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi181 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-25_19:09:45-fs-master-distro-default-smithi/6707429