2023-03-03T06:01:11.369 INFO:root:teuthology version: 0.0.1.dev49+gc7c3cc8
2023-03-03T06:01:11.526 INFO:teuthology.lock.ops:Start node 'smithi090.front.sepia.ceph.com' reimaging
2023-03-03T06:01:11.527 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: reset os type and version on server
2023-03-03T06:01:11.527 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-03-03T06:01:11.556 INFO:teuthology.lock.ops:Node 'smithi090.front.sepia.ceph.com' reimaging is complete
2023-03-03T06:01:11.557 INFO:teuthology.lock.ops:Start node 'smithi121.front.sepia.ceph.com' reimaging
2023-03-03T06:01:11.557 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: reset os type and version on server
2023-03-03T06:01:11.557 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2023-03-03T06:01:11.581 INFO:teuthology.lock.ops:Node 'smithi121.front.sepia.ceph.com' reimaging is complete
2023-03-03T06:01:11.714 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of rhel 8.6
2023-03-03T06:01:11.739 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of rhel 8.6
2023-03-03T06:01:11.938 INFO:teuthology.orchestra.console:Power off smithi090
2023-03-03T06:01:11.995 INFO:teuthology.orchestra.console:Power off smithi121
2023-03-03T06:01:20.411 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2023-03-03T06:01:20.512 INFO:teuthology.orchestra.console:Power on smithi090
2023-03-03T06:01:24.511 INFO:teuthology.orchestra.console:Power off for smithi121 completed
2023-03-03T06:01:24.613 INFO:teuthology.orchestra.console:Power on smithi121
2023-03-03T06:01:28.868 INFO:teuthology.orchestra.console:Power on for smithi121 completed
2023-03-03T06:01:28.876 INFO:teuthology.orchestra.console:Power on for smithi090 completed
2023-03-03T06:01:28.970 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish
2023-03-03T06:01:28.978 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2023-03-03T06:04:18.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-03-03T06:04:27.209 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-03-03T06:04:33.221 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:04:36.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-03-03T06:04:42.277 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:04:45.353 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-03-03T06:04:51.333 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:04:54.405 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-03-03T06:05:00.389 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:05:03.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-03-03T06:05:09.449 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:05:18.537 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:05:27.621 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:05:36.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:05:45.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-03T06:06:09.478 WARNING:teuthology.provision.fog:timed out
2023-03-03T06:06:15.816 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-03T06:06:16.090 INFO:teuthology.provision.fog.smithi121:Node is ready
2023-03-03T06:06:16.107 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121
2023-03-03T06:06:16.166 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121
2023-03-03T06:06:16.500 INFO:teuthology.provision.fog.smithi121:Deploy complete!
2023-03-03T06:06:51.773 WARNING:teuthology.provision.fog:timed out
2023-03-03T06:06:58.120 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-03T06:06:58.425 INFO:teuthology.provision.fog.smithi090:Node is ready
2023-03-03T06:06:58.443 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090
2023-03-03T06:06:58.502 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090
2023-03-03T06:06:58.874 INFO:teuthology.provision.fog.smithi090:Deploy complete!
2023-03-03T06:06:59.056 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com
2023-03-03T06:06:59.057 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com
2023-03-03T06:06:59.058 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-03T06:06:59.082 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server
2023-03-03T06:06:59.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-03T06:06:59.276 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64
2023-03-03T06:06:59.337 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Red Hat Enterprise Linux"
2023-03-03T06:06:59.337 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8.6 (Ootpa)"
2023-03-03T06:06:59.338 INFO:teuthology.orchestra.run.smithi090.stdout:ID="rhel"
2023-03-03T06:06:59.338 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="fedora"
2023-03-03T06:06:59.338 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8.6"
2023-03-03T06:06:59.338 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8"
2023-03-03T06:06:59.338 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-03T06:06:59.338 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31"
2023-03-03T06:06:59.338 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-03T06:06:59.339 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.redhat.com/"
2023-03-03T06:06:59.339 INFO:teuthology.orchestra.run.smithi090.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-03T06:06:59.339 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-03T06:06:59.339 INFO:teuthology.orchestra.run.smithi090.stdout:
2023-03-03T06:06:59.339 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-03T06:06:59.339 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-03T06:06:59.340 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-03T06:06:59.340 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-03T06:06:59.341 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-03-03T06:06:59.363 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server
2023-03-03T06:06:59.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com'
2023-03-03T06:06:59.578 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64
2023-03-03T06:06:59.635 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux"
2023-03-03T06:06:59.636 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.6 (Ootpa)"
2023-03-03T06:06:59.636 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel"
2023-03-03T06:06:59.637 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora"
2023-03-03T06:06:59.637 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.6"
2023-03-03T06:06:59.637 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8"
2023-03-03T06:06:59.638 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-03T06:06:59.638 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31"
2023-03-03T06:06:59.638 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-03T06:06:59.638 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/"
2023-03-03T06:06:59.639 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-03T06:06:59.639 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-03T06:06:59.639 INFO:teuthology.orchestra.run.smithi121.stdout:
2023-03-03T06:06:59.639 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-03T06:06:59.640 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-03T06:06:59.640 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-03T06:06:59.641 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-03T06:06:59.642 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2023-03-03T06:06:59.778 INFO:teuthology.dispatcher.supervisor:Running job 7192140
2023-03-03T06:06:59.778 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140 --name vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi --description fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} -- /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140/orig.config.yaml
2023-03-03T06:06:59.782 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192140
2023-03-03T06:06:59.782 INFO:teuthology.dispatcher.supervisor:Job PID: 29812
2023-03-03T06:06:59.783 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-03T18:07:53.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-03T18:07:54.013 INFO:teuthology.kill:Killing Pids: {29812}
2023-03-03T18:07:54.829 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-03T18:07:54.830 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-03-03T18:07:54.830 INFO:teuthology.misc:Compressing logs...
2023-03-03T18:07:54.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-03T18:07:55.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com'
2023-03-03T18:07:55.321 INFO:teuthology.misc:Compressing logs...
2023-03-03T18:07:55.491 INFO:teuthology.misc:Compressing logs...
2023-03-03T18:09:41.655 INFO:teuthology.kill:No teuthology processes running
2023-03-03T18:09:42.433 INFO:teuthology.kill:Nuking machines: ['smithi090', 'smithi121']
2023-03-03T18:09:43.027 INFO:teuthology.kill:2023-03-03 18:09:43,027.027 INFO:teuthology.nuke:targets:
2023-03-03T18:09:43.028 INFO:teuthology.kill:  smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-03T18:09:43.028 INFO:teuthology.kill:  smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-03T18:09:43.029 INFO:teuthology.kill:2023-03-03 18:09:43,028.028 INFO:teuthology.nuke:Checking targets against current locks
2023-03-03T18:09:43.087 INFO:teuthology.kill:2023-03-03 18:09:43,087.087 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-03T18:09:43.104 INFO:teuthology.kill:2023-03-03 18:09:43,103.103 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-03T18:09:43.160 INFO:teuthology.kill:2023-03-03 18:09:43,159.159 INFO:teuthology.orchestra.console:Power off smithi090
2023-03-03T18:09:43.176 INFO:teuthology.kill:2023-03-03 18:09:43,175.175 INFO:teuthology.orchestra.console:Power off smithi121
2023-03-03T18:09:55.650 INFO:teuthology.kill:2023-03-03 18:09:55,649.649 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2023-03-03T18:09:55.656 INFO:teuthology.kill:2023-03-03 18:09:55,656.656 INFO:teuthology.orchestra.console:Power off for smithi121 completed
2023-03-03T18:09:55.813 INFO:teuthology.kill:2023-03-03 18:09:55,812.812 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com
2023-03-03T18:09:55.835 INFO:teuthology.kill:2023-03-03 18:09:55,834.834 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com
2023-03-03T18:11:56.182 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-03T18:11:56.288 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi090 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-03-03_17:39:20-powercycle-reef-distro-default-smithi/7192505