2023-03-29T22:37:51.908 INFO:root:teuthology version: 0.0.1.dev78+ge372916
2023-03-29T22:37:52.067 INFO:teuthology.lock.ops:Start node 'smithi032.front.sepia.ceph.com' reimaging
2023-03-29T22:37:52.069 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: reset os type and version on server
2023-03-29T22:37:52.069 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2023-03-29T22:37:52.093 INFO:teuthology.lock.ops:Node 'smithi032.front.sepia.ceph.com' reimaging is complete
2023-03-29T22:37:52.093 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging
2023-03-29T22:37:52.093 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server
2023-03-29T22:37:52.094 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2023-03-29T22:37:52.117 INFO:teuthology.lock.ops:Node 'smithi097.front.sepia.ceph.com' reimaging is complete
2023-03-29T22:37:52.250 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of rhel 8.6
2023-03-29T22:37:52.266 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of rhel 8.6
2023-03-29T22:37:52.566 INFO:teuthology.orchestra.console:Power off smithi032
2023-03-29T22:37:52.583 INFO:teuthology.orchestra.console:Power off smithi097
2023-03-29T22:38:00.948 INFO:teuthology.orchestra.console:Power off for smithi032 completed
2023-03-29T22:38:01.049 INFO:teuthology.orchestra.console:Power on smithi032
2023-03-29T22:38:05.067 INFO:teuthology.orchestra.console:Power off for smithi097 completed
2023-03-29T22:38:05.169 INFO:teuthology.orchestra.console:Power on smithi097
2023-03-29T22:38:09.418 INFO:teuthology.orchestra.console:Power on for smithi032 completed
2023-03-29T22:38:09.519 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish
2023-03-29T22:38:13.542 INFO:teuthology.orchestra.console:Power on for smithi097 completed
2023-03-29T22:38:13.644 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish
2023-03-29T22:40:44.509 ERROR:teuthology.orchestra.connection:Error authenticating with smithi097.front.sepia.ceph.com: Authentication failed.
2023-03-29T22:40:58.885 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-03-29T22:41:04.009 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97
2023-03-29T22:41:07.941 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-03-29T22:41:13.065 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97
2023-03-29T22:41:16.997 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-03-29T22:41:22.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97
2023-03-29T22:41:31.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97
2023-03-29T22:41:40.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97
2023-03-29T22:41:54.969 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-29T22:41:55.326 INFO:teuthology.provision.fog.smithi032:Node is ready
2023-03-29T22:41:55.343 INFO:teuthology.orchestra.run.smithi032.stdout:smithi022
2023-03-29T22:41:55.402 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-03-29T22:41:55.809 INFO:teuthology.provision.fog.smithi032:Deploy complete!
2023-03-29T22:42:04.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97
2023-03-29T22:42:13.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97
2023-03-29T22:42:20.074 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-29T22:42:20.355 INFO:teuthology.provision.fog.smithi097:Node is ready
2023-03-29T22:42:20.419 INFO:teuthology.orchestra.run.smithi097.stdout:smithi022
2023-03-29T22:42:20.477 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-03-29T22:42:20.865 INFO:teuthology.provision.fog.smithi097:Deploy complete!
2023-03-29T22:42:21.047 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com
2023-03-29T22:42:21.048 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-29T22:42:21.078 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com
2023-03-29T22:42:21.078 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-29T22:42:21.103 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server
2023-03-29T22:42:21.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-29T22:42:21.366 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64
2023-03-29T22:42:21.424 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Red Hat Enterprise Linux"
2023-03-29T22:42:21.424 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8.6 (Ootpa)"
2023-03-29T22:42:21.425 INFO:teuthology.orchestra.run.smithi032.stdout:ID="rhel"
2023-03-29T22:42:21.425 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="fedora"
2023-03-29T22:42:21.425 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8.6"
2023-03-29T22:42:21.425 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8"
2023-03-29T22:42:21.425 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-29T22:42:21.425 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31"
2023-03-29T22:42:21.427 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-29T22:42:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.redhat.com/"
2023-03-29T22:42:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-29T22:42:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-29T22:42:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:
2023-03-29T22:42:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-29T22:42:21.428 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-29T22:42:21.429 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-29T22:42:21.429 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-29T22:42:21.430 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2023-03-29T22:42:21.452 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server
2023-03-29T22:42:21.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com'
2023-03-29T22:42:21.668 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64
2023-03-29T22:42:21.729 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="Red Hat Enterprise Linux"
2023-03-29T22:42:21.729 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8.6 (Ootpa)"
2023-03-29T22:42:21.729 INFO:teuthology.orchestra.run.smithi097.stdout:ID="rhel"
2023-03-29T22:42:21.730 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="fedora"
2023-03-29T22:42:21.730 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8.6"
2023-03-29T22:42:21.730 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8"
2023-03-29T22:42:21.730 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-29T22:42:21.730 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31"
2023-03-29T22:42:21.730 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-29T22:42:21.730 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.redhat.com/"
2023-03-29T22:42:21.731 INFO:teuthology.orchestra.run.smithi097.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-29T22:42:21.731 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-29T22:42:21.731 INFO:teuthology.orchestra.run.smithi097.stdout:
2023-03-29T22:42:21.731 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-29T22:42:21.731 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-29T22:42:21.731 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-29T22:42:21.731 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-29T22:42:21.732 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2023-03-29T22:42:21.849 INFO:teuthology.dispatcher.supervisor:Running job 7224526
2023-03-29T22:42:21.850 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e3729161eecf8e6d09a4069acbb883cdcd5e9f50/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi/7224526 --name benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} -- /home/teuthworker/archive/benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi/7224526/orig.config.yaml
2023-03-29T22:42:21.854 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi/7224526
2023-03-29T22:42:21.855 INFO:teuthology.dispatcher.supervisor:Job PID: 18420
2023-03-29T22:42:21.856 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-30T10:42:45.892 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-30T10:42:46.035 INFO:teuthology.kill:Killing Pids: {18420}
2023-03-30T10:42:46.995 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2023-03-30T10:42:46.996 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2023-03-30T10:42:46.996 INFO:teuthology.misc:Compressing logs...
2023-03-30T10:42:46.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-30T10:42:47.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com'
2023-03-30T10:42:47.630 INFO:teuthology.misc:Compressing logs...
2023-03-30T10:42:48.640 INFO:teuthology.orchestra.run.smithi032.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-30T10:42:48.869 INFO:teuthology.orchestra.run.smithi097.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-30T10:43:30.178 INFO:teuthology.misc:Compressing logs...
2023-03-30T10:43:38.749 INFO:teuthology.kill:No teuthology processes running
2023-03-30T10:43:39.585 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi097']
2023-03-30T10:43:40.284 INFO:teuthology.kill:2023-03-30 10:43:40,284.284 INFO:teuthology.nuke:targets:
2023-03-30T10:43:40.285 INFO:teuthology.kill:  smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-30T10:43:40.285 INFO:teuthology.kill:  smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-03-30T10:43:40.286 INFO:teuthology.kill:2023-03-30 10:43:40,284.284 INFO:teuthology.nuke:Checking targets against current locks
2023-03-30T10:43:40.340 INFO:teuthology.kill:2023-03-30 10:43:40,340.340 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-30T10:43:40.366 INFO:teuthology.kill:2023-03-30 10:43:40,366.366 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-30T10:43:40.418 INFO:teuthology.kill:2023-03-30 10:43:40,417.417 INFO:teuthology.orchestra.console:Power off smithi032
2023-03-30T10:43:40.462 INFO:teuthology.kill:2023-03-30 10:43:40,461.461 INFO:teuthology.orchestra.console:Power off smithi097
2023-03-30T10:43:48.798 INFO:teuthology.kill:2023-03-30 10:43:48,797.797 INFO:teuthology.orchestra.console:Power off for smithi032 completed
2023-03-30T10:43:48.829 INFO:teuthology.kill:2023-03-30 10:43:48,829.829 INFO:teuthology.orchestra.console:Power off for smithi097 completed
2023-03-30T10:43:48.963 INFO:teuthology.kill:2023-03-30 10:43:48,963.963 INFO:teuthology.lock.ops:unlocked: smithi032.front.sepia.ceph.com
2023-03-30T10:43:48.978 INFO:teuthology.kill:2023-03-30 10:43:48,978.978 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com
2023-03-30T10:45:49.218 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-30T10:45:49.331 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi032 but it was locked by another job: /home/teuthworker/archive/matan-2023-03-30_06:52:21-crimson-rados-wip-matanb-crimson-only-send-map-distro-crimson-smithi/7226015