2022-09-14T15:32:21.981 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-14T15:32:22.123 INFO:teuthology.lock.ops:Start node 'smithi096.front.sepia.ceph.com' reimaging
2022-09-14T15:32:22.125 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: reset os type and version on server
2022-09-14T15:32:22.125 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server
2022-09-14T15:32:22.146 INFO:teuthology.lock.ops:Node 'smithi096.front.sepia.ceph.com' reimaging is complete
2022-09-14T15:32:22.146 INFO:teuthology.lock.ops:Start node 'smithi157.front.sepia.ceph.com' reimaging
2022-09-14T15:32:22.147 INFO:teuthology.lock.ops:Updating [smithi157.front.sepia.ceph.com]: reset os type and version on server
2022-09-14T15:32:22.147 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2022-09-14T15:32:22.169 INFO:teuthology.lock.ops:Node 'smithi157.front.sepia.ceph.com' reimaging is complete
2022-09-14T15:32:22.362 INFO:teuthology.provision.fog.smithi096:Scheduling deploy of rhel 8.4
2022-09-14T15:32:22.380 INFO:teuthology.provision.fog.smithi157:Scheduling deploy of rhel 8.4
2022-09-14T15:32:22.696 INFO:teuthology.orchestra.console:Power off smithi096
2022-09-14T15:32:22.725 INFO:teuthology.orchestra.console:Power off smithi157
2022-09-14T15:32:31.088 INFO:teuthology.orchestra.console:Power off for smithi096 completed
2022-09-14T15:32:31.102 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2022-09-14T15:32:31.191 INFO:teuthology.orchestra.console:Power on smithi096
2022-09-14T15:32:31.203 INFO:teuthology.orchestra.console:Power on smithi157
2022-09-14T15:32:39.566 INFO:teuthology.orchestra.console:Power on for smithi096 completed
2022-09-14T15:32:39.596 INFO:teuthology.orchestra.console:Power on for smithi157 completed
2022-09-14T15:32:39.668 INFO:teuthology.provision.fog.smithi096:Waiting for deploy to finish
2022-09-14T15:32:39.698 INFO:teuthology.provision.fog.smithi157:Waiting for deploy to finish
2022-09-14T15:36:11.242 ERROR:teuthology.orchestra.connection:Error authenticating with smithi157.front.sepia.ceph.com: Authentication failed.
2022-09-14T15:37:17.854 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-09-14T15:37:18.083 INFO:teuthology.provision.fog.smithi096:Node is ready
2022-09-14T15:37:18.139 INFO:teuthology.orchestra.run.smithi096.stdout:smithi096.front.sepia.ceph.com
2022-09-14T15:37:18.197 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96 smithi096.front.sepia.ceph.com smithi096
2022-09-14T15:37:18.531 INFO:teuthology.provision.fog.smithi096:Deploy complete!
2022-09-14T15:38:37.360 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-09-14T15:38:37.607 INFO:teuthology.provision.fog.smithi157:Node is ready
2022-09-14T15:38:37.626 INFO:teuthology.orchestra.run.smithi157.stdout:smithi157.front.sepia.ceph.com
2022-09-14T15:38:37.683 INFO:teuthology.orchestra.run.smithi157.stdout:172.21.15.157 smithi157.front.sepia.ceph.com smithi157
2022-09-14T15:38:37.995 INFO:teuthology.provision.fog.smithi157:Deploy complete!
2022-09-14T15:38:38.209 INFO:teuthology.lock.ops:Checking smithi096.front.sepia.ceph.com
2022-09-14T15:38:38.210 INFO:teuthology.lock.ops:New key found. Updating...
2022-09-14T15:38:38.232 INFO:teuthology.lock.ops:Checking smithi157.front.sepia.ceph.com
2022-09-14T15:38:38.232 INFO:teuthology.lock.ops:New key found. Updating...
2022-09-14T15:38:38.253 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: set os type and version on server
2022-09-14T15:38:38.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com'
2022-09-14T15:38:38.505 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64
2022-09-14T15:38:38.561 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="Red Hat Enterprise Linux"
2022-09-14T15:38:38.561 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="8.4 (Ootpa)"
2022-09-14T15:38:38.562 INFO:teuthology.orchestra.run.smithi096.stdout:ID="rhel"
2022-09-14T15:38:38.562 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="fedora"
2022-09-14T15:38:38.562 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="8.4"
2022-09-14T15:38:38.563 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el8"
2022-09-14T15:38:38.563 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-09-14T15:38:38.563 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31"
2022-09-14T15:38:38.563 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-09-14T15:38:38.564 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.redhat.com/"
2022-09-14T15:38:38.564 INFO:teuthology.orchestra.run.smithi096.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-09-14T15:38:38.564 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-09-14T15:38:38.565 INFO:teuthology.orchestra.run.smithi096.stdout:
2022-09-14T15:38:38.565 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-09-14T15:38:38.565 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-09-14T15:38:38.565 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-09-14T15:38:38.566 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-09-14T15:38:38.567 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server
2022-09-14T15:38:38.591 INFO:teuthology.lock.ops:Updating [smithi157.front.sepia.ceph.com]: set os type and version on server
2022-09-14T15:38:38.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2022-09-14T15:38:38.802 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64
2022-09-14T15:38:38.857 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Red Hat Enterprise Linux"
2022-09-14T15:38:38.857 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8.4 (Ootpa)"
2022-09-14T15:38:38.858 INFO:teuthology.orchestra.run.smithi157.stdout:ID="rhel"
2022-09-14T15:38:38.858 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="fedora"
2022-09-14T15:38:38.858 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8.4"
2022-09-14T15:38:38.859 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8"
2022-09-14T15:38:38.859 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-09-14T15:38:38.859 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31"
2022-09-14T15:38:38.859 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-09-14T15:38:38.860 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.redhat.com/"
2022-09-14T15:38:38.860 INFO:teuthology.orchestra.run.smithi157.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-09-14T15:38:38.860 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-09-14T15:38:38.860 INFO:teuthology.orchestra.run.smithi157.stdout:
2022-09-14T15:38:38.860 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-09-14T15:38:38.861 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-09-14T15:38:38.861 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-09-14T15:38:38.861 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-09-14T15:38:38.862 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2022-09-14T15:38:38.960 INFO:teuthology.dispatcher.supervisor:Running job 7032596
2022-09-14T15:38:38.960 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi/7032596 --name lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} -- /home/teuthworker/archive/lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi/7032596/orig.config.yaml
2022-09-14T15:38:38.966 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi/7032596
2022-09-14T15:38:38.967 INFO:teuthology.dispatcher.supervisor:Job PID: 7913
2022-09-14T15:38:38.967 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-09-15T03:39:02.481 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-09-15T03:39:02.621 INFO:teuthology.kill:Killing Pids: {7913}
2022-09-15T03:39:04.066 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2022-09-15T03:39:04.067 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2022-09-15T03:39:04.067 INFO:teuthology.misc:Compressing logs...
2022-09-15T03:39:04.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com'
2022-09-15T03:39:04.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2022-09-15T03:39:04.623 INFO:teuthology.misc:Compressing logs...
2022-09-15T03:39:05.735 INFO:teuthology.orchestra.run.smithi096.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-09-15T03:39:05.755 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-09-15T03:39:06.399 INFO:teuthology.misc:Compressing logs...
2022-09-15T03:45:13.571 INFO:teuthology.kill:No teuthology processes running
2022-09-15T03:45:15.023 INFO:teuthology.kill:Nuking machines: ['smithi096', 'smithi157']
2022-09-15T03:45:16.335 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-09-15T03:45:16.336 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-09-15T03:45:16.337 INFO:teuthology.kill:2022-09-15 03:45:16,335.335 INFO:teuthology.nuke:targets:
2022-09-15T03:45:16.337 INFO:teuthology.kill:  smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzYTaEzCKF2PImDyYnht/tc7yh6u4AZW0pA/ONtlvr/Ox+rF1R7pcYxKjJoCaOiuw20L08Mkd+50LP3o59NZtJyvJNHQo3BBHWwN4es5k2jMvwADPwInwXIIKfMsy5+EMNhyLL6d2nEtSjFCm90QVtHJrrYHsPQG4YKH/e9jvx1u+nUAdDu+MRFgiXCWxOOSvlq1iU0y/YVKbHqKUcbTzngTxWYlqNsURjafKVxvb/D3GwfQSWYVaokzSOlqe7bPzAM36lOocZ1rDHaMXBzoJqT1tkcFenx80L+r3JpYBmYl/+/+ADGRi9C0/dR1A/GAf8LnxxYOsWJ4iiD2ROWm1bvic5AhLoz50GVOAY0ffhc8D2QbKlneiiNQZwNTA/KDob6d21jvkdAALHUhuOlBxnlxxr5z4t2udjZiF+oNTm9UCIND93iqwGr/vFzKxr8oK33CXIdX9MyHnToSSzf9m+pFkcqajoEJDX5Oyj07blxRhYoyxMduoT15A7jAUgKyc=
2022-09-15T03:45:16.337 INFO:teuthology.kill:  smithi157.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChpS7zihEG6lgwr88RTlXOJjOVrIISWLYOkZ7G4Prn7RBTtCXQTm5OY83e7exW+zp4L2awcwmeoXKHoxbHOUh2BBD89S+X16CrM6e+ZptORU4iQfqQpSLiHLd75J1kYamCexj27wYhVDFrL5IGX/rmgBCj5JaEO4q5zB+BZXyjwhuNWaV1UDCRTUQ3sLV003iYdjSO+Cs7OUqDw2Z2b0AnMaFgLK4n/NOpayVRStCcTpYBZW4ASPwit2oPpdmIk0ZBKAwQgiIAcp+tl8xYx5RcxCS6Soadm3El77eO73t7SL5xFD/HIK6LogKgFz9YXH6khoc5EfU/GVfZ0av8zgUkiHggfRHh6MBlpzbM1WJN7ffqnJ6Amf/WrW+j0olu5xGD4qZplCAvv/M3YNKn0w2JzniaqlePm/VSahiHt+9T1D6hILZkYOup/EPc7Lb+Gs/cK4q6UdRudkot4vsx8Re9p4X0i21uehrdPo8uMVrJkeIA6ir/8IdIyglchrqjYP0=
2022-09-15T03:45:16.338 INFO:teuthology.kill:2022-09-15 03:45:16,336.336 INFO:teuthology.nuke:Checking targets against current locks
2022-09-15T03:45:16.393 INFO:teuthology.kill:2022-09-15 03:45:16,393.393 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-09-15T03:45:16.421 INFO:teuthology.kill:2022-09-15 03:45:16,420.420 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-09-15T03:45:16.492 INFO:teuthology.kill:2022-09-15 03:45:16,492.492 INFO:teuthology.orchestra.console:Power off smithi096
2022-09-15T03:45:16.535 INFO:teuthology.kill:2022-09-15 03:45:16,534.534 INFO:teuthology.orchestra.console:Power off smithi157
2022-09-15T03:45:24.931 INFO:teuthology.kill:2022-09-15 03:45:24,930.930 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2022-09-15T03:45:25.080 INFO:teuthology.kill:2022-09-15 03:45:25,079.079 INFO:teuthology.lock.ops:unlocked: smithi157.front.sepia.ceph.com
2022-09-15T03:45:28.999 INFO:teuthology.kill:2022-09-15 03:45:28,998.998 INFO:teuthology.orchestra.console:Power off for smithi096 completed
2022-09-15T03:45:29.179 INFO:teuthology.kill:2022-09-15 03:45:29,179.179 INFO:teuthology.lock.ops:unlocked: smithi096.front.sepia.ceph.com
2022-09-15T03:47:29.498 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-09-15T03:47:29.631 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi157 but it was locked by another job: /home/teuthworker/archive/rishabh-2022-09-14_12:48:43-fs-wip-rishabh-testing-2022Sep9-1708-testing-default-smithi/7032044