2022-03-26T17:33:03.924 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-26T17:33:04.043 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi124.front.sepia.ceph.com' reimaging 2022-03-26T17:33:04.044 INFO:teuthology.lock.ops:Updating [ubuntu@smithi124.front.sepia.ceph.com]: reset os type and version on server 2022-03-26T17:33:04.045 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2022-03-26T17:33:04.064 INFO:teuthology.lock.ops:Node 'ubuntu@smithi124.front.sepia.ceph.com' reimaging is complete 2022-03-26T17:33:04.064 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging 2022-03-26T17:33:04.065 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server 2022-03-26T17:33:04.065 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2022-03-26T17:33:04.085 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete 2022-03-26T17:33:04.269 INFO:teuthology.provision.fog.smithi124:Scheduling deploy of rhel 8.4 2022-03-26T17:33:04.291 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of rhel 8.4 2022-03-26T17:33:04.583 INFO:teuthology.orchestra.console:Power off smithi142 2022-03-26T17:33:04.598 INFO:teuthology.orchestra.console:Power off smithi124 2022-03-26T17:33:04.725 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2022-03-26T17:33:04.737 INFO:teuthology.orchestra.console:Power off for smithi124 completed 2022-03-26T17:33:04.826 INFO:teuthology.orchestra.console:Power on smithi142 2022-03-26T17:33:04.838 INFO:teuthology.orchestra.console:Power on smithi124 2022-03-26T17:33:09.088 INFO:teuthology.orchestra.console:Power on for smithi124 completed 2022-03-26T17:33:09.189 INFO:teuthology.provision.fog.smithi124:Waiting for deploy to finish 2022-03-26T17:33:13.195 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2022-03-26T17:33:13.296 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2022-03-26T17:37:40.792 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-26T17:37:41.039 INFO:teuthology.provision.fog.smithi124:Node is ready 2022-03-26T17:37:41.055 INFO:teuthology.orchestra.run.smithi124.stdout:smithi124.front.sepia.ceph.com 2022-03-26T17:37:41.111 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.15.124 smithi124.front.sepia.ceph.com smithi124 2022-03-26T17:37:41.447 INFO:teuthology.provision.fog.smithi124:Deploy complete! 2022-03-26T17:37:44.776 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-26T17:37:45.024 INFO:teuthology.provision.fog.smithi142:Node is ready 2022-03-26T17:37:45.039 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2022-03-26T17:37:45.094 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2022-03-26T17:37:45.409 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2022-03-26T17:37:45.550 INFO:teuthology.lock.ops:Checking smithi124.front.sepia.ceph.com 2022-03-26T17:37:45.551 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-26T17:37:45.577 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2022-03-26T17:37:45.577 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-26T17:37:45.596 INFO:teuthology.lock.ops:Updating [smithi124.front.sepia.ceph.com]: set os type and version on server 2022-03-26T17:37:45.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2022-03-26T17:37:45.831 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2022-03-26T17:37:45.887 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="Red Hat Enterprise Linux" 2022-03-26T17:37:45.887 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="8.4 (Ootpa)" 2022-03-26T17:37:45.888 INFO:teuthology.orchestra.run.smithi124.stdout:ID="rhel" 2022-03-26T17:37:45.888 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE="fedora" 2022-03-26T17:37:45.888 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="8.4" 2022-03-26T17:37:45.888 INFO:teuthology.orchestra.run.smithi124.stdout:PLATFORM_ID="platform:el8" 2022-03-26T17:37:45.889 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-26T17:37:45.889 INFO:teuthology.orchestra.run.smithi124.stdout:ANSI_COLOR="0;31" 2022-03-26T17:37:45.889 INFO:teuthology.orchestra.run.smithi124.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-26T17:37:45.889 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://www.redhat.com/" 2022-03-26T17:37:45.889 INFO:teuthology.orchestra.run.smithi124.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-26T17:37:45.890 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-26T17:37:45.890 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-03-26T17:37:45.890 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-26T17:37:45.890 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-26T17:37:45.891 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-26T17:37:45.891 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-26T17:37:45.892 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2022-03-26T17:37:45.910 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2022-03-26T17:37:45.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-03-26T17:37:46.134 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2022-03-26T17:37:46.188 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Red Hat Enterprise Linux" 2022-03-26T17:37:46.188 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8.4 (Ootpa)" 2022-03-26T17:37:46.189 INFO:teuthology.orchestra.run.smithi142.stdout:ID="rhel" 2022-03-26T17:37:46.189 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="fedora" 2022-03-26T17:37:46.189 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8.4" 2022-03-26T17:37:46.189 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2022-03-26T17:37:46.190 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-26T17:37:46.190 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2022-03-26T17:37:46.190 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-26T17:37:46.190 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.redhat.com/" 2022-03-26T17:37:46.191 INFO:teuthology.orchestra.run.smithi142.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-26T17:37:46.191 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-26T17:37:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-03-26T17:37:46.191 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-26T17:37:46.191 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-26T17:37:46.192 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-26T17:37:46.192 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-26T17:37:46.193 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2022-03-26T17:37:46.286 INFO:teuthology.dispatcher.supervisor:Running job 6757518 2022-03-26T17:37:46.286 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757518 --name vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757518/orig.config.yaml 2022-03-26T17:37:46.289 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757518 2022-03-26T17:37:46.290 INFO:teuthology.dispatcher.supervisor:Job PID: 5219 2022-03-26T17:37:46.290 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-27T00:08:14.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-27T00:08:14.387 INFO:teuthology.kill:Killing Pids: {5219} 2022-03-27T00:08:15.519 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-03-27T00:08:15.520 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-03-27T00:08:15.520 INFO:teuthology.misc:Compressing logs... 2022-03-27T00:08:15.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2022-03-27T00:08:15.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-03-27T00:08:16.254 INFO:teuthology.misc:Compressing logs... 2022-03-27T00:08:16.425 INFO:teuthology.misc:Compressing logs... 2022-03-27T00:18:56.073 INFO:teuthology.kill:No teuthology processes running 2022-03-27T00:18:57.202 INFO:teuthology.kill:Nuking machines: ['smithi124', 'smithi142'] 2022-03-27T00:18:58.153 INFO:teuthology.kill:2022-03-27 00:18:58,153.153 INFO:teuthology.nuke:targets: 2022-03-27T00:18:58.154 INFO:teuthology.kill: smithi124.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7BE1qLdeeZhvosXho+phfFOOsqIRiLpx6XrLQYQgrsKsRmeB6Bq9eL868P+lbj9CHMPHoMF4TLMZNdttQR1luuraetud4zxncIw1Jbcj9HLfwdRy4tF0UQeuDG25G8PpXV/dyyk3hsS5giLaSnjes/CY+9PR0X4Zc203WrZ3E6A9y6ZNh23jQwIk0kp/kNhCJI08BPGtEbp5JqEqcN6dx94TKLD29A8+7owaP660yipZcDlc6bdNGMisxZsCYpy6VVOfN6YK0YOG5Yf7PqoOaIuKwa0zqNAnODRtkMQ+keUgt3pfrkceDncuyyMDWf+/MIGzfPekNtxU7tjyb04KV5rQMQDkhbDyuBRmPoeX9h63QwLBLYB72WWOEjAFhddlM4IFmEDTWX6rU80Su4kRLIE6RWhclbfnVUQGbqgFuczvexipj8ohCnZp1CYDx8Q55f+Z2QdWS4Tzm5GWcizQ7H51owbQRt+MksZvNf+vFY+dg7eqcU6KlL7AfAyKJpiM= 2022-03-27T00:18:58.154 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+38Y/3CLAyfR+pUAhEIg+a47zlFU+Y2QuXCrdNcaHqGcFRLYltUbGnS7zCwVgzy4z6BURJmdxhswjoIIlMZ0LHqslXTUXt/MN3+fiol/cjMNLTtrXbx3ju4kNYwG0WVq2zMcHp0enFJurJaavs6S3+9/ilsKUroU/ObRl7Eezt6/a91Qj4l0NYfGLOTJZU/Juw42n0ohvdUGwRjyZ5R7xJ8SritR+kBUQNfS+rrD6XBac8N2+NDlwKz6dbI8PLYjZgk7qH5FhiykVBmBO0csszvaGcLdP4pkD9UqKLmGLUA/QKgOegdkOv8KebChs6X0Odkbh+G+4zrGxOryp7rk4U1csfSaysSzGCfPyiZ2vd8CYh52TJLR4UY2vCap7aM9gyxDqFyTqgy5U2EPM3s6TaEBkQdyGPd1UyxYcI7EDKaF9WQpBLSe7DbVOE13336QQhKckJ/W0LdG5/0vO574r45FF9FhssPGjkQdKjcss4O/iTB0bQR+EVyaQX+IFpxs= 2022-03-27T00:18:58.180 INFO:teuthology.kill:2022-03-27 00:18:58,179.179 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-27T00:18:58.182 INFO:teuthology.kill:2022-03-27 00:18:58,182.182 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-27T00:18:58.256 INFO:teuthology.kill:2022-03-27 00:18:58,256.256 INFO:teuthology.orchestra.console:Power off smithi124 2022-03-27T00:18:58.269 INFO:teuthology.kill:2022-03-27 00:18:58,269.269 INFO:teuthology.orchestra.console:Power off smithi142 2022-03-27T00:19:06.632 INFO:teuthology.kill:2022-03-27 00:19:06,632.632 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2022-03-27T00:19:06.778 INFO:teuthology.kill:2022-03-27 00:19:06,777.777 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com 2022-03-27T00:19:10.739 INFO:teuthology.kill:2022-03-27 00:19:10,738.738 INFO:teuthology.orchestra.console:Power off for smithi124 completed 2022-03-27T00:19:10.881 INFO:teuthology.kill:2022-03-27 00:19:10,881.881 INFO:teuthology.lock.ops:unlocked: smithi124.front.sepia.ceph.com 2022-03-27T00:21:11.171 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-27T00:21:11.282 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757624