2022-03-20T10:41:52.905 INFO:root:teuthology version: 1.1.0-a4e18662
2022-03-20T10:41:53.028 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging
2022-03-20T10:41:53.029 INFO:teuthology.lock.ops:Updating [ubuntu@smithi090.front.sepia.ceph.com]: reset os type and version on server
2022-03-20T10:41:53.030 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2022-03-20T10:41:53.047 INFO:teuthology.lock.ops:Node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging is complete
2022-03-20T10:41:53.047 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging
2022-03-20T10:41:53.048 INFO:teuthology.lock.ops:Updating [ubuntu@smithi100.front.sepia.ceph.com]: reset os type and version on server
2022-03-20T10:41:53.048 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server
2022-03-20T10:41:53.067 INFO:teuthology.lock.ops:Node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging is complete
2022-03-20T10:41:53.278 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of rhel 8.4
2022-03-20T10:41:53.283 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of rhel 8.4
2022-03-20T10:41:53.588 INFO:teuthology.orchestra.console:Power off smithi100
2022-03-20T10:41:53.630 INFO:teuthology.orchestra.console:Power off smithi090
2022-03-20T10:42:01.969 INFO:teuthology.orchestra.console:Power off for smithi100 completed
2022-03-20T10:42:02.070 INFO:teuthology.orchestra.console:Power on smithi100
2022-03-20T10:42:06.100 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2022-03-20T10:42:06.202 INFO:teuthology.orchestra.console:Power on smithi090
2022-03-20T10:42:10.452 INFO:teuthology.orchestra.console:Power on for smithi090 completed
2022-03-20T10:42:10.501 INFO:teuthology.orchestra.console:Power on for smithi100 completed
2022-03-20T10:42:10.554 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2022-03-20T10:42:10.602 INFO:teuthology.provision.fog.smithi100:Waiting for deploy to finish
2022-03-20T10:46:16.764 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-20T10:46:31.734 INFO:teuthology.provision.fog.smithi100:Node is ready
2022-03-20T10:46:31.753 INFO:teuthology.orchestra.run.smithi100.stdout:smithi093
2022-03-20T10:46:31.811 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.15.93 smithi093.front.sepia.ceph.com smithi093
2022-03-20T10:46:32.168 INFO:teuthology.provision.fog.smithi100:Deploy complete!
2022-03-20T10:47:45.628 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-20T10:47:45.861 INFO:teuthology.provision.fog.smithi090:Node is ready
2022-03-20T10:47:45.876 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com
2022-03-20T10:47:45.936 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090
2022-03-20T10:47:46.285 INFO:teuthology.provision.fog.smithi090:Deploy complete!
2022-03-20T10:47:46.451 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com
2022-03-20T10:47:46.452 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-20T10:47:46.478 INFO:teuthology.lock.ops:Checking smithi100.front.sepia.ceph.com
2022-03-20T10:47:46.478 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-20T10:47:46.495 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server
2022-03-20T10:47:46.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2022-03-20T10:47:46.717 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64
2022-03-20T10:47:46.770 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Red Hat Enterprise Linux"
2022-03-20T10:47:46.770 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8.4 (Ootpa)"
2022-03-20T10:47:46.771 INFO:teuthology.orchestra.run.smithi090.stdout:ID="rhel"
2022-03-20T10:47:46.771 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="fedora"
2022-03-20T10:47:46.771 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8.4"
2022-03-20T10:47:46.771 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8"
2022-03-20T10:47:46.772 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-03-20T10:47:46.772 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31"
2022-03-20T10:47:46.772 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-03-20T10:47:46.772 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.redhat.com/"
2022-03-20T10:47:46.772 INFO:teuthology.orchestra.run.smithi090.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-03-20T10:47:46.773 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-20T10:47:46.773 INFO:teuthology.orchestra.run.smithi090.stdout:
2022-03-20T10:47:46.773 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-20T10:47:46.773 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-03-20T10:47:46.773 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-03-20T10:47:46.774 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-03-20T10:47:46.775 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2022-03-20T10:47:46.794 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: set os type and version on server
2022-03-20T10:47:46.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com'
2022-03-20T10:47:47.028 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64
2022-03-20T10:47:47.083 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="Red Hat Enterprise Linux"
2022-03-20T10:47:47.083 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8.4 (Ootpa)"
2022-03-20T10:47:47.084 INFO:teuthology.orchestra.run.smithi100.stdout:ID="rhel"
2022-03-20T10:47:47.084 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="fedora"
2022-03-20T10:47:47.084 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8.4"
2022-03-20T10:47:47.084 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8"
2022-03-20T10:47:47.084 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-03-20T10:47:47.085 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31"
2022-03-20T10:47:47.085 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-03-20T10:47:47.086 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://www.redhat.com/"
2022-03-20T10:47:47.086 INFO:teuthology.orchestra.run.smithi100.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-03-20T10:47:47.086 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-20T10:47:47.087 INFO:teuthology.orchestra.run.smithi100.stdout:
2022-03-20T10:47:47.088 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-20T10:47:47.088 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-03-20T10:47:47.088 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-03-20T10:47:47.088 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-03-20T10:47:47.089 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server
2022-03-20T10:47:47.185 INFO:teuthology.dispatcher.supervisor:Running job 6747744
2022-03-20T10:47:47.186 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747744 --name vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} -- /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747744/orig.config.yaml
2022-03-20T10:47:47.190 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747744
2022-03-20T10:47:47.191 INFO:teuthology.dispatcher.supervisor:Job PID: 25308
2022-03-20T10:47:47.191 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-20T17:18:15.051 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-20T17:18:15.236 INFO:teuthology.kill:Killing Pids: {25308}
2022-03-20T17:18:16.424 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-03-20T17:18:16.425 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-03-20T17:18:16.425 INFO:teuthology.misc:Compressing logs...
2022-03-20T17:18:16.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2022-03-20T17:18:16.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com'
2022-03-20T17:18:18.548 INFO:teuthology.misc:Compressing logs...
2022-03-20T17:18:18.855 INFO:teuthology.misc:Compressing logs...
2022-03-20T17:30:45.056 INFO:teuthology.kill:No teuthology processes running
2022-03-20T17:30:46.249 INFO:teuthology.kill:Nuking machines: ['smithi090', 'smithi100']
2022-03-20T17:30:47.197 INFO:teuthology.kill:2022-03-20 17:30:47,197.197 INFO:teuthology.nuke:targets:
2022-03-20T17:30:47.198 INFO:teuthology.kill:  smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC40lBz0HRwgA7L+SSRoL72RYe05t/oGj/eQYv5Pa84/3iAZYU+b4fAeQqlydZwiLc6vT6ea8ID91RfQet9LmMsYxRQe4B6RJk/y8chEn996TlDKQJUqfdk6WCpyVTvAhTppm5PYSW+S/lxjPywOVCUpfVDcuuuD5+iuJtQcSUdOTlBiCvj0h8ZfScWHT4rV7AuPCBQRAwIsIUSz5pnFw2coNY49FtGBemIPMGH/ukUTE8SeL+r5v4igsMqh1mgphNSx+HNAfk0L7Fvr3PIMcRQp/Q1Vgut5Mx/Vh3BLg7FYcJOi0oEAm8ILC/XJtGwthtfrUn4AUzj74mtUA6eFKsbHGWPQgvWVnE/3JNGoMdLPqPKYn73w4B6ZQj4iJnfIcAqYctJUsP/mkJNTpVTDcJ6KacvlBqYvTaYUtK5aavx2xWi28DqRwFsGHO+TM1S2O6nZec3z3SJBGK7FR3wMGckcg1dAb5DOTtS6WwF1l9xPMF07ry9VBwW8PGSA/7C18c=
2022-03-20T17:30:47.199 INFO:teuthology.kill:  smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4ZTwAKO4EtoJ35vbp0W+paGlkuDZefEH0iXcYQ+nuIW38vqmwklZwGcWieyuPHStQkrBZLLl+Ws9uJQZRrml0QZcd1jwU5Ddo5Z3BRFPQm639aSX4vtolUzZVcarmkyne6nM6utOa0HqFInaBJdQjG8IxTB0+CLJ37ojvE0aI24wEmWU+Mjb7rUy/NoDhVXknvMmbPmlD9CKwe3EKoQQygPMxQCXWCWTqj9DO5vx/4Qmi54EFl117AG4W6o5ae023zSjzJp0NzY4znL4AomI73gWA6Obp8A58+PWL3IY1qZwDY9oc6fJxn6Y9p3YmQsSX3BkUes0kirenu+xa31AsnvERN9IhxRgdABsJSqkHsHzSO5oH8/e6D0pisxLc+a+CTDDYKiXDnAOGMjpc9y1M84cSKrc+e1CgGE6bLb82ijLGuhs90F9wv1J+pYWD8Dl4X8lb8W1NFiVVkLPnTiBgoZ2ly2pvORrGCKJFREhOhi44SvD006oWjTfYNPanMS8=
2022-03-20T17:30:47.224 INFO:teuthology.kill:2022-03-20 17:30:47,224.224 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-20T17:30:47.227 INFO:teuthology.kill:2022-03-20 17:30:47,227.227 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-20T17:30:47.304 INFO:teuthology.kill:2022-03-20 17:30:47,303.303 INFO:teuthology.orchestra.console:Power off smithi090
2022-03-20T17:30:47.317 INFO:teuthology.kill:2022-03-20 17:30:47,317.317 INFO:teuthology.orchestra.console:Power off smithi100
2022-03-20T17:30:59.779 INFO:teuthology.kill:2022-03-20 17:30:59,779.779 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2022-03-20T17:30:59.787 INFO:teuthology.kill:2022-03-20 17:30:59,787.787 INFO:teuthology.orchestra.console:Power off for smithi100 completed
2022-03-20T17:30:59.928 INFO:teuthology.kill:2022-03-20 17:30:59,928.928 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com
2022-03-20T17:30:59.937 INFO:teuthology.kill:2022-03-20 17:30:59,937.937 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com
2022-03-20T17:33:00.241 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-20T17:33:00.357 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi090 but it was locked by another job: /home/teuthworker/archive/gsalomon-2022-03-20_17:26:33-rgw:verify-wip-test-arrow-submodule-distro-basic-smithi/6749041
2022-03-20T17:33:00.357 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi100 but it was locked by another job: /home/teuthworker/archive/gsalomon-2022-03-20_17:26:33-rgw:verify-wip-test-arrow-submodule-distro-basic-smithi/6749041