2022-03-14T16:21:13.311 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-14T16:21:13.428 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi006.front.sepia.ceph.com' reimaging 2022-03-14T16:21:13.429 INFO:teuthology.lock.ops:Updating [ubuntu@smithi006.front.sepia.ceph.com]: reset os type and version on server 2022-03-14T16:21:13.430 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2022-03-14T16:21:13.448 INFO:teuthology.lock.ops:Node 'ubuntu@smithi006.front.sepia.ceph.com' reimaging is complete 2022-03-14T16:21:13.449 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi188.front.sepia.ceph.com' reimaging 2022-03-14T16:21:13.449 INFO:teuthology.lock.ops:Updating [ubuntu@smithi188.front.sepia.ceph.com]: reset os type and version on server 2022-03-14T16:21:13.449 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2022-03-14T16:21:13.468 INFO:teuthology.lock.ops:Node 'ubuntu@smithi188.front.sepia.ceph.com' reimaging is complete 2022-03-14T16:21:13.665 INFO:teuthology.provision.fog.smithi006:Scheduling deploy of rhel 8.4 2022-03-14T16:21:13.672 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of rhel 8.4 2022-03-14T16:21:13.951 INFO:teuthology.orchestra.console:Power off smithi006 2022-03-14T16:21:13.965 INFO:teuthology.orchestra.console:Power off smithi188 2022-03-14T16:21:14.092 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2022-03-14T16:21:14.102 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2022-03-14T16:21:14.194 INFO:teuthology.orchestra.console:Power on smithi006 2022-03-14T16:21:14.203 INFO:teuthology.orchestra.console:Power on smithi188 2022-03-14T16:21:22.557 INFO:teuthology.orchestra.console:Power on for smithi006 completed 2022-03-14T16:21:22.562 INFO:teuthology.orchestra.console:Power on for smithi188 completed 2022-03-14T16:21:22.659 INFO:teuthology.provision.fog.smithi006:Waiting for deploy to finish 2022-03-14T16:21:22.663 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish 2022-03-14T16:25:36.562 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-14T16:25:36.769 INFO:teuthology.provision.fog.smithi006:Node is ready 2022-03-14T16:25:36.786 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006.front.sepia.ceph.com 2022-03-14T16:25:36.842 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6 smithi006.front.sepia.ceph.com smithi006 2022-03-14T16:25:37.175 INFO:teuthology.provision.fog.smithi006:Deploy complete! 2022-03-14T16:26:03.419 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-14T16:26:03.638 INFO:teuthology.provision.fog.smithi188:Node is ready 2022-03-14T16:26:03.671 INFO:teuthology.orchestra.run.smithi188.stdout:smithi188.front.sepia.ceph.com 2022-03-14T16:26:03.751 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.15.188 smithi188.front.sepia.ceph.com smithi188 2022-03-14T16:26:04.250 INFO:teuthology.provision.fog.smithi188:Deploy complete! 2022-03-14T16:26:04.497 INFO:teuthology.lock.ops:Checking smithi006.front.sepia.ceph.com 2022-03-14T16:26:04.499 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-14T16:26:04.526 INFO:teuthology.lock.ops:Checking smithi188.front.sepia.ceph.com 2022-03-14T16:26:04.527 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-14T16:26:04.559 INFO:teuthology.lock.ops:Updating [smithi006.front.sepia.ceph.com]: set os type and version on server 2022-03-14T16:26:04.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2022-03-14T16:26:04.860 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2022-03-14T16:26:04.919 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Red Hat Enterprise Linux" 2022-03-14T16:26:04.920 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="8.4 (Ootpa)" 2022-03-14T16:26:04.920 INFO:teuthology.orchestra.run.smithi006.stdout:ID="rhel" 2022-03-14T16:26:04.920 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="fedora" 2022-03-14T16:26:04.921 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="8.4" 2022-03-14T16:26:04.921 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el8" 2022-03-14T16:26:04.921 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-14T16:26:04.921 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2022-03-14T16:26:04.922 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-14T16:26:04.922 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.redhat.com/" 2022-03-14T16:26:04.922 INFO:teuthology.orchestra.run.smithi006.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-14T16:26:04.923 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-14T16:26:04.923 INFO:teuthology.orchestra.run.smithi006.stdout: 2022-03-14T16:26:04.924 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-14T16:26:04.924 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-14T16:26:04.924 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-14T16:26:04.925 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-14T16:26:04.927 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2022-03-14T16:26:04.949 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: set os type and version on server 2022-03-14T16:26:04.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2022-03-14T16:26:05.254 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2022-03-14T16:26:05.315 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Red Hat Enterprise Linux" 2022-03-14T16:26:05.316 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8.4 (Ootpa)" 2022-03-14T16:26:05.316 INFO:teuthology.orchestra.run.smithi188.stdout:ID="rhel" 2022-03-14T16:26:05.317 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="fedora" 2022-03-14T16:26:05.317 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8.4" 2022-03-14T16:26:05.317 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2022-03-14T16:26:05.318 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-14T16:26:05.318 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2022-03-14T16:26:05.318 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-14T16:26:05.318 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.redhat.com/" 2022-03-14T16:26:05.319 INFO:teuthology.orchestra.run.smithi188.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-14T16:26:05.319 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-14T16:26:05.319 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-03-14T16:26:05.319 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-14T16:26:05.320 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-14T16:26:05.320 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-14T16:26:05.320 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-14T16:26:05.321 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2022-03-14T16:26:05.454 INFO:teuthology.dispatcher.supervisor:Running job 6735926 2022-03-14T16:26:05.455 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-14_08:09:40-fs-wip-vshankar-testing1-20220314-103400-testing-default-smithi/6735926 --name vshankar-2022-03-14_08:09:40-fs-wip-vshankar-testing1-20220314-103400-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} -- /home/teuthworker/archive/vshankar-2022-03-14_08:09:40-fs-wip-vshankar-testing1-20220314-103400-testing-default-smithi/6735926/orig.config.yaml 2022-03-14T16:26:05.459 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-14_08:09:40-fs-wip-vshankar-testing1-20220314-103400-testing-default-smithi/6735926 2022-03-14T16:26:05.460 INFO:teuthology.dispatcher.supervisor:Job PID: 9214 2022-03-14T16:26:05.461 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-14T22:56:34.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-14T22:56:34.541 INFO:teuthology.kill:Killing Pids: {9214} 2022-03-14T22:56:35.770 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-03-14T22:56:35.771 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-03-14T22:56:35.771 INFO:teuthology.misc:Compressing logs... 2022-03-14T22:56:35.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2022-03-14T22:56:36.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2022-03-14T22:56:36.403 INFO:teuthology.misc:Compressing logs... 2022-03-14T22:56:36.624 INFO:teuthology.misc:Compressing logs... 2022-03-14T22:58:36.526 INFO:teuthology.kill:No teuthology processes running 2022-03-14T22:58:37.700 INFO:teuthology.kill:Nuking machines: ['smithi006', 'smithi188'] 2022-03-14T22:58:38.660 INFO:teuthology.kill:2022-03-14 22:58:38,660.660 INFO:teuthology.nuke:targets: 2022-03-14T22:58:38.661 INFO:teuthology.kill: smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3/0wiystkH638hXGQindmcSW1tmWQPaDnc9ccsMUlnx9roWOpSNHCOooSUA06dMDPQ8jKTBmbElQ/P5+k20epREejYa8meRIdTUzvCM71PHrOWnDZn4G2coBqxVJ3wix0KxWiJJkWN/3LjSk6B510GpFxrVVku935eOb3NJ0CXQAp/1ktdmyoNzpcvnWuheRWq/Nz2rsuw8ZHtHbzRdhFF2s6x6CNlkHSxHTQ4tv4DNw1+1pgLkmYwcnqsGm/o+2GoLg6AI0NHOAR0B9dWQyPXn3f9WXOteQc45wrr2QWbgOokrOc1bxx0L+/Gigwer1l4hI+A+qyRpICeXXVsAs+enUBHUEJqQNcf0RN4BzS9eWkh0J0/fWI8uGz4vka3eUFJ8kiCYc9v1j411hFAsTHYw0inZ6/N7aSJu287MDgJIxPRCOarduBRaGWpSguD3Iq4PX/aPrdnEY70uJcvJ0HMCdDyv0gInSi8jOmV8A0APHmVw08wrwAiG8zyi0p7Ec= 2022-03-14T22:58:38.662 INFO:teuthology.kill: smithi188.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrRWZaPLG6FTFddZXJceutaYDMYSPHdv711GhnIG7MW7jQvXPv3AuO4UOV6wCJjywh2eSvBxZmRuSLMubYBmmXit02kMXbqdG+zPOMJBUBgHChh1MAhNOPsSI1gMd/FczhK05QbwnLptb+LaNQ4URqKtzLUg+Yk77ZHzy6Cy/eUK3LCDUxFwEXblQi2D7fauXIhsZnw1YErhMOA+i0ENPrgQmMYFMNn2MepCs5o2Xun0IgUupuU7wn7ZWPgy7frmPX3MlxM/fKpgx4NYY0hcF79F22oT914HoaZtTaBVmX6jSoY+nMy4a1t5Dl7fXTSwqcjkOe4f79wmoPcae+Y1rCL1ZYBxusK900XXUk58SM05tuYHAwKaRYYvutqVeOB+Dq4gTl8c1YcHDwYjLJiy8uVzdHMjNjAoEYLiKTktWRslr7BGyMMSwX/IqBwaXaXwG6ibMCP6nEtyEq6XlRHw8aBD9VjrM5dKLguwsM8aXoJZmuxt7vEizyOjdv6QNJX+M= 2022-03-14T22:58:38.687 INFO:teuthology.kill:2022-03-14 22:58:38,687.687 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-14T22:58:38.691 INFO:teuthology.kill:2022-03-14 22:58:38,691.691 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-14T22:58:38.761 INFO:teuthology.kill:2022-03-14 22:58:38,761.761 INFO:teuthology.orchestra.console:Power off smithi188 2022-03-14T22:58:38.774 INFO:teuthology.kill:2022-03-14 22:58:38,773.773 INFO:teuthology.orchestra.console:Power off smithi006 2022-03-14T22:58:47.131 INFO:teuthology.kill:2022-03-14 22:58:47,130.130 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2022-03-14T22:58:47.275 INFO:teuthology.kill:2022-03-14 22:58:47,275.275 INFO:teuthology.lock.ops:unlocked: smithi188.front.sepia.ceph.com 2022-03-14T22:58:51.250 INFO:teuthology.kill:2022-03-14 22:58:51,250.250 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2022-03-14T22:58:51.396 INFO:teuthology.kill:2022-03-14 22:58:51,396.396 INFO:teuthology.lock.ops:unlocked: smithi006.front.sepia.ceph.com 2022-03-14T23:00:51.684 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-14T23:00:51.809 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi006 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-14_18:49:17-rados-wip-yuri2-testing-2022-03-14-0946-quincy-distro-default-smithi/6736310 2022-03-14T23:00:51.809 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi188 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-14_18:49:17-rados-wip-yuri2-testing-2022-03-14-0946-quincy-distro-default-smithi/6736310