2022-03-18T11:27:55.814 INFO:root:teuthology version: 1.1.0-aa078dbe 2022-03-18T11:27:55.938 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging 2022-03-18T11:27:55.939 INFO:teuthology.lock.ops:Updating [ubuntu@smithi121.front.sepia.ceph.com]: reset os type and version on server 2022-03-18T11:27:55.940 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2022-03-18T11:27:55.959 INFO:teuthology.lock.ops:Node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging is complete 2022-03-18T11:27:55.960 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi151.front.sepia.ceph.com' reimaging 2022-03-18T11:27:55.960 INFO:teuthology.lock.ops:Updating [ubuntu@smithi151.front.sepia.ceph.com]: reset os type and version on server 2022-03-18T11:27:55.960 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2022-03-18T11:27:55.979 INFO:teuthology.lock.ops:Node 'ubuntu@smithi151.front.sepia.ceph.com' reimaging is complete 2022-03-18T11:27:56.170 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of centos 8.stream 2022-03-18T11:27:56.187 INFO:teuthology.provision.fog.smithi151:Scheduling deploy of centos 8.stream 2022-03-18T11:27:56.531 INFO:teuthology.orchestra.console:Power off smithi121 2022-03-18T11:27:56.569 INFO:teuthology.orchestra.console:Power off smithi151 2022-03-18T11:28:04.930 INFO:teuthology.orchestra.console:Power off for smithi151 completed 2022-03-18T11:28:05.031 INFO:teuthology.orchestra.console:Power on smithi151 2022-03-18T11:28:09.017 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2022-03-18T11:28:09.119 INFO:teuthology.orchestra.console:Power on smithi121 2022-03-18T11:28:09.281 INFO:teuthology.orchestra.console:Power on for smithi151 completed 2022-03-18T11:28:09.382 INFO:teuthology.provision.fog.smithi151:Waiting for deploy to finish 2022-03-18T11:28:13.370 INFO:teuthology.orchestra.console:Power on for smithi121 completed 2022-03-18T11:28:13.471 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish 2022-03-18T11:32:57.902 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-18T11:32:58.136 INFO:teuthology.provision.fog.smithi121:Node is ready 2022-03-18T11:32:58.195 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121.front.sepia.ceph.com 2022-03-18T11:32:58.252 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121 2022-03-18T11:32:58.593 INFO:teuthology.provision.fog.smithi121:Deploy complete! 2022-03-18T11:33:02.864 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-18T11:33:03.105 INFO:teuthology.provision.fog.smithi151:Node is ready 2022-03-18T11:33:03.122 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151.front.sepia.ceph.com 2022-03-18T11:33:03.180 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151 smithi151.front.sepia.ceph.com smithi151 2022-03-18T11:33:03.512 INFO:teuthology.provision.fog.smithi151:Deploy complete! 2022-03-18T11:33:03.675 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com 2022-03-18T11:33:03.676 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-18T11:33:03.695 INFO:teuthology.lock.ops:Checking smithi151.front.sepia.ceph.com 2022-03-18T11:33:03.695 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-18T11:33:03.714 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server 2022-03-18T11:33:03.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-03-18T11:33:03.910 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2022-03-18T11:33:03.969 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2022-03-18T11:33:03.970 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8" 2022-03-18T11:33:03.970 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2022-03-18T11:33:03.970 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2022-03-18T11:33:03.971 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8" 2022-03-18T11:33:03.971 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2022-03-18T11:33:03.971 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-18T11:33:03.971 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2022-03-18T11:33:03.972 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-18T11:33:03.972 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2022-03-18T11:33:03.972 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-18T11:33:03.972 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-18T11:33:03.973 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-18T11:33:03.974 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2022-03-18T11:33:03.993 INFO:teuthology.lock.ops:Updating [smithi151.front.sepia.ceph.com]: set os type and version on server 2022-03-18T11:33:03.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2022-03-18T11:33:04.179 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2022-03-18T11:33:04.235 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2022-03-18T11:33:04.236 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="8" 2022-03-18T11:33:04.236 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2022-03-18T11:33:04.236 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2022-03-18T11:33:04.237 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="8" 2022-03-18T11:33:04.237 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el8" 2022-03-18T11:33:04.237 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-18T11:33:04.238 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2022-03-18T11:33:04.238 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-18T11:33:04.238 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2022-03-18T11:33:04.238 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-18T11:33:04.238 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-18T11:33:04.239 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-18T11:33:04.239 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2022-03-18T11:33:04.341 INFO:teuthology.dispatcher.supervisor:Running job 6744586 2022-03-18T11:33:04.341 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_aa078dbed6d14c1ad98beb5a02b956bbe0a741c8/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744586 --name vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} -- /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744586/orig.config.yaml 2022-03-18T11:33:04.345 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744586 2022-03-18T11:33:04.345 INFO:teuthology.dispatcher.supervisor:Job PID: 14525 2022-03-18T11:33:04.346 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-18T18:03:32.354 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-18T18:03:32.535 INFO:teuthology.kill:Killing Pids: {14525} 2022-03-18T18:03:33.705 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-03-18T18:03:33.706 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-03-18T18:03:33.706 INFO:teuthology.misc:Compressing logs... 2022-03-18T18:03:33.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-03-18T18:03:37.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2022-03-18T18:03:37.852 INFO:teuthology.misc:Compressing logs... 2022-03-18T18:03:38.106 INFO:teuthology.misc:Compressing logs... 2022-03-18T18:17:06.010 INFO:teuthology.kill:No teuthology processes running 2022-03-18T18:17:07.145 INFO:teuthology.kill:Nuking machines: ['smithi121', 'smithi151'] 2022-03-18T18:17:08.118 INFO:teuthology.kill:2022-03-18 18:17:08,118.118 INFO:teuthology.nuke:targets: 2022-03-18T18:17:08.119 INFO:teuthology.kill: smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChoHaiR65UcE76rUCP8fOhU12fJMu6W4GOvANeFrx50cD4e5p/tc4JMUGwrZSPyeMqn1VbXGziFzJ/5K2L2fHkIzSy7r7pJc+wMBmnUH1j/vMr5Q3zs83YCpLM+n6btQQnfxBJUbNWz6fJQeFSmCcQ/wgJ525BpKF5guFV6OGMKfnR+5n8GFTklgtpmtRx9T6Qu+8pTok1ry96A4kTWPaEEY0lBVNtNOBjQ0KJOCWKOjHg9/fwfmgsnS9Gp4f/WQgUU9xGfITEQhKhplirFWv1g646OOK+hb6KagIGFZ4j7S3PVqKgEoCdzYkP527llzjQQBLSs492pQm7bvPtxNQ/GtfkHd4hAsg5ZIdR1WxkNFuDbWIiAn5BoMLeKru8QiGJZeT1dyc9JefV+QNKRrcnnNkWK2cMdEWl/QydP/R99St53XpvLP95/NH7RPcQy/cojq0MQBWFi+EkFFnC0V0FHiYQgkXLLtvAFFVBLMABKUx7jLLqfFiTXUyXWd3XNBM= 2022-03-18T18:17:08.120 INFO:teuthology.kill: smithi151.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdg1St22UWu/YHCjE7FE/3dTARCEP5uC5lrufMLQ00p3ZwNE6c/ktDxl63ulXSHve4J4gANriekkjvurUvK1nCAtd81DSN+xGcvXPyOKG8hssrd+ZrwhaI5e31vn39gwyLKAe+//EOdhl8qn32vkRvEVKoHhOLxZh0l8MNSOsgPBOInGgR9Nm0HLXNGT9bKok151wocqDV5nTh00fi1VFLKpumfOxNvm2dmoFOMK8y130Js/k5PpkORHeny1ZGxvqevTVz4dgS6xD15fXQuVH83vSrHmIsHIUWhFA5cMlim7/NB3znRD8hMFtyAqagZvy5jWksaom46+0fgCxxocUX1NYmr/cf70LjWvvP5eYeNpwLxw4aWwB5mIqUVNtXTgh24jaEZM+WeBc7oZpzR5GpTCfuxTHHWbo9Gjj+3o0DJ7lE213tEZfvX1ip8kmaldwQIk7SVOmpF6ichbsuW7x5CS7APC2v1xgNJevIhWTeY+dWhB9n7PQhnMY+y/sQTOM= 2022-03-18T18:17:08.145 INFO:teuthology.kill:2022-03-18 18:17:08,144.144 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-18T18:17:08.148 INFO:teuthology.kill:2022-03-18 18:17:08,147.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-18T18:17:08.223 INFO:teuthology.kill:2022-03-18 18:17:08,223.223 INFO:teuthology.orchestra.console:Power off smithi121 2022-03-18T18:17:08.238 INFO:teuthology.kill:2022-03-18 18:17:08,237.237 INFO:teuthology.orchestra.console:Power off smithi151 2022-03-18T18:17:16.597 INFO:teuthology.kill:2022-03-18 18:17:16,597.597 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2022-03-18T18:17:16.747 INFO:teuthology.kill:2022-03-18 18:17:16,746.746 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com 2022-03-18T18:17:20.715 INFO:teuthology.kill:2022-03-18 18:17:20,715.715 INFO:teuthology.orchestra.console:Power off for smithi151 completed 2022-03-18T18:17:20.860 INFO:teuthology.kill:2022-03-18 18:17:20,860.860 INFO:teuthology.lock.ops:unlocked: smithi151.front.sepia.ceph.com 2022-03-18T18:19:21.186 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-18T18:19:21.297 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi121 but it was locked by another job: /home/teuthworker/archive/mrfreezeex-2022-03-18_14:52:08-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745251 2022-03-18T18:19:21.297 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi151 but it was locked by another job: /home/teuthworker/archive/mrfreezeex-2022-03-18_14:52:08-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745250