2022-03-19T05:29:25.215 INFO:root:teuthology version: 1.1.0-aa078dbe 2022-03-19T05:29:25.340 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi115.front.sepia.ceph.com' reimaging 2022-03-19T05:29:25.341 INFO:teuthology.lock.ops:Updating [ubuntu@smithi115.front.sepia.ceph.com]: reset os type and version on server 2022-03-19T05:29:25.341 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2022-03-19T05:29:25.359 INFO:teuthology.lock.ops:Node 'ubuntu@smithi115.front.sepia.ceph.com' reimaging is complete 2022-03-19T05:29:25.359 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging 2022-03-19T05:29:25.360 INFO:teuthology.lock.ops:Updating [ubuntu@smithi152.front.sepia.ceph.com]: reset os type and version on server 2022-03-19T05:29:25.360 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2022-03-19T05:29:25.377 INFO:teuthology.lock.ops:Node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging is complete 2022-03-19T05:29:25.567 INFO:teuthology.provision.fog.smithi115:Scheduling deploy of ubuntu 20.04 2022-03-19T05:29:25.574 INFO:teuthology.provision.fog.smithi152:Scheduling deploy of ubuntu 20.04 2022-03-19T05:29:25.865 INFO:teuthology.orchestra.console:Power off smithi115 2022-03-19T05:29:25.900 INFO:teuthology.orchestra.console:Power off smithi152 2022-03-19T05:29:34.236 INFO:teuthology.orchestra.console:Power off for smithi115 completed 2022-03-19T05:29:34.337 INFO:teuthology.orchestra.console:Power on smithi115 2022-03-19T05:29:38.370 INFO:teuthology.orchestra.console:Power off for smithi152 completed 2022-03-19T05:29:38.472 INFO:teuthology.orchestra.console:Power on smithi152 2022-03-19T05:29:42.697 INFO:teuthology.orchestra.console:Power on for smithi115 completed 2022-03-19T05:29:42.799 INFO:teuthology.provision.fog.smithi115:Waiting for deploy to finish 2022-03-19T05:29:46.834 INFO:teuthology.orchestra.console:Power on for smithi152 completed 2022-03-19T05:29:46.935 INFO:teuthology.provision.fog.smithi152:Waiting for deploy to finish 2022-03-19T05:32:43.937 ERROR:teuthology.orchestra.connection:Error authenticating with smithi115.front.sepia.ceph.com: Authentication failed. 2022-03-19T05:34:58.601 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-19T05:34:59.049 INFO:teuthology.provision.fog.smithi115:Node is ready 2022-03-19T05:34:59.055 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115.front.sepia.ceph.com 2022-03-19T05:34:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115 smithi115.front.sepia.ceph.com smithi115 2022-03-19T05:34:59.244 INFO:teuthology.provision.fog.smithi115:Deploy complete! 2022-03-19T05:35:02.182 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-19T05:35:02.608 INFO:teuthology.provision.fog.smithi152:Node is ready 2022-03-19T05:35:02.613 INFO:teuthology.orchestra.run.smithi152.stdout:smithi152.front.sepia.ceph.com 2022-03-19T05:35:02.621 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.15.152 smithi152.front.sepia.ceph.com smithi152 2022-03-19T05:35:02.841 INFO:teuthology.provision.fog.smithi152:Deploy complete! 2022-03-19T05:35:03.028 INFO:teuthology.lock.ops:Checking smithi115.front.sepia.ceph.com 2022-03-19T05:35:03.029 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-19T05:35:03.048 INFO:teuthology.lock.ops:Checking smithi152.front.sepia.ceph.com 2022-03-19T05:35:03.048 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-19T05:35:03.064 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: set os type and version on server 2022-03-19T05:35:03.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-03-19T05:35:03.238 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2022-03-19T05:35:03.245 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="Ubuntu" 2022-03-19T05:35:03.245 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-19T05:35:03.245 INFO:teuthology.orchestra.run.smithi115.stdout:ID=ubuntu 2022-03-19T05:35:03.246 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE=debian 2022-03-19T05:35:03.246 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-19T05:35:03.246 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="20.04" 2022-03-19T05:35:03.246 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-19T05:35:03.247 INFO:teuthology.orchestra.run.smithi115.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-19T05:35:03.247 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-19T05:35:03.247 INFO:teuthology.orchestra.run.smithi115.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-19T05:35:03.247 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_CODENAME=focal 2022-03-19T05:35:03.247 INFO:teuthology.orchestra.run.smithi115.stdout:UBUNTU_CODENAME=focal 2022-03-19T05:35:03.248 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2022-03-19T05:35:03.266 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: set os type and version on server 2022-03-19T05:35:03.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2022-03-19T05:35:03.445 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2022-03-19T05:35:03.452 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="Ubuntu" 2022-03-19T05:35:03.452 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-19T05:35:03.452 INFO:teuthology.orchestra.run.smithi152.stdout:ID=ubuntu 2022-03-19T05:35:03.453 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE=debian 2022-03-19T05:35:03.453 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-19T05:35:03.453 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="20.04" 2022-03-19T05:35:03.453 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-19T05:35:03.454 INFO:teuthology.orchestra.run.smithi152.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-19T05:35:03.454 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-19T05:35:03.454 INFO:teuthology.orchestra.run.smithi152.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-19T05:35:03.454 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_CODENAME=focal 2022-03-19T05:35:03.455 INFO:teuthology.orchestra.run.smithi152.stdout:UBUNTU_CODENAME=focal 2022-03-19T05:35:03.455 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2022-03-19T05:35:03.550 INFO:teuthology.dispatcher.supervisor:Running job 6744827 2022-03-19T05:35:03.551 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_aa078dbed6d14c1ad98beb5a02b956bbe0a741c8/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744827 --name vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744827/orig.config.yaml 2022-03-19T05:35:03.555 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744827 2022-03-19T05:35:03.555 INFO:teuthology.dispatcher.supervisor:Job PID: 2720 2022-03-19T05:35:03.556 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-19T12:05:14.151 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-19T12:05:14.706 INFO:teuthology.kill:Killing Pids: {2720} 2022-03-19T12:05:15.793 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-03-19T12:05:15.794 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-03-19T12:05:15.794 INFO:teuthology.misc:Compressing logs... 2022-03-19T12:05:15.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-03-19T12:05:15.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2022-03-19T12:05:16.225 INFO:teuthology.misc:Compressing logs... 2022-03-19T12:05:16.372 INFO:teuthology.misc:Compressing logs... 2022-03-19T12:08:11.539 INFO:teuthology.kill:No teuthology processes running 2022-03-19T12:08:12.716 INFO:teuthology.kill:Nuking machines: ['smithi115', 'smithi152'] 2022-03-19T12:08:13.664 INFO:teuthology.kill:2022-03-19 12:08:13,664.664 INFO:teuthology.nuke:targets: 2022-03-19T12:08:13.665 INFO:teuthology.kill: smithi115.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYrHSk+n95GhyoAnEns7VoU8bw0JsAlMPfL2hS96h+ZcinaA0Q7/a7NJ6nJZpYIvJStXVSDd+CEncmTGDLlawtsk4jhvC133Rmc/G/F5Krlu3lSvRs7e7nc958nt+hpP/DNOdOWK91rsP4QiTko6KYEBgZbSZyoVuV1C9B0e4m1CD8IKtAPbkJGhdEF/k71QEKAFrwvGfbSuZY1VCqGY/LqKsoLexR9AYMAbCW2Wd2LkEEdhPbf9cEZC9GCt5UnD8kHzoOKluH4zc05NG0kIKntgh0lojp9PIsLX6rc4GUmPny6/AhgydcOreZflO1khztGostsb90JdkTW3DOuH0zIaGWA5a0EFtCCWp8qmT6oEYiZtlKI4Z9IXdVEeQsplBqPnItqhAWTvwG0XlBlR8Gvqn+dKXgM/+kaX52tulGS7cmLTk/YCWoD+j0z2JI+Lu9daoszR6PcaqllslK5VrXJBbSFI8tdgHamMcEIVaoO+Dnal8KCiv8pXGbp6iDDwk= 2022-03-19T12:08:13.665 INFO:teuthology.kill: smithi152.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD3AxQZgJHDSdLtsLqoN1inzR1vZBfcA+vsxSsgtSF6SBn0u7qyuOyY57bMIMGV8urgsTJae3yvHuiwCmNPK47xH5uoIFTVnlb+o3jUAqm8GqpDizokU8OoNioFo9iID13M194OLv38p7HNmR3L7vwno8rqCOwymXvDB6PGfzA3mpmowsidzziYs2IltTlrlZBTJCDhoHqCqosMA+Eyd1c8aQk18ce2Jor5pP9Nnh64ngHeSlNnkNiWT8eWu+KtWpVXbVzyMOsSAbRzuBFbfQpwtTMcxXHquo55hC9LrLdfGmLRxR3BUwExjdlJ6qWa7RbII6C3eeb9O76Yd5JWuQ5o3doxwz+V3xGaBlbtqOq55FQCPHj8o8WWbB5r8MuOBJd3au7nwP6bIvJKPTJhmhlh0LN3YNdV0URtF2eIzk0a5av0HdnedUUQxHX8Nx6os1xvTJZTupKnB1vTmMRM4dIYvY/FXailVGyMUwEOzJ9QnDjm2jciIvjQIGi4NSnjZJ8= 2022-03-19T12:08:13.690 INFO:teuthology.kill:2022-03-19 12:08:13,690.690 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-19T12:08:13.694 INFO:teuthology.kill:2022-03-19 12:08:13,694.694 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-19T12:08:13.771 INFO:teuthology.kill:2022-03-19 12:08:13,770.770 INFO:teuthology.orchestra.console:Power off smithi115 2022-03-19T12:08:13.785 INFO:teuthology.kill:2022-03-19 12:08:13,784.784 INFO:teuthology.orchestra.console:Power off smithi152 2022-03-19T12:08:22.144 INFO:teuthology.kill:2022-03-19 12:08:22,143.143 INFO:teuthology.orchestra.console:Power off for smithi115 completed 2022-03-19T12:08:22.293 INFO:teuthology.kill:2022-03-19 12:08:22,293.293 INFO:teuthology.lock.ops:unlocked: smithi115.front.sepia.ceph.com 2022-03-19T12:08:26.264 INFO:teuthology.kill:2022-03-19 12:08:26,263.263 INFO:teuthology.orchestra.console:Power off for smithi152 completed 2022-03-19T12:08:26.408 INFO:teuthology.kill:2022-03-19 12:08:26,408.408 INFO:teuthology.lock.ops:unlocked: smithi152.front.sepia.ceph.com 2022-03-19T12:10:26.613 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-19T12:10:26.752 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi115 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744934 2022-03-19T12:10:26.752 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi152 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-03-18_07:57:03-fs-wip-vshankar-testing1-20220317-183011-testing-default-smithi/6744934