2022-02-07T15:13:27.128 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-07T15:13:27.267 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging 2022-02-07T15:13:27.268 INFO:teuthology.lock.ops:Updating [ubuntu@smithi183.front.sepia.ceph.com]: reset os type and version on server 2022-02-07T15:13:27.269 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2022-02-07T15:13:27.292 INFO:teuthology.lock.ops:Node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging is complete 2022-02-07T15:13:27.292 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi184.front.sepia.ceph.com' reimaging 2022-02-07T15:13:27.293 INFO:teuthology.lock.ops:Updating [ubuntu@smithi184.front.sepia.ceph.com]: reset os type and version on server 2022-02-07T15:13:27.293 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2022-02-07T15:13:27.312 INFO:teuthology.lock.ops:Node 'ubuntu@smithi184.front.sepia.ceph.com' reimaging is complete 2022-02-07T15:13:27.501 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of ubuntu 20.04 2022-02-07T15:13:27.514 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of ubuntu 20.04 2022-02-07T15:13:27.925 INFO:teuthology.orchestra.console:Power off smithi183 2022-02-07T15:13:27.938 INFO:teuthology.orchestra.console:Power off smithi184 2022-02-07T15:13:36.298 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2022-02-07T15:13:36.400 INFO:teuthology.orchestra.console:Power on smithi184 2022-02-07T15:13:40.403 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2022-02-07T15:13:40.504 INFO:teuthology.orchestra.console:Power on smithi183 2022-02-07T15:13:44.766 INFO:teuthology.orchestra.console:Power on for smithi184 completed 2022-02-07T15:13:44.867 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2022-02-07T15:13:48.871 INFO:teuthology.orchestra.console:Power on for smithi183 completed 2022-02-07T15:13:48.972 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish 2022-02-07T15:18:39.393 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-07T15:18:39.858 INFO:teuthology.provision.fog.smithi184:Node is ready 2022-02-07T15:18:39.865 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184.front.sepia.ceph.com 2022-02-07T15:18:39.872 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.15.184 smithi184.front.sepia.ceph.com smithi184 2022-02-07T15:18:40.043 INFO:teuthology.provision.fog.smithi184:Deploy complete! 2022-02-07T15:18:51.042 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-07T15:18:51.558 INFO:teuthology.provision.fog.smithi183:Node is ready 2022-02-07T15:18:51.565 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183.front.sepia.ceph.com 2022-02-07T15:18:51.572 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.183 smithi183.front.sepia.ceph.com smithi183 2022-02-07T15:18:51.785 INFO:teuthology.provision.fog.smithi183:Deploy complete! 2022-02-07T15:18:51.982 INFO:teuthology.lock.ops:Checking smithi183.front.sepia.ceph.com 2022-02-07T15:18:51.983 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-07T15:18:52.002 INFO:teuthology.lock.ops:Checking smithi184.front.sepia.ceph.com 2022-02-07T15:18:52.003 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-07T15:18:52.030 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: set os type and version on server 2022-02-07T15:18:52.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2022-02-07T15:18:52.211 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2022-02-07T15:18:52.216 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Ubuntu" 2022-02-07T15:18:52.216 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-07T15:18:52.217 INFO:teuthology.orchestra.run.smithi183.stdout:ID=ubuntu 2022-02-07T15:18:52.217 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE=debian 2022-02-07T15:18:52.217 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-07T15:18:52.217 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="20.04" 2022-02-07T15:18:52.217 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-07T15:18:52.218 INFO:teuthology.orchestra.run.smithi183.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-07T15:18:52.218 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-07T15:18:52.218 INFO:teuthology.orchestra.run.smithi183.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-07T15:18:52.218 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_CODENAME=focal 2022-02-07T15:18:52.219 INFO:teuthology.orchestra.run.smithi183.stdout:UBUNTU_CODENAME=focal 2022-02-07T15:18:52.220 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2022-02-07T15:18:52.238 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: set os type and version on server 2022-02-07T15:18:52.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2022-02-07T15:18:52.467 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2022-02-07T15:18:52.473 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2022-02-07T15:18:52.473 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-07T15:18:52.474 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2022-02-07T15:18:52.474 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2022-02-07T15:18:52.474 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-07T15:18:52.475 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="20.04" 2022-02-07T15:18:52.475 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-07T15:18:52.475 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-07T15:18:52.475 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-07T15:18:52.475 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-07T15:18:52.476 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=focal 2022-02-07T15:18:52.476 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=focal 2022-02-07T15:18:52.476 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2022-02-07T15:18:52.570 INFO:teuthology.dispatcher.supervisor:Running job 6666516 2022-02-07T15:18:52.571 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-07_14:20:29-rados-WIP_GBH_safe_shutdown_v2A-distro-basic-smithi/6666516 --name benhanokh-2022-02-07_14:20:29-rados-WIP_GBH_safe_shutdown_v2A-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} -- /home/teuthworker/archive/benhanokh-2022-02-07_14:20:29-rados-WIP_GBH_safe_shutdown_v2A-distro-basic-smithi/6666516/orig.config.yaml 2022-02-07T15:18:52.576 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-07_14:20:29-rados-WIP_GBH_safe_shutdown_v2A-distro-basic-smithi/6666516 2022-02-07T15:18:52.576 INFO:teuthology.dispatcher.supervisor:Job PID: 26967 2022-02-07T15:18:52.577 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-07T21:49:21.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-07T21:49:21.907 INFO:teuthology.kill:Killing Pids: {26967} 2022-02-07T21:49:23.084 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-02-07T21:49:23.085 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-02-07T21:49:23.086 INFO:teuthology.misc:Compressing logs... 2022-02-07T21:49:23.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2022-02-07T21:49:23.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2022-02-07T21:49:23.630 INFO:teuthology.misc:Compressing logs... 2022-02-07T21:49:24.488 INFO:teuthology.misc:Compressing logs... 2022-02-07T21:49:33.299 INFO:teuthology.kill:No teuthology processes running 2022-02-07T21:49:34.453 INFO:teuthology.kill:Nuking machines: ['smithi183', 'smithi184'] 2022-02-07T21:49:35.385 INFO:teuthology.kill:2022-02-07 21:49:35,385.385 INFO:teuthology.nuke:targets: 2022-02-07T21:49:35.386 INFO:teuthology.kill: smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmZS6xqFp/I1nc7j8RoV13qSJk5PuqwTltUSUSPho9IQSwoJ8fZNr2KP68Fr0ldCvwMPLG94nKCHP9ugmnYl8/rt1tBKzniVjlX744KqXOCFMt187HX4UXHhbFSUO8WlYOswYpFwafk3oPE/LdgrN3WBxEXTxt5LRLdVLNJXmFSNRogQ866jACYKXDQ2BTWnNJMSGhQlqUHSHXi6zPveMwY+Rj5urD0Y356VEbbe+yjRkU3d1a153D/pkPKRXjdU295G7dcrUzwZBxGvvVEHZtivENFVzlUbMpHbY2a89doYjwm93+ZXMnE5GmhjUfcT0Lpl/I43O/mInqLsKn7AiIkARnzr0pHB0pu4yaZI/aEQ/FrZdDCa8pmUqaSQNtbkQUrltNJ9HLIfh9JPHuAHurOKuk0ishS/DPfaeNsRzxz6YPkM6mLY7LKYcogsiWyDqrQ68f0tboBR7uP7CrYmfGkiwuMt337ZX2T/+EMC7L9EfskGMjxhElzHQwb33Z7zs= 2022-02-07T21:49:35.386 INFO:teuthology.kill: smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1XU0wxV1VAzFg8o+R71iQcA5UdxX8urGxrMnw7MGajpsgm+97twMrk4FE4ExDOUjHb+Q2NqTRiJJ+/P0stp69j6HUQiOzGsA/r0RvRD6ncFb5jeYJCXj9LVWNFeCMMXhcYcBPQQRt9zeOYIAuUc7ulukpnylERtIrRvLKAZUtMnpnnnIigYigLHj7H/gBSoE1DvGk4Zg1ZE3v4ZbM9p6+VM1lKaXxIyQSPHJpbzjXh5w7dWt680Ly8G0z4rZB2pe8EhSxQgvNrvPQ7WdSnsK3JqoRRCniN3gq48w/fC0KA/4braOhICqxced/G2pSExcwEsXIFHhvM1pl5kjtwbMW7R/xBYOc043c0biY+x3juELd9AAf7L89qzpWZn01U+6OyP0MpGKRiHWIwcoQGkBkPayKVHuvvYsbpn+iaDWoR9mv4LTI+dWSTus4iZvKKozhrg7Nhma/B+eo3q3tYM3TXDdhrvJsvbTck4Ivd9HaER2s+VPLaHXXPaILdtWaz2s= 2022-02-07T21:49:35.411 INFO:teuthology.kill:2022-02-07 21:49:35,410.410 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-07T21:49:35.414 INFO:teuthology.kill:2022-02-07 21:49:35,414.414 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-07T21:49:35.491 INFO:teuthology.kill:2022-02-07 21:49:35,491.491 INFO:teuthology.orchestra.console:Power off smithi184 2022-02-07T21:49:35.507 INFO:teuthology.kill:2022-02-07 21:49:35,507.507 INFO:teuthology.orchestra.console:Power off smithi183 2022-02-07T21:49:43.870 INFO:teuthology.kill:2022-02-07 21:49:43,869.869 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2022-02-07T21:49:44.015 INFO:teuthology.kill:2022-02-07 21:49:44,015.015 INFO:teuthology.lock.ops:unlocked: smithi183.front.sepia.ceph.com 2022-02-07T21:49:47.977 INFO:teuthology.kill:2022-02-07 21:49:47,976.976 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2022-02-07T21:49:48.124 INFO:teuthology.kill:2022-02-07 21:49:48,124.124 INFO:teuthology.lock.ops:unlocked: smithi184.front.sepia.ceph.com 2022-02-07T21:51:48.463 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-07T21:51:48.571 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi183 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-07_20:08:36-rados-wip-yuri4-testing-2022-02-07-1051-octopus-distro-default-smithi/6666943 2022-02-07T21:51:48.571 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi184 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-07_20:08:36-rados-wip-yuri4-testing-2022-02-07-1051-octopus-distro-default-smithi/6666946