2022-05-01T07:43:40.937 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-05-01T07:43:41.104 INFO:teuthology.lock.ops:Start node 'smithi107.front.sepia.ceph.com' reimaging 2022-05-01T07:43:41.105 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: reset os type and version on server 2022-05-01T07:43:41.105 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2022-05-01T07:43:41.127 INFO:teuthology.lock.ops:Node 'smithi107.front.sepia.ceph.com' reimaging is complete 2022-05-01T07:43:41.127 INFO:teuthology.lock.ops:Start node 'smithi160.front.sepia.ceph.com' reimaging 2022-05-01T07:43:41.127 INFO:teuthology.lock.ops:Updating [smithi160.front.sepia.ceph.com]: reset os type and version on server 2022-05-01T07:43:41.127 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2022-05-01T07:43:41.144 INFO:teuthology.lock.ops:Node 'smithi160.front.sepia.ceph.com' reimaging is complete 2022-05-01T07:43:41.145 INFO:teuthology.lock.ops:Start node 'smithi169.front.sepia.ceph.com' reimaging 2022-05-01T07:43:41.145 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: reset os type and version on server 2022-05-01T07:43:41.145 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2022-05-01T07:43:41.166 INFO:teuthology.lock.ops:Node 'smithi169.front.sepia.ceph.com' reimaging is complete 2022-05-01T07:43:41.335 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of ubuntu 18.04 2022-05-01T07:43:41.345 INFO:teuthology.provision.fog.smithi160:Scheduling deploy of ubuntu 18.04 2022-05-01T07:43:41.392 INFO:teuthology.provision.fog.smithi169:Scheduling deploy of ubuntu 18.04 2022-05-01T07:43:41.693 INFO:teuthology.orchestra.console:Power off smithi107 2022-05-01T07:43:41.736 INFO:teuthology.orchestra.console:Power off smithi169 2022-05-01T07:43:41.778 INFO:teuthology.orchestra.console:Power off smithi160 2022-05-01T07:43:50.069 INFO:teuthology.orchestra.console:Power off for smithi107 completed 2022-05-01T07:43:50.105 INFO:teuthology.orchestra.console:Power off for smithi169 completed 2022-05-01T07:43:50.140 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2022-05-01T07:43:50.169 INFO:teuthology.orchestra.console:Power on smithi107 2022-05-01T07:43:50.207 INFO:teuthology.orchestra.console:Power on smithi169 2022-05-01T07:43:50.241 INFO:teuthology.orchestra.console:Power on smithi160 2022-05-01T07:43:58.545 INFO:teuthology.orchestra.console:Power on for smithi107 completed 2022-05-01T07:43:58.609 INFO:teuthology.orchestra.console:Power on for smithi169 completed 2022-05-01T07:43:58.640 INFO:teuthology.orchestra.console:Power on for smithi160 completed 2022-05-01T07:43:58.691 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish 2022-05-01T07:43:58.710 INFO:teuthology.provision.fog.smithi169:Waiting for deploy to finish 2022-05-01T07:43:58.741 INFO:teuthology.provision.fog.smithi160:Waiting for deploy to finish 2022-05-01T07:49:00.797 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-01T07:49:00.864 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-01T07:49:01.153 INFO:teuthology.provision.fog.smithi160:Node is ready 2022-05-01T07:49:01.202 INFO:teuthology.orchestra.run.smithi160.stdout:smithi160.front.sepia.ceph.com 2022-05-01T07:49:01.236 INFO:teuthology.provision.fog.smithi169:Node is ready 2022-05-01T07:49:01.251 INFO:teuthology.orchestra.run.smithi160.stdout:172.21.15.160 smithi160.front.sepia.ceph.com smithi160 2022-05-01T07:49:01.288 INFO:teuthology.orchestra.run.smithi169.stdout:smithi169.front.sepia.ceph.com 2022-05-01T07:49:01.337 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169 smithi169.front.sepia.ceph.com smithi169 2022-05-01T07:49:01.578 INFO:teuthology.provision.fog.smithi160:Deploy complete! 2022-05-01T07:49:01.664 INFO:teuthology.provision.fog.smithi169:Deploy complete! 2022-05-01T07:49:09.739 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-01T07:49:10.069 INFO:teuthology.provision.fog.smithi107:Node is ready 2022-05-01T07:49:10.118 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107.front.sepia.ceph.com 2022-05-01T07:49:10.166 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107 2022-05-01T07:49:10.478 INFO:teuthology.provision.fog.smithi107:Deploy complete! 2022-05-01T07:49:10.714 INFO:teuthology.lock.ops:Checking smithi107.front.sepia.ceph.com 2022-05-01T07:49:10.714 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-01T07:49:10.732 INFO:teuthology.lock.ops:Checking smithi160.front.sepia.ceph.com 2022-05-01T07:49:10.732 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-01T07:49:10.747 INFO:teuthology.lock.ops:Checking smithi169.front.sepia.ceph.com 2022-05-01T07:49:10.747 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-01T07:49:10.764 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: set os type and version on server 2022-05-01T07:49:10.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-05-01T07:49:10.999 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2022-05-01T07:49:11.046 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Ubuntu" 2022-05-01T07:49:11.046 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-05-01T07:49:11.046 INFO:teuthology.orchestra.run.smithi107.stdout:ID=ubuntu 2022-05-01T07:49:11.046 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE=debian 2022-05-01T07:49:11.046 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-05-01T07:49:11.047 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="18.04" 2022-05-01T07:49:11.047 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.ubuntu.com/" 2022-05-01T07:49:11.047 INFO:teuthology.orchestra.run.smithi107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-05-01T07:49:11.049 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-05-01T07:49:11.049 INFO:teuthology.orchestra.run.smithi107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-05-01T07:49:11.049 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_CODENAME=bionic 2022-05-01T07:49:11.049 INFO:teuthology.orchestra.run.smithi107.stdout:UBUNTU_CODENAME=bionic 2022-05-01T07:49:11.050 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2022-05-01T07:49:11.068 INFO:teuthology.lock.ops:Updating [smithi160.front.sepia.ceph.com]: set os type and version on server 2022-05-01T07:49:11.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2022-05-01T07:49:11.367 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2022-05-01T07:49:11.414 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="Ubuntu" 2022-05-01T07:49:11.414 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:ID=ubuntu 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE=debian 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="18.04" 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://www.ubuntu.com/" 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_CODENAME=bionic 2022-05-01T07:49:11.415 INFO:teuthology.orchestra.run.smithi160.stdout:UBUNTU_CODENAME=bionic 2022-05-01T07:49:11.416 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2022-05-01T07:49:11.435 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: set os type and version on server 2022-05-01T07:49:11.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2022-05-01T07:49:11.688 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2022-05-01T07:49:11.736 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="Ubuntu" 2022-05-01T07:49:11.736 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-05-01T07:49:11.736 INFO:teuthology.orchestra.run.smithi169.stdout:ID=ubuntu 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE=debian 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="18.04" 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://www.ubuntu.com/" 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_CODENAME=bionic 2022-05-01T07:49:11.737 INFO:teuthology.orchestra.run.smithi169.stdout:UBUNTU_CODENAME=bionic 2022-05-01T07:49:11.738 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2022-05-01T07:49:11.843 INFO:teuthology.dispatcher.supervisor:Running job 6810221 2022-05-01T07:49:11.844 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_135488acc76a85490d37201ec506706cc4ab2d62/virtualenv/bin/teuthology -v --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6810221 --name lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi --description rados:thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_18.04} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6810221/orig.config.yaml 2022-05-01T07:49:11.847 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/lflores-2022-04-27_17:53:15-rados:thrash-erasure-code-big-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6810221 2022-05-01T07:49:11.848 INFO:teuthology.dispatcher.supervisor:Job PID: 22779 2022-05-01T07:49:11.848 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-05-01T14:19:22.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-01T14:19:23.072 INFO:teuthology.kill:Killing Pids: {22779} 2022-05-01T14:19:24.530 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2022-05-01T14:19:24.530 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2022-05-01T14:19:24.531 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2022-05-01T14:19:24.531 INFO:teuthology.misc:Compressing logs... 2022-05-01T14:19:24.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2022-05-01T14:19:24.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2022-05-01T14:19:25.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2022-05-01T14:19:25.914 INFO:teuthology.kill:No teuthology processes running 2022-05-01T14:19:27.299 INFO:teuthology.kill:Nuking machines: ['smithi107', 'smithi160', 'smithi169'] 2022-05-01T14:19:28.468 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_135488acc76a85490d37201ec506706cc4ab2d62/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-05-01T14:19:28.469 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-05-01T14:19:28.469 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_135488acc76a85490d37201ec506706cc4ab2d62/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated 2022-05-01T14:19:28.469 INFO:teuthology.kill: "class": algorithms.Blowfish, 2022-05-01T14:19:28.469 INFO:teuthology.kill:2022-05-01 14:19:28,468.468 INFO:teuthology.nuke:targets: 2022-05-01T14:19:28.469 INFO:teuthology.kill: smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5OUU/GiwS3YlEV6VcIB1YJrkXtSMSdXx2o3O/976Jzn7Z5KeTzQXlsTXKjsdR9v/kAVb0H2XaB/QC4O8L9n4p4qbsMRd+whZzuu9fPmThdYpbVBQ9P7xJHpWYjTb0klu70yA80vChBFZ0p0PoUW4iSREG6Z+XS5nx0YrCQ+Aeft+STs+0PrU+jTgBNxSOk51MBpHY78sOnLOxYT6IV4MwKfgNBS17YpZXxsPvII9OYcJLBq4fUZwqo+NjdnVHYlokuWzwHefyclOxZobIF45xYSjgxDRcHJzn1iT+aIIUNNfENLqQHgFHKzirHf/Y+YdtqM6d9S1eJvtNS+DJInAN 2022-05-01T14:19:28.469 INFO:teuthology.kill: smithi160.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MG3DbxzQaMERNz3AWp0vlhyQphB/FKbcx287ZcBQjRO2poMt4sQYpJNvEnqn9t4LA3fQXgpA8lLdrsbJjX9sLqnOYrBUrOAL6KcKfKJSnQhEyg0DGh3mTQpaHib7DhVEoIP9t2uynouoJbLQqSmyj2W1nj4cuLZhD+3fj9SuPyLWwXDt3EbIqJ0hh6bAlhSifKDcy9jhS3n1g2lz0YEXTUBc7OWBIc+IubY2f9jrbhox8O5YWBjy0l9DuUuntnZR/xH5eD2QQgR2TL1O38aBpuv/LhBt60HDyZFJu7aHDJ5/wM1ho8hO/rd3ubgVY8f7hOsLNzu5vXCIX8fAbZX7 2022-05-01T14:19:28.469 INFO:teuthology.kill: smithi169.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdb5QvgwsSExZsrlKtaK5N/Opr0mscq/8L0YZrRf058L8UZQLKhbAHJyy/C9INn1m9onjUiaGtJcI88YlIrnyAdULW9f1boy3Vw+NoWdhZFL7O7Pfcxgs3NmNJEnmT17F4IE6xVY8CZLTWzRJ3xBc1GtgyAfrNPc1pH4iSmMWNmYOmA3ziobQ0SXuYrcNzpQR/9DyB2xvsjJ5X1v/zdFmM8rMI50f7fq4YkLMQCLC7tE+Hjg7KjQfjaOOBP206EhTJtpUkirUiJDoCRbvE5BGTLADsySIEGBqrHkHt4XAGqIcYVaqFwxp5Dbi1v+VxT6UzhDX7kPKz3bSTAAjHI/iZ 2022-05-01T14:19:28.497 INFO:teuthology.kill:2022-05-01 14:19:28,497.497 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-01T14:19:28.553 INFO:teuthology.kill:2022-05-01 14:19:28,501.501 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-01T14:19:28.553 INFO:teuthology.kill:2022-05-01 14:19:28,502.502 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-01T14:19:28.586 INFO:teuthology.kill:2022-05-01 14:19:28,586.586 INFO:teuthology.orchestra.console:Power off smithi160 2022-05-01T14:19:28.600 INFO:teuthology.kill:2022-05-01 14:19:28,599.599 INFO:teuthology.orchestra.console:Power off smithi169 2022-05-01T14:19:28.610 INFO:teuthology.kill:2022-05-01 14:19:28,610.610 INFO:teuthology.orchestra.console:Power off smithi107 2022-05-01T14:19:41.077 INFO:teuthology.kill:2022-05-01 14:19:41,076.076 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2022-05-01T14:19:41.080 INFO:teuthology.kill:2022-05-01 14:19:41,080.080 INFO:teuthology.orchestra.console:Power off for smithi169 completed 2022-05-01T14:19:41.085 INFO:teuthology.kill:2022-05-01 14:19:41,085.085 INFO:teuthology.orchestra.console:Power off for smithi107 completed 2022-05-01T14:19:41.228 INFO:teuthology.kill:2022-05-01 14:19:41,226.226 INFO:teuthology.lock.ops:unlocked: smithi169.front.sepia.ceph.com 2022-05-01T14:19:41.230 INFO:teuthology.kill:2022-05-01 14:19:41,230.230 INFO:teuthology.lock.ops:unlocked: smithi160.front.sepia.ceph.com 2022-05-01T14:19:41.231 INFO:teuthology.kill:2022-05-01 14:19:41,231.231 INFO:teuthology.lock.ops:unlocked: smithi107.front.sepia.ceph.com 2022-05-01T14:21:41.544 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-05-01T14:21:41.667 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi107 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-05-01_13:18:44-fs-wip-vshankar-testing1-20220428-204527-testing-default-smithi/6817539 2022-05-01T14:21:41.667 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi160 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-05-01_13:18:44-fs-wip-vshankar-testing1-20220428-204527-testing-default-smithi/6817539