2021-09-29T21:03:09.743 INFO:root:teuthology version: 1.1.0-f08b273e
2021-09-29T21:03:09.894 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi123.front.sepia.ceph.com' reimaging
2021-09-29T21:03:09.895 INFO:teuthology.lock.ops:Updating [ubuntu@smithi123.front.sepia.ceph.com]: reset os type and version on server
2021-09-29T21:03:09.896 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2021-09-29T21:03:09.917 INFO:teuthology.lock.ops:Node 'ubuntu@smithi123.front.sepia.ceph.com' reimaging is complete
2021-09-29T21:03:09.917 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging
2021-09-29T21:03:09.918 INFO:teuthology.lock.ops:Updating [ubuntu@smithi134.front.sepia.ceph.com]: reset os type and version on server
2021-09-29T21:03:09.918 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server
2021-09-29T21:03:09.938 INFO:teuthology.lock.ops:Node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging is complete
2021-09-29T21:03:09.939 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi169.front.sepia.ceph.com' reimaging
2021-09-29T21:03:09.939 INFO:teuthology.lock.ops:Updating [ubuntu@smithi169.front.sepia.ceph.com]: reset os type and version on server
2021-09-29T21:03:09.939 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server
2021-09-29T21:03:09.959 INFO:teuthology.lock.ops:Node 'ubuntu@smithi169.front.sepia.ceph.com' reimaging is complete
2021-09-29T21:03:10.136 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of centos 8.3
2021-09-29T21:03:10.152 INFO:teuthology.provision.fog.smithi134:Scheduling deploy of centos 8.3
2021-09-29T21:03:10.171 INFO:teuthology.provision.fog.smithi169:Scheduling deploy of centos 8.3
2021-09-29T21:03:10.529 INFO:teuthology.orchestra.console:Power off smithi169
2021-09-29T21:03:10.544 INFO:teuthology.orchestra.console:Power off smithi134
2021-09-29T21:03:10.556 INFO:teuthology.orchestra.console:Power off smithi123
2021-09-29T21:03:10.689 INFO:teuthology.orchestra.console:Power off for smithi134 completed
2021-09-29T21:03:10.693 INFO:teuthology.orchestra.console:Power off for smithi169 completed
2021-09-29T21:03:10.791 INFO:teuthology.orchestra.console:Power on smithi134
2021-09-29T21:03:10.800 INFO:teuthology.orchestra.console:Power on smithi169
2021-09-29T21:03:15.049 INFO:teuthology.orchestra.console:Power on for smithi134 completed
2021-09-29T21:03:15.059 INFO:teuthology.orchestra.console:Power on for smithi169 completed
2021-09-29T21:03:15.151 INFO:teuthology.provision.fog.smithi134:Waiting for deploy to finish
2021-09-29T21:03:15.160 INFO:teuthology.provision.fog.smithi169:Waiting for deploy to finish
2021-09-29T21:03:23.038 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2021-09-29T21:03:23.139 INFO:teuthology.orchestra.console:Power on smithi123
2021-09-29T21:03:27.391 INFO:teuthology.orchestra.console:Power on for smithi123 completed
2021-09-29T21:03:27.493 INFO:teuthology.provision.fog.smithi123:Waiting for deploy to finish
2021-09-29T21:07:21.657 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-09-29T21:07:38.588 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-09-29T21:07:41.904 INFO:teuthology.provision.fog.smithi134:Node is ready
2021-09-29T21:07:41.924 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134.front.sepia.ceph.com
2021-09-29T21:07:41.981 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134 smithi134.front.sepia.ceph.com smithi134
2021-09-29T21:07:42.329 INFO:teuthology.provision.fog.smithi134:Deploy complete!
2021-09-29T21:07:45.739 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2021-09-29T21:07:59.031 INFO:teuthology.provision.fog.smithi123:Node is ready
2021-09-29T21:07:59.051 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123.front.sepia.ceph.com
2021-09-29T21:07:59.110 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123 smithi123.front.sepia.ceph.com smithi123
2021-09-29T21:07:59.474 INFO:teuthology.provision.fog.smithi123:Deploy complete!
2021-09-29T21:08:11.254 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-09-29T21:08:11.536 INFO:teuthology.provision.fog.smithi169:Node is ready
2021-09-29T21:08:11.550 INFO:teuthology.orchestra.run.smithi169.stdout:smithi169.front.sepia.ceph.com
2021-09-29T21:08:11.604 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169 smithi169.front.sepia.ceph.com smithi169
2021-09-29T21:08:11.920 INFO:teuthology.provision.fog.smithi169:Deploy complete!
2021-09-29T21:08:12.092 INFO:teuthology.lock.ops:Checking smithi123.front.sepia.ceph.com
2021-09-29T21:08:12.093 INFO:teuthology.lock.ops:New key found. Updating...
2021-09-29T21:08:12.113 INFO:teuthology.lock.ops:Checking smithi134.front.sepia.ceph.com
2021-09-29T21:08:12.114 INFO:teuthology.lock.ops:New key found. Updating...
2021-09-29T21:08:12.133 INFO:teuthology.lock.ops:Checking smithi169.front.sepia.ceph.com
2021-09-29T21:08:12.133 INFO:teuthology.lock.ops:New key found. Updating...
2021-09-29T21:08:12.152 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: set os type and version on server
2021-09-29T21:08:12.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com'
2021-09-29T21:08:12.468 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64
2021-09-29T21:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Linux"
2021-09-29T21:08:12.523 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8"
2021-09-29T21:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos"
2021-09-29T21:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora"
2021-09-29T21:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8"
2021-09-29T21:08:12.524 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8"
2021-09-29T21:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Linux 8"
2021-09-29T21:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31"
2021-09-29T21:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-09-29T21:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/"
2021-09-29T21:08:12.525 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-09-29T21:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-09-29T21:08:12.526 INFO:teuthology.orchestra.run.smithi123.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-09-29T21:08:12.527 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2021-09-29T21:08:12.554 INFO:teuthology.lock.ops:Updating [smithi134.front.sepia.ceph.com]: set os type and version on server
2021-09-29T21:08:12.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com'
2021-09-29T21:08:12.821 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64
2021-09-29T21:08:12.880 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Linux"
2021-09-29T21:08:12.881 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8"
2021-09-29T21:08:12.881 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos"
2021-09-29T21:08:12.881 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora"
2021-09-29T21:08:12.882 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8"
2021-09-29T21:08:12.882 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8"
2021-09-29T21:08:12.882 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Linux 8"
2021-09-29T21:08:12.882 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31"
2021-09-29T21:08:12.883 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-09-29T21:08:12.883 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/"
2021-09-29T21:08:12.883 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-09-29T21:08:12.883 INFO:teuthology.orchestra.run.smithi134.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-09-29T21:08:12.883 INFO:teuthology.orchestra.run.smithi134.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-09-29T21:08:12.884 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server
2021-09-29T21:08:12.905 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: set os type and version on server
2021-09-29T21:08:12.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com'
2021-09-29T21:08:13.176 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64
2021-09-29T21:08:13.231 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Linux"
2021-09-29T21:08:13.232 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="8"
2021-09-29T21:08:13.232 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos"
2021-09-29T21:08:13.232 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora"
2021-09-29T21:08:13.233 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="8"
2021-09-29T21:08:13.233 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el8"
2021-09-29T21:08:13.233 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Linux 8"
2021-09-29T21:08:13.233 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31"
2021-09-29T21:08:13.234 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-09-29T21:08:13.234 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/"
2021-09-29T21:08:13.236 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-09-29T21:08:13.237 INFO:teuthology.orchestra.run.smithi169.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-09-29T21:08:13.237 INFO:teuthology.orchestra.run.smithi169.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-09-29T21:08:13.238 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server
2021-09-29T21:08:13.338 INFO:teuthology.dispatcher.supervisor:Running job 6413909
2021-09-29T21:08:13.339 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-09-29_18:13:53-rados-wip-yuri3-testing-2021-09-29-0859-octopus-distro-basic-smithi/6413909 --name yuriw-2021-09-29_18:13:53-rados-wip-yuri3-testing-2021-09-29-0859-octopus-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/yuriw-2021-09-29_18:13:53-rados-wip-yuri3-testing-2021-09-29-0859-octopus-distro-basic-smithi/6413909/orig.config.yaml
2021-09-29T21:08:13.344 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-09-29_18:13:53-rados-wip-yuri3-testing-2021-09-29-0859-octopus-distro-basic-smithi/6413909
2021-09-29T21:08:13.345 INFO:teuthology.dispatcher.supervisor:Job PID: 10044
2021-09-29T21:08:13.346 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-09-30T00:16:26.400 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-09-29_18:13:53-rados-wip-yuri3-testing-2021-09-29-0859-octopus-distro-basic-smithi/jobs/ failed with status 500: A server error occurred.  Please contact the administrator.
2021-09-30T09:09:02.256 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-09-30T09:09:02.456 INFO:teuthology.kill:Killing Pids: {10044}
2021-09-30T09:09:03.577 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a']
2021-09-30T09:09:03.578 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x']
2021-09-30T09:09:03.578 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']
2021-09-30T09:09:03.579 INFO:teuthology.misc:Compressing logs...
2021-09-30T09:09:03.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com'
2021-09-30T09:09:03.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com'
2021-09-30T09:09:04.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com'
2021-09-30T09:09:04.768 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-09-30T09:09:05.084 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-09-30T09:09:05.766 INFO:teuthology.kill:No teuthology processes running
2021-09-30T09:09:06.780 INFO:teuthology.kill:Nuking machines: ['smithi123', 'smithi134', 'smithi169']
2021-09-30T09:09:07.621 INFO:teuthology.kill:2021-09-30 09:09:07,621.621 INFO:teuthology.nuke:targets:
2021-09-30T09:09:07.622 INFO:teuthology.kill:  smithi123.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0xaLqhA8WQg+/qqWpw8c+zvWYEq+j2qYaqHDZAfpF84OOVl6fban0P2Nzgt4Dt3PcCITbSlSgfUQCroOFlBrXg+AwF1AouCOAZjGbkZi8EnYd6e1usrl7PxjA2syiVh7Df+x5HIdD7UtiDUJ6cKL3AmJDUsCmP0kIw3/8pPF2jb3i5RvwPhO1YGPYZBHSVKJ5MbSosHtR2V9X/XpaKL4DegltTatT3OS+pNKRPaHGMecDvTf0gYkOyrh+1oBUzShRS1KNA08sTkafYfUyuMzB8kr876FMgwCZvtgyiMKpZLji/H4LW/gsezmQ0wiW8xvsIml1jyHLlu5igk2QVnL+NQCzvlkc/HK21aD8noNoFlUTEHTXYt4Fw46jYQjtsmrdfJDNrwp8MVrppI+rK2e/eMwRKHXumHum4tFfPLf2IRCcx0A3w2tBuBoMWeHkAaIMMR+9IGF0UOh82SCfx4HtUADO57oCzv3ogls8vvhAK9lyoitOI1j9+QaD9nEo0pc=
2021-09-30T09:09:07.622 INFO:teuthology.kill:  smithi134.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD44kqqizTr23pfBbal80LD1aiWG9hRSZG4aT9X9Wa/toYOSbx/8fssCJY82MqKWR913VxykR0RI1o3PYBQmw+NgDT+R7l4Is9585zLCHNi5VLxpE7pZ8O39DxomWsai7MkOV7xeVbr6TrMLPfOEOwgBAGAui7nIwG2HKEAr7ItreX4fZ4Ux5KyBja2oTzWd1cj9vcv//QZqD0PHwqg41J60nvzNaslZwpWfqE+kzBrREJnV6ELpIi4HWI3fvDRt6khUTG6g7RHePcTcueV52pP4F2RP9Q3b8wRr3/0y9Itfu1HapfLA229uYzIujyVEEbOEXKDpyi9HotvzS7/679rxLeNwkYfRCr9GacSjiBd+blM9za6xoCJDqRK0lpUjLlYNXSPyXCuuXLWVqHRmI6FgmTptGNLO9UfrAabV3jZfuloSJVTm7IUwTRWVufWHvvXQUijdGE1IX+1jok1Ml/fxCtDNeSRZwGLGAwhF/1cSK9UIZpR8WVS76oUuHFXT6U=
2021-09-30T09:09:07.623 INFO:teuthology.kill:  smithi169.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvJokIx9EZRSsm/a++DChOaPfczOFEBPivnMj1dUaHOPd+HwUsLvpE9bKLsLz2NpaqAZCTcYBXDGdw8h7Cp+gdR6NIOMdRgz4vRg5QTlYBxun8YNgT/v3W+p0gavN1EreUzhx21Z6p9Vqn4KLVbIYiJCDVegoWyF6ESBwEH0upLQDwbmOc+AVa8OltFIIRllGm5VzlJPAK0GvamRACQBJPk/FE0bc1tbsjSv65pBVPMqaUEdIUYutk22LHs1wcNoUEaCiVACsRW5yD7RudtJPopfBEydQw1n6FYltjXLylHG3e42vQUqkV5WvGqeWkiS9qmW3TKao0j7ZfhFVREDCu/R87QsSKATFfboW+7mw+2SorPWVG7lma99r15JnqjObYtaJTIrpU617vV0ahvoA4YEAkBkMMEDBrZYQ9BaU2zu1S+lpsrJtZn4h6d2nIWH5JZMipd6Ot4hYlBo1cjzPqhFFQ+mf81fvJBKNyl0JUjFJ+rJQgbo4nYgFEUUjcyEc=
2021-09-30T09:09:07.646 INFO:teuthology.kill:2021-09-30 09:09:07,646.646 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-09-30T09:09:07.649 INFO:teuthology.kill:2021-09-30 09:09:07,649.649 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-09-30T09:09:07.650 INFO:teuthology.kill:2021-09-30 09:09:07,650.650 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-09-30T09:09:07.717 INFO:teuthology.kill:2021-09-30 09:09:07,717.717 INFO:teuthology.orchestra.console:Power off smithi123
2021-09-30T09:09:07.730 INFO:teuthology.kill:2021-09-30 09:09:07,730.730 INFO:teuthology.orchestra.console:Power off smithi134
2021-09-30T09:09:07.741 INFO:teuthology.kill:2021-09-30 09:09:07,741.741 INFO:teuthology.orchestra.console:Power off smithi169
2021-09-30T09:09:16.096 INFO:teuthology.kill:2021-09-30 09:09:16,095.095 INFO:teuthology.orchestra.console:Power off for smithi134 completed
2021-09-30T09:09:16.105 INFO:teuthology.kill:2021-09-30 09:09:16,105.105 INFO:teuthology.orchestra.console:Power off for smithi169 completed
2021-09-30T09:09:16.244 INFO:teuthology.kill:2021-09-30 09:09:16,243.243 INFO:teuthology.lock.ops:unlocked: smithi134.front.sepia.ceph.com
2021-09-30T09:09:16.276 INFO:teuthology.kill:2021-09-30 09:09:16,276.276 INFO:teuthology.lock.ops:unlocked: smithi169.front.sepia.ceph.com
2021-09-30T09:09:20.202 INFO:teuthology.kill:2021-09-30 09:09:20,202.202 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2021-09-30T09:09:20.351 INFO:teuthology.kill:2021-09-30 09:09:20,351.351 INFO:teuthology.lock.ops:unlocked: smithi123.front.sepia.ceph.com
2021-09-30T09:11:20.616 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15