2021-12-08T19:05:06.430 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-08T19:05:06.564 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi002.front.sepia.ceph.com' reimaging 2021-12-08T19:05:06.566 INFO:teuthology.lock.ops:Updating [ubuntu@smithi002.front.sepia.ceph.com]: reset os type and version on server 2021-12-08T19:05:06.567 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2021-12-08T19:05:06.586 INFO:teuthology.lock.ops:Node 'ubuntu@smithi002.front.sepia.ceph.com' reimaging is complete 2021-12-08T19:05:06.587 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi003.front.sepia.ceph.com' reimaging 2021-12-08T19:05:06.587 INFO:teuthology.lock.ops:Updating [ubuntu@smithi003.front.sepia.ceph.com]: reset os type and version on server 2021-12-08T19:05:06.587 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2021-12-08T19:05:06.625 INFO:teuthology.lock.ops:Node 'ubuntu@smithi003.front.sepia.ceph.com' reimaging is complete 2021-12-08T19:05:06.782 INFO:teuthology.provision.fog.smithi002:Scheduling deploy of centos 8.3 2021-12-08T19:05:06.818 INFO:teuthology.provision.fog.smithi003:Scheduling deploy of centos 8.3 2021-12-08T19:05:07.130 INFO:teuthology.orchestra.console:Power off smithi002 2021-12-08T19:05:07.169 INFO:teuthology.orchestra.console:Power off smithi003 2021-12-08T19:05:15.502 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2021-12-08T19:05:15.604 INFO:teuthology.orchestra.console:Power on smithi002 2021-12-08T19:05:19.658 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2021-12-08T19:05:19.760 INFO:teuthology.orchestra.console:Power on smithi003 2021-12-08T19:05:24.012 INFO:teuthology.orchestra.console:Power on for smithi003 completed 2021-12-08T19:05:24.099 INFO:teuthology.orchestra.console:Power on for smithi002 completed 2021-12-08T19:05:24.113 INFO:teuthology.provision.fog.smithi003:Waiting for deploy to finish 2021-12-08T19:05:24.200 INFO:teuthology.provision.fog.smithi002:Waiting for deploy to finish 2021-12-08T19:09:58.868 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-08T19:09:58.915 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-08T19:09:59.148 INFO:teuthology.provision.fog.smithi002:Node is ready 2021-12-08T19:09:59.164 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002.front.sepia.ceph.com 2021-12-08T19:09:59.215 INFO:teuthology.provision.fog.smithi003:Node is ready 2021-12-08T19:09:59.219 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2 smithi002.front.sepia.ceph.com smithi002 2021-12-08T19:09:59.231 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003.front.sepia.ceph.com 2021-12-08T19:09:59.290 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3 smithi003.front.sepia.ceph.com smithi003 2021-12-08T19:09:59.547 INFO:teuthology.provision.fog.smithi002:Deploy complete! 2021-12-08T19:09:59.616 INFO:teuthology.provision.fog.smithi003:Deploy complete! 2021-12-08T19:09:59.784 INFO:teuthology.lock.ops:Checking smithi002.front.sepia.ceph.com 2021-12-08T19:09:59.785 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-08T19:09:59.804 INFO:teuthology.lock.ops:Checking smithi003.front.sepia.ceph.com 2021-12-08T19:09:59.805 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-08T19:09:59.848 INFO:teuthology.lock.ops:Updating [smithi002.front.sepia.ceph.com]: set os type and version on server 2021-12-08T19:09:59.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2021-12-08T19:10:00.116 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2021-12-08T19:10:00.170 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Linux" 2021-12-08T19:10:00.171 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8" 2021-12-08T19:10:00.171 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2021-12-08T19:10:00.171 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2021-12-08T19:10:00.172 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8" 2021-12-08T19:10:00.172 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2021-12-08T19:10:00.172 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Linux 8" 2021-12-08T19:10:00.173 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2021-12-08T19:10:00.174 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-08T19:10:00.174 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2021-12-08T19:10:00.174 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-12-08T19:10:00.175 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-12-08T19:10:00.175 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-12-08T19:10:00.177 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2021-12-08T19:10:00.197 INFO:teuthology.lock.ops:Updating [smithi003.front.sepia.ceph.com]: set os type and version on server 2021-12-08T19:10:00.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2021-12-08T19:10:00.468 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2021-12-08T19:10:00.523 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Linux" 2021-12-08T19:10:00.524 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8" 2021-12-08T19:10:00.524 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2021-12-08T19:10:00.524 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2021-12-08T19:10:00.524 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8" 2021-12-08T19:10:00.525 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2021-12-08T19:10:00.525 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Linux 8" 2021-12-08T19:10:00.525 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2021-12-08T19:10:00.525 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-08T19:10:00.526 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2021-12-08T19:10:00.526 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-12-08T19:10:00.526 INFO:teuthology.orchestra.run.smithi003.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-12-08T19:10:00.527 INFO:teuthology.orchestra.run.smithi003.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-12-08T19:10:00.527 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2021-12-08T19:10:00.657 INFO:teuthology.dispatcher.supervisor:Running job 6552886 2021-12-08T19:10:00.658 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2021-12-08_18:58:51-rados-master-distro-basic-smithi/6552886 --name rzarzynski-2021-12-08_18:58:51-rados-master-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-balanced} -- /home/teuthworker/archive/rzarzynski-2021-12-08_18:58:51-rados-master-distro-basic-smithi/6552886/orig.config.yaml 2021-12-08T19:10:00.661 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rzarzynski-2021-12-08_18:58:51-rados-master-distro-basic-smithi/6552886 2021-12-08T19:10:00.662 INFO:teuthology.dispatcher.supervisor:Job PID: 2660 2021-12-08T19:10:00.663 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-09T07:10:56.785 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-09T07:10:56.987 INFO:teuthology.kill:Killing Pids: {2660} 2021-12-09T07:10:58.156 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-12-09T07:10:58.157 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-12-09T07:10:58.158 INFO:teuthology.misc:Compressing logs... 2021-12-09T07:10:58.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2021-12-09T07:10:58.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2021-12-09T07:10:58.753 INFO:teuthology.misc:Compressing logs... 2021-12-09T07:10:59.831 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-09T07:10:59.843 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-09T07:11:00.443 INFO:teuthology.misc:Compressing logs... 2021-12-09T07:11:03.041 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/ceph-client.0.39461.log: file size changed while zipping 2021-12-09T07:11:12.187 INFO:teuthology.kill:No teuthology processes running 2021-12-09T07:11:13.319 INFO:teuthology.kill:Nuking machines: ['smithi002', 'smithi003'] 2021-12-09T07:11:14.247 INFO:teuthology.kill:2021-12-09 07:11:14,247.247 INFO:teuthology.nuke:targets: 2021-12-09T07:11:14.248 INFO:teuthology.kill: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1VDTocazC6fWCcjkCfrh0t8LQPJfb5WDPv+7i5lvJIs71oZ2k4fyvhuZAVhbfacTu0noS5W3d7BH5rHcQAML2nvMRG9pljdDvuIfxx+Nm+4nlOAMMSy9kl6zATAN2P+1KnZ6fR/09H4I0P+6TqiCMQrtZscQgpVEJ4TNG2HxyeYuT6UynfAHQFdA9+MnXvUj++z7uHnMk2VCqGlJEggL7DZjLxFROmfnEK84/Nmmy5nU7+jrYrPtTA3ULCfB23sRBgiLx6MGMHCEwHZgnoPUDxHORaLNjQY5sIWhxMu3zr8du+NYZYvrucvKGMz3a/X2cR3Wot1anKb4AXgRdA700lSgQ6gzmO+sxYgtpH5FeBpKjck6mtwRyggmU6E0Y+kQxXN31Wr+qAAJK4Js4LcIb8kN/XoPA2fdhIip78+b6p3STw53lG7XZXpXPJKbEHQNGC93pSUQdw4Plqmc1HNoba1E08vALBzg8WuUGAPFNY1cXRjjQBmtwPw1jBWvlbOU= 2021-12-09T07:11:14.249 INFO:teuthology.kill: smithi003.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDkXv8gLTbe8hhZhnEPlnH0Z5Oe1kwW+yVatxt/WZ1cNRZaGFZp78Pz1JmmixUpweXRk3KusOFjVFKp97H0rzoAFKrg3TvEqrNXsBycUELcymy7VJFaUr1QJQzBtMLwNrvTx5nabYjC6gikca6iZDATI8oJy2CNZ2v67qISEQZ6EPatDKv1beLu4NjnSX0lQBM7potnDHaxIc28QPJrCF9kcfdUSCCDEO8A8XwWBc7kcw4M2ApHJV092YFRWRuIz/smcC104jS15dAZg/y124hBYkZ/0+/jajPl+71C+p11S87UcU4l68NbIXfcpKWKfrAvmLw46c8SLwr2WjKeozsgd6l10skVh/kKBl8OLLSfnpzRDPgMmeMSFD99WdyASc1hNCdwnvwH3UjoVNMXCmvfMD6Y7LZ50l0oe0jR2WZyo/VKiU38Vk15xuI7W6yBb3nuMczVD+6s/Aq7V4uZXjxL4ClvLG8pDP2CIAspZnD70Z2AoPtvCqpm2mCvaJS9ak= 2021-12-09T07:11:14.274 INFO:teuthology.kill:2021-12-09 07:11:14,274.274 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-09T07:11:14.275 INFO:teuthology.kill:2021-12-09 07:11:14,275.275 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-09T07:11:14.354 INFO:teuthology.kill:2021-12-09 07:11:14,353.353 INFO:teuthology.orchestra.console:Power off smithi002 2021-12-09T07:11:14.366 INFO:teuthology.kill:2021-12-09 07:11:14,366.366 INFO:teuthology.orchestra.console:Power off smithi003 2021-12-09T07:11:22.720 INFO:teuthology.kill:2021-12-09 07:11:22,720.720 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2021-12-09T07:11:22.728 INFO:teuthology.kill:2021-12-09 07:11:22,727.727 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2021-12-09T07:11:22.867 INFO:teuthology.kill:2021-12-09 07:11:22,866.866 INFO:teuthology.lock.ops:unlocked: smithi002.front.sepia.ceph.com 2021-12-09T07:11:22.873 INFO:teuthology.kill:2021-12-09 07:11:22,873.873 INFO:teuthology.lock.ops:unlocked: smithi003.front.sepia.ceph.com 2021-12-09T07:13:23.183 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-12-09T07:13:23.298 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi002 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6553345 2021-12-09T07:13:23.299 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi003 but it was locked by another job: /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6553345