2021-11-23T16:11:54.948 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-23T16:11:55.095 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging 2021-11-23T16:11:55.096 INFO:teuthology.lock.ops:Updating [ubuntu@smithi078.front.sepia.ceph.com]: reset os type and version on server 2021-11-23T16:11:55.097 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2021-11-23T16:11:55.116 INFO:teuthology.lock.ops:Node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging is complete 2021-11-23T16:11:55.120 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging 2021-11-23T16:11:55.121 INFO:teuthology.lock.ops:Updating [ubuntu@smithi110.front.sepia.ceph.com]: reset os type and version on server 2021-11-23T16:11:55.123 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2021-11-23T16:11:55.146 INFO:teuthology.lock.ops:Node 'ubuntu@smithi110.front.sepia.ceph.com' reimaging is complete 2021-11-23T16:11:55.334 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of centos 8.3 2021-11-23T16:11:55.358 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of centos 8.3 2021-11-23T16:11:55.683 INFO:teuthology.orchestra.console:Power off smithi110 2021-11-23T16:11:55.698 INFO:teuthology.orchestra.console:Power off smithi078 2021-11-23T16:12:04.053 INFO:teuthology.orchestra.console:Power off for smithi110 completed 2021-11-23T16:12:04.060 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-11-23T16:12:04.155 INFO:teuthology.orchestra.console:Power on smithi110 2021-11-23T16:12:04.164 INFO:teuthology.orchestra.console:Power on smithi078 2021-11-23T16:12:12.531 INFO:teuthology.orchestra.console:Power on for smithi110 completed 2021-11-23T16:12:12.538 INFO:teuthology.orchestra.console:Power on for smithi078 completed 2021-11-23T16:12:12.633 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish 2021-11-23T16:12:12.639 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish 2021-11-23T16:17:02.312 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-23T16:17:02.317 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-23T16:17:02.609 INFO:teuthology.provision.fog.smithi110:Node is ready 2021-11-23T16:17:02.616 INFO:teuthology.provision.fog.smithi078:Node is ready 2021-11-23T16:17:02.628 INFO:teuthology.orchestra.run.smithi110.stdout:smithi110.front.sepia.ceph.com 2021-11-23T16:17:02.643 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com 2021-11-23T16:17:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110 smithi110.front.sepia.ceph.com smithi110 2021-11-23T16:17:02.713 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078 2021-11-23T16:17:03.018 INFO:teuthology.provision.fog.smithi110:Deploy complete! 2021-11-23T16:17:03.031 INFO:teuthology.provision.fog.smithi078:Deploy complete! 2021-11-23T16:17:03.184 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com 2021-11-23T16:17:03.185 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-23T16:17:03.208 INFO:teuthology.lock.ops:Checking smithi110.front.sepia.ceph.com 2021-11-23T16:17:03.208 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-23T16:17:03.226 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server 2021-11-23T16:17:03.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2021-11-23T16:17:03.483 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2021-11-23T16:17:03.537 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Linux" 2021-11-23T16:17:03.537 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8" 2021-11-23T16:17:03.538 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2021-11-23T16:17:03.538 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2021-11-23T16:17:03.538 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8" 2021-11-23T16:17:03.538 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2021-11-23T16:17:03.539 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Linux 8" 2021-11-23T16:17:03.539 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2021-11-23T16:17:03.539 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-23T16:17:03.539 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2021-11-23T16:17:03.540 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-11-23T16:17:03.540 INFO:teuthology.orchestra.run.smithi078.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-11-23T16:17:03.540 INFO:teuthology.orchestra.run.smithi078.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-11-23T16:17:03.541 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2021-11-23T16:17:03.563 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server 2021-11-23T16:17:03.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2021-11-23T16:17:03.828 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2021-11-23T16:17:03.881 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Linux" 2021-11-23T16:17:03.882 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8" 2021-11-23T16:17:03.882 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2021-11-23T16:17:03.882 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2021-11-23T16:17:03.882 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8" 2021-11-23T16:17:03.883 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8" 2021-11-23T16:17:03.883 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Linux 8" 2021-11-23T16:17:03.883 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2021-11-23T16:17:03.883 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-23T16:17:03.884 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2021-11-23T16:17:03.884 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-11-23T16:17:03.884 INFO:teuthology.orchestra.run.smithi110.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-11-23T16:17:03.884 INFO:teuthology.orchestra.run.smithi110.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-11-23T16:17:03.885 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2021-11-23T16:17:03.992 INFO:teuthology.dispatcher.supervisor:Running job 6522345 2021-11-23T16:17:03.993 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-23_15:13:21-rados-wip-yuri2-testing-2021-11-22-1532-distro-basic-smithi/6522345 --name yuriw-2021-11-23_15:13:21-rados-wip-yuri2-testing-2021-11-22-1532-distro-basic-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} -- /home/teuthworker/archive/yuriw-2021-11-23_15:13:21-rados-wip-yuri2-testing-2021-11-22-1532-distro-basic-smithi/6522345/orig.config.yaml 2021-11-23T16:17:03.996 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-23_15:13:21-rados-wip-yuri2-testing-2021-11-22-1532-distro-basic-smithi/6522345 2021-11-23T16:17:03.997 INFO:teuthology.dispatcher.supervisor:Job PID: 16364 2021-11-23T16:17:03.997 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-24T04:18:00.123 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-24T04:18:00.278 INFO:teuthology.kill:Killing Pids: {16364} 2021-11-24T04:18:01.460 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-11-24T04:18:01.460 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-11-24T04:18:01.461 INFO:teuthology.misc:Compressing logs... 2021-11-24T04:18:01.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2021-11-24T04:18:01.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2021-11-24T04:18:02.098 INFO:teuthology.misc:Compressing logs... 2021-11-24T04:18:03.230 INFO:teuthology.orchestra.run.smithi078.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-11-24T04:18:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-11-24T04:18:06.385 INFO:teuthology.misc:Compressing logs... 2021-11-24T04:18:14.887 INFO:teuthology.orchestra.run.smithi078.stderr:tar: .: file changed as we read it 2021-11-24T04:18:16.223 INFO:teuthology.kill:No teuthology processes running 2021-11-24T04:18:17.494 INFO:teuthology.kill:Nuking machines: ['smithi078', 'smithi110'] 2021-11-24T04:18:18.484 INFO:teuthology.kill:2021-11-24 04:18:18,484.484 INFO:teuthology.nuke:targets: 2021-11-24T04:18:18.485 INFO:teuthology.kill: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/oSbnTx3dKbc6Zn03X8cfZ0TZdQdn51gXKg+qELpeHz7hPg1z1xpUx02vVgrfq8+dS4hCmzAMxTfh8UIUMLP1DhamYWJa8unbyc1N1b4VUMRO3q2ueei/dJk1EYh8j97xpFN9xo50vSa/XiQNHKROEqiGu16VMdC6ght5f9Bb6H6pytgVXVGcGO7mXdYt7MNKORZNYzed4Zn+G3uh6dxCC1E6yHCT1JTVWLry9vCgpZ9Mnsa8ooXrtPicreLtAdevsKOUT2pYi8/eFldZxAndzBYvK4ej4fRi6LaQc1fuEQjEvLTZOH7WMgPLPIVgMWsnAog4hbFZG/2pgyevzncVMeqaNdbtW56UpNJniaLC7uZyFAN/k10M38CA11paIAi7+kPk7hoGBbOcOCp3m84HUsACPESEIWI0KQsJ/l0pX2IZ0fHAam8I/8f7vxHfBtQkyS7rT/5RXJvawubwB0AWlwndHeyZqFN/Ov/MppkQCO8z4ctnpfoY9Iov5j9cKJ0= 2021-11-24T04:18:18.486 INFO:teuthology.kill: smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiDTRQpz1eQGB2QpFz6EywIyoYFVoxsG4pI6xwpXePy5jgqww4gOf5GuICSyiQUth3ytEhJCOof4vsb/K43QiX13zBflMrB02prhrfeKTGXAhMtw/ZbknUE8Vwza996bmU5zHcCpNMFeFA/9XdoD9s7hbdRNovkHnyaBtf1b/e9bYGvH7Zr6BXO00pnbjnznnPPqh/9Qi563Ll3LmFWL09jCxpEF8qmZ+F5fBjv7sjcrQGHNjq7Kshaaa7p92B73pNc57zxPdAtPE34QYoFm/QQdPWfzKT6Hn//R0aZH45mVoydABU0ttPZOpuLwThlFwP1TuR9pEAfodRP26zAOgRzU3RDfVzlyObTN3x+lnO52p2KUm+V3gjfmfwDrd0izRbEQBKVq0EQ4j/+UOhj/NfoZrSjw9mqPtQjL5YU0nFeYK5Tc8zCTOIeC/huu8pBSkr17GefKHA5lgdIQSdoYq/Xc4ge4CxprnYp6lcmCxiAUSNjg2x1E5G6yTJTNBdpyE= 2021-11-24T04:18:18.514 INFO:teuthology.kill:2021-11-24 04:18:18,513.513 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-24T04:18:18.517 INFO:teuthology.kill:2021-11-24 04:18:18,517.517 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-24T04:18:18.599 INFO:teuthology.kill:2021-11-24 04:18:18,598.598 INFO:teuthology.orchestra.console:Power off smithi110 2021-11-24T04:18:18.616 INFO:teuthology.kill:2021-11-24 04:18:18,616.616 INFO:teuthology.orchestra.console:Power off smithi078 2021-11-24T04:18:26.985 INFO:teuthology.kill:2021-11-24 04:18:26,983.983 INFO:teuthology.orchestra.console:Power off for smithi110 completed 2021-11-24T04:18:27.008 INFO:teuthology.kill:2021-11-24 04:18:27,007.007 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-11-24T04:18:27.148 INFO:teuthology.kill:2021-11-24 04:18:27,147.147 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com 2021-11-24T04:18:27.191 INFO:teuthology.kill:2021-11-24 04:18:27,180.180 INFO:teuthology.lock.ops:unlocked: smithi078.front.sepia.ceph.com 2021-11-24T04:20:27.642 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-24T04:20:27.754 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi078 but it was locked by another job: /home/teuthworker/archive/ifed01-2021-11-23_22:21:23-rados-wip-ifed-faster-rm-p2-distro-default-smithi/6522884 2021-11-24T04:20:27.754 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi110 but it was locked by another job: /home/teuthworker/archive/ifed01-2021-11-23_22:21:23-rados-wip-ifed-faster-rm-p2-distro-default-smithi/6522884