2022-02-06T17:59:52.861 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-06T17:59:52.991 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi092.front.sepia.ceph.com' reimaging 2022-02-06T17:59:52.992 INFO:teuthology.lock.ops:Updating [ubuntu@smithi092.front.sepia.ceph.com]: reset os type and version on server 2022-02-06T17:59:52.992 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2022-02-06T17:59:53.012 INFO:teuthology.lock.ops:Node 'ubuntu@smithi092.front.sepia.ceph.com' reimaging is complete 2022-02-06T17:59:53.012 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi096.front.sepia.ceph.com' reimaging 2022-02-06T17:59:53.013 INFO:teuthology.lock.ops:Updating [ubuntu@smithi096.front.sepia.ceph.com]: reset os type and version on server 2022-02-06T17:59:53.013 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2022-02-06T17:59:53.034 INFO:teuthology.lock.ops:Node 'ubuntu@smithi096.front.sepia.ceph.com' reimaging is complete 2022-02-06T17:59:53.255 INFO:teuthology.provision.fog.smithi092:Scheduling deploy of ubuntu 20.04 2022-02-06T17:59:53.290 INFO:teuthology.provision.fog.smithi096:Scheduling deploy of ubuntu 20.04 2022-02-06T17:59:53.802 INFO:teuthology.orchestra.console:Power off smithi092 2022-02-06T17:59:53.845 INFO:teuthology.orchestra.console:Power off smithi096 2022-02-06T18:00:06.293 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2022-02-06T18:00:06.315 INFO:teuthology.orchestra.console:Power off for smithi096 completed 2022-02-06T18:00:06.395 INFO:teuthology.orchestra.console:Power on smithi092 2022-02-06T18:00:06.423 INFO:teuthology.orchestra.console:Power on smithi096 2022-02-06T18:00:10.648 INFO:teuthology.orchestra.console:Power on for smithi092 completed 2022-02-06T18:00:10.749 INFO:teuthology.provision.fog.smithi092:Waiting for deploy to finish 2022-02-06T18:00:14.787 INFO:teuthology.orchestra.console:Power on for smithi096 completed 2022-02-06T18:00:14.888 INFO:teuthology.provision.fog.smithi096:Waiting for deploy to finish 2022-02-06T18:06:34.645 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-06T18:06:35.091 INFO:teuthology.provision.fog.smithi096:Node is ready 2022-02-06T18:06:35.098 INFO:teuthology.orchestra.run.smithi096.stdout:smithi096.front.sepia.ceph.com 2022-02-06T18:06:35.106 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96 smithi096.front.sepia.ceph.com smithi096 2022-02-06T18:06:35.322 INFO:teuthology.provision.fog.smithi096:Deploy complete! 2022-02-06T18:08:56.107 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-06T18:08:56.531 INFO:teuthology.provision.fog.smithi092:Node is ready 2022-02-06T18:08:56.674 INFO:teuthology.orchestra.run.smithi092.stdout:smithi092.front.sepia.ceph.com 2022-02-06T18:08:56.789 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.92 smithi092.front.sepia.ceph.com smithi092 2022-02-06T18:08:57.364 INFO:teuthology.provision.fog.smithi092:Deploy complete! 2022-02-06T18:08:58.132 INFO:teuthology.lock.ops:Checking smithi092.front.sepia.ceph.com 2022-02-06T18:08:58.133 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-06T18:08:58.231 INFO:teuthology.lock.ops:Checking smithi096.front.sepia.ceph.com 2022-02-06T18:08:58.232 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-06T18:08:58.348 INFO:teuthology.lock.ops:Updating [smithi092.front.sepia.ceph.com]: set os type and version on server 2022-02-06T18:08:58.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-02-06T18:08:58.670 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2022-02-06T18:08:58.782 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu" 2022-02-06T18:08:58.783 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-06T18:08:58.791 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu 2022-02-06T18:08:58.791 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian 2022-02-06T18:08:58.792 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-06T18:08:58.800 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="20.04" 2022-02-06T18:08:58.842 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-06T18:08:58.842 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-06T18:08:58.843 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-06T18:08:58.843 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-06T18:08:58.843 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=focal 2022-02-06T18:08:58.843 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=focal 2022-02-06T18:08:58.863 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2022-02-06T18:08:58.943 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: set os type and version on server 2022-02-06T18:08:58.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2022-02-06T18:08:59.670 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2022-02-06T18:08:59.854 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="Ubuntu" 2022-02-06T18:08:59.884 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-06T18:08:59.885 INFO:teuthology.orchestra.run.smithi096.stdout:ID=ubuntu 2022-02-06T18:08:59.885 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE=debian 2022-02-06T18:08:59.885 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-06T18:08:59.885 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="20.04" 2022-02-06T18:08:59.886 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-06T18:08:59.886 INFO:teuthology.orchestra.run.smithi096.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-06T18:08:59.886 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-06T18:08:59.902 INFO:teuthology.orchestra.run.smithi096.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-06T18:08:59.902 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_CODENAME=focal 2022-02-06T18:08:59.903 INFO:teuthology.orchestra.run.smithi096.stdout:UBUNTU_CODENAME=focal 2022-02-06T18:08:59.910 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2022-02-06T18:09:00.432 INFO:teuthology.dispatcher.supervisor:Running job 6665106 2022-02-06T18:09:00.432 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi/6665106 --name benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi --description rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} -- /home/teuthworker/archive/benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi/6665106/orig.config.yaml 2022-02-06T18:09:00.542 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi/6665106 2022-02-06T18:09:00.552 INFO:teuthology.dispatcher.supervisor:Job PID: 25627 2022-02-06T18:09:00.552 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-07T00:39:11.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-07T00:39:11.519 INFO:teuthology.kill:Killing Pids: {25627} 2022-02-07T00:39:12.737 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-02-07T00:39:12.738 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-02-07T00:39:12.739 INFO:teuthology.misc:Compressing logs... 2022-02-07T00:39:12.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-02-07T00:39:12.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2022-02-07T00:39:13.147 INFO:teuthology.misc:Compressing logs... 2022-02-07T00:39:14.074 INFO:teuthology.misc:Compressing logs... 2022-02-07T00:39:16.220 INFO:teuthology.kill:No teuthology processes running 2022-02-07T00:39:17.352 INFO:teuthology.kill:Nuking machines: ['smithi092', 'smithi096'] 2022-02-07T00:39:18.301 INFO:teuthology.kill:2022-02-07 00:39:18,300.300 INFO:teuthology.nuke:targets: 2022-02-07T00:39:18.302 INFO:teuthology.kill: smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE6eEg05O9HLGCqiT04tMRFnsqROHHxv9WSE0WJ1p8bl4HfowYs5gCyqBCV/pBRpZcVgLaGDpC0LVf8lO3LDKZIjAbX76VC7Ks/AHX0H1hgiLDnHmS+hu1+FdqaRWUsOeG0q/UR2FKyFEBe+1zPi9wg45ZoiTfHUOhNCLCpeqtGzXaEPsubSubTodSFbhk734lLEd7wSIDNh6hT6COTZuTALHA1pAh7aMCSWJNuznKXrhUWa+CxS76h37C7556W+5TWpdMkDi/av/mv4murtDHzH/TePRndPvovNLcC+7n2Xc0JDJuhflAjKQuvrYmpOGMlIvie2ezkKLUb906zdPC3uNC06i2VBFkPfmv1+7IrYZlBW5QQPoetzQ0WGebAfcoC3Z4Z38mPP74dkBT/+paRq1vxed+jGTPqRyCH3x4xJlBrz4J8jr0HEhyHDvomLXS3LdjdqUNnxYP/HJSA0fJ1WnqOkXJoq7+X7H4H1ZGae1tB+BduUFuYPmbdIFuNcM= 2022-02-07T00:39:18.302 INFO:teuthology.kill: smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkPMt0knjZyiYSMgE3L664DJ2bMdjNkgIomzUZeiqptL4NhF83OpH5LCe0l3WHjtwhF8eBEkDxcRAoK3J/iOCcPGc1TEPGgR1QTJmY/tJoptuo1xI2dkFAAu2T+chTlus40GMuBnSMFX4MNC3uEtkZg/ST6mEJYc/+zpEyQ1bcpqIv8zuCFgSj95Noajw+PYYsbejG3KYtOkJnyZUtDls9nySosvSou+IoM5duqionoizsrtzJA1x40Vd/mft37AedNFJ4cienQOHwDXxfAtenE8WvUC1g/kcar/wcdxkKftH1+B23HMP1AbKNEQYQ0wb+k5WpafBQO1eJ/aal1SG5TyXM3oIno+gCbVnaEjI+XJjYYTvc4ljWRfGNlfYVKn5vCwHZ1PnipCqlXi5ybu4uoqnbhiCv9pb+otfbCjVqVJD1e29+hMqwsda94X8WvM8Nxjgl1d8fXsb+3s698NnESxQYdPHk9EOH04/HMCFSq1xWUciKdKlCIjMQGR94i00= 2022-02-07T00:39:18.329 INFO:teuthology.kill:2022-02-07 00:39:18,329.329 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-07T00:39:18.332 INFO:teuthology.kill:2022-02-07 00:39:18,332.332 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-07T00:39:18.409 INFO:teuthology.kill:2022-02-07 00:39:18,409.409 INFO:teuthology.orchestra.console:Power off smithi092 2022-02-07T00:39:18.423 INFO:teuthology.kill:2022-02-07 00:39:18,422.422 INFO:teuthology.orchestra.console:Power off smithi096 2022-02-07T00:39:26.785 INFO:teuthology.kill:2022-02-07 00:39:26,784.784 INFO:teuthology.orchestra.console:Power off for smithi096 completed 2022-02-07T00:39:26.946 INFO:teuthology.kill:2022-02-07 00:39:26,945.945 INFO:teuthology.lock.ops:unlocked: smithi096.front.sepia.ceph.com 2022-02-07T00:39:30.888 INFO:teuthology.kill:2022-02-07 00:39:30,887.887 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2022-02-07T00:39:31.031 INFO:teuthology.kill:2022-02-07 00:39:31,031.031 INFO:teuthology.lock.ops:unlocked: smithi092.front.sepia.ceph.com 2022-02-07T00:41:31.362 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-07T00:41:31.492 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi092 but it was locked by another job: /home/teuthworker/archive/dis-2022-02-06_22:53:48-rbd-wip-rbd-quincy-batch-2-distro-default-smithi/6665482 2022-02-07T00:41:31.493 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi096 but it was locked by another job: /home/teuthworker/archive/dis-2022-02-06_22:53:48-rbd-wip-rbd-quincy-batch-2-distro-default-smithi/6665480