2022-02-07T23:18:19.764 INFO:root:teuthology version: 1.1.0-7cebb3f2 2022-02-07T23:18:19.918 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging 2022-02-07T23:18:19.919 INFO:teuthology.lock.ops:Updating [ubuntu@smithi039.front.sepia.ceph.com]: reset os type and version on server 2022-02-07T23:18:19.920 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2022-02-07T23:18:19.944 INFO:teuthology.lock.ops:Node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging is complete 2022-02-07T23:18:19.944 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging 2022-02-07T23:18:19.945 INFO:teuthology.lock.ops:Updating [ubuntu@smithi078.front.sepia.ceph.com]: reset os type and version on server 2022-02-07T23:18:19.945 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2022-02-07T23:18:19.972 INFO:teuthology.lock.ops:Node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging is complete 2022-02-07T23:18:20.193 INFO:teuthology.provision.fog.smithi039:Scheduling deploy of centos 8.stream 2022-02-07T23:18:20.207 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of centos 8.stream 2022-02-07T23:18:20.596 INFO:teuthology.orchestra.console:Power off smithi039 2022-02-07T23:18:20.702 INFO:teuthology.orchestra.console:Power off smithi078 2022-02-07T23:18:29.070 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2022-02-07T23:18:29.171 INFO:teuthology.orchestra.console:Power on smithi078 2022-02-07T23:18:33.084 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2022-02-07T23:18:33.186 INFO:teuthology.orchestra.console:Power on smithi039 2022-02-07T23:18:37.445 INFO:teuthology.orchestra.console:Power on for smithi039 completed 2022-02-07T23:18:37.545 INFO:teuthology.orchestra.console:Power on for smithi078 completed 2022-02-07T23:18:37.546 INFO:teuthology.provision.fog.smithi039:Waiting for deploy to finish 2022-02-07T23:18:37.646 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish 2022-02-07T23:23:19.389 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-07T23:23:19.638 INFO:teuthology.provision.fog.smithi078:Node is ready 2022-02-07T23:23:19.697 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com 2022-02-07T23:23:19.755 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078 2022-02-07T23:23:20.104 INFO:teuthology.provision.fog.smithi078:Deploy complete! 2022-02-07T23:24:07.730 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-07T23:24:07.968 INFO:teuthology.provision.fog.smithi039:Node is ready 2022-02-07T23:24:08.029 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039.front.sepia.ceph.com 2022-02-07T23:24:08.097 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39 smithi039.front.sepia.ceph.com smithi039 2022-02-07T23:24:08.473 INFO:teuthology.provision.fog.smithi039:Deploy complete! 2022-02-07T23:24:08.628 INFO:teuthology.lock.ops:Checking smithi039.front.sepia.ceph.com 2022-02-07T23:24:08.629 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-07T23:24:08.648 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com 2022-02-07T23:24:08.649 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-07T23:24:08.667 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: set os type and version on server 2022-02-07T23:24:08.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2022-02-07T23:24:08.862 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2022-02-07T23:24:08.919 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2022-02-07T23:24:08.920 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8" 2022-02-07T23:24:08.920 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2022-02-07T23:24:08.920 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2022-02-07T23:24:08.921 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8" 2022-02-07T23:24:08.921 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2022-02-07T23:24:08.921 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-07T23:24:08.921 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2022-02-07T23:24:08.922 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-07T23:24:08.922 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2022-02-07T23:24:08.922 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-07T23:24:08.922 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-07T23:24:08.922 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-07T23:24:08.924 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2022-02-07T23:24:08.948 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server 2022-02-07T23:24:08.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2022-02-07T23:24:09.202 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2022-02-07T23:24:09.264 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2022-02-07T23:24:09.264 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8" 2022-02-07T23:24:09.264 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2022-02-07T23:24:09.264 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2022-02-07T23:24:09.265 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8" 2022-02-07T23:24:09.265 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2022-02-07T23:24:09.265 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-07T23:24:09.266 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2022-02-07T23:24:09.266 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-07T23:24:09.266 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2022-02-07T23:24:09.266 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-07T23:24:09.266 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-07T23:24:09.267 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-07T23:24:09.267 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2022-02-07T23:24:09.363 INFO:teuthology.dispatcher.supervisor:Running job 6666722 2022-02-07T23:24:09.364 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7cebb3f2319fd6c8340c0f7cd15a137e747fd32e/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-07_19:41:02-rados-WIP_GBH_safe_shutdown_v2B-distro-basic-smithi/6666722 --name benhanokh-2022-02-07_19:41:02-rados-WIP_GBH_safe_shutdown_v2B-distro-basic-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/benhanokh-2022-02-07_19:41:02-rados-WIP_GBH_safe_shutdown_v2B-distro-basic-smithi/6666722/orig.config.yaml 2022-02-07T23:24:09.367 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-07_19:41:02-rados-WIP_GBH_safe_shutdown_v2B-distro-basic-smithi/6666722 2022-02-07T23:24:09.367 INFO:teuthology.dispatcher.supervisor:Job PID: 30779 2022-02-07T23:24:09.368 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-08T05:54:20.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-08T05:54:20.718 INFO:teuthology.kill:Killing Pids: {30779} 2022-02-08T05:54:21.914 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-02-08T05:54:21.915 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-02-08T05:54:21.915 INFO:teuthology.misc:Compressing logs... 2022-02-08T05:54:21.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2022-02-08T05:54:22.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2022-02-08T05:54:22.447 INFO:teuthology.misc:Compressing logs... 2022-02-08T05:54:22.931 INFO:teuthology.orchestra.run.smithi078.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-08T05:54:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-08T05:54:43.249 INFO:teuthology.misc:Compressing logs... 2022-02-08T05:54:45.393 INFO:teuthology.kill:No teuthology processes running 2022-02-08T05:54:46.525 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi078'] 2022-02-08T05:54:47.507 INFO:teuthology.kill:2022-02-08 05:54:47,506.506 INFO:teuthology.nuke:targets: 2022-02-08T05:54:47.507 INFO:teuthology.kill: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr3DbVE0edAi2GIvbmWlGxVpiLriWD/rO8YG18P53XG/+4q2owchV8evK53T26PZ5S4Lvprobr1ysbf8y4Udlmv53kRRX6nAHDelYEK2D61CePK4llxazTH8jHRCrkYBoC/PpzjaoL7Tzr86ByQ7sATlYTL6GF3ZFmVXuwbSRFZkG01QZ7DyIJWbQg6wuAtvak8XQ4mIz1EsSWf24ebnOPNjN9196qrB3+FTK5wxygTcbs2ru9NSpFLDPKz5j1HXSJM2DMApXpNClAKb9uSz+dL7yANlz2uEyB4pcx2DVRm9PVh3sXzuAcKHNgFaSZGhmiWWZNxKx9uFQrB6Vre1eZ+Z2Jd4p/FssqGqwnXjXojF5NYM0VQXEYodWZtibwUYBfnDJp7UDN4Ju+Hil0v79lzEThCDmJoFiAzwYVcpav2bh76461+2z1Elxe4xA/gdKYx5fWnCDYowZSCJ1MSjtrDsyugDdkmC5ZDA6yHoca2D2Xzy2hiOXNwTAaLIV5EVU= 2022-02-08T05:54:47.508 INFO:teuthology.kill: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+dQFcU00HGR2/ZlEGN8q7V54UMT6hi/y6hjeK1Og88xgClDH7Pa3/+zACg1kBACl6dB7yYRg1+DVMCiYTUCa4nVVbLArtwBVWXBTKHA8b8ajxQ7/LOtwpdsR33b3yTL6pfNJF464zMJpHzPy30l4YsBje9AO7o4CYYdZc64qYRJi2IU1fC0XlUvTxwcGq8XYP51NRo9DVvwyZstGMDChRLrl6hZokapp0NKo6hVQvwHZUHPrycNGgOmrlnDwoekoMv9K2tKBnB2lDzRJhmaavinc3qdXqMdNEdczUsweHQZwtBYiEtItoxaqglpT5Cvq50TUblBAsAHa2ofbvwcavYE69vO/0k1ufZ8ZgumpffEWtYH4QpbXoHLyXL6cIXtmUn+weJR17yZJo2vaGs6CU5wRl1Om0ZpcJG7yWjKr2q09xJdz4cx4Ds5xXQrFVCuBwmv4YuvSB/oUdgEzaEXEg8PbOXNub7bu7p3v625iuoUWqPOz/HNF/gLCQMNfTfuc= 2022-02-08T05:54:47.534 INFO:teuthology.kill:2022-02-08 05:54:47,533.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-08T05:54:47.537 INFO:teuthology.kill:2022-02-08 05:54:47,537.537 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-08T05:54:47.616 INFO:teuthology.kill:2022-02-08 05:54:47,615.615 INFO:teuthology.orchestra.console:Power off smithi039 2022-02-08T05:54:47.628 INFO:teuthology.kill:2022-02-08 05:54:47,628.628 INFO:teuthology.orchestra.console:Power off smithi078 2022-02-08T05:54:55.991 INFO:teuthology.kill:2022-02-08 05:54:55,990.990 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2022-02-08T05:54:56.137 INFO:teuthology.kill:2022-02-08 05:54:56,136.136 INFO:teuthology.lock.ops:unlocked: smithi078.front.sepia.ceph.com 2022-02-08T05:55:00.099 INFO:teuthology.kill:2022-02-08 05:55:00,098.098 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2022-02-08T05:55:00.249 INFO:teuthology.kill:2022-02-08 05:55:00,248.248 INFO:teuthology.lock.ops:unlocked: smithi039.front.sepia.ceph.com 2022-02-08T05:57:00.816 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-08T05:57:00.921 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi039 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-02-08_05:00:02-smoke-master-distro-default-smithi/6668950 2022-02-08T05:57:00.922 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi078 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-02-08_05:00:02-smoke-master-distro-default-smithi/6668950