2023-03-27T09:49:18.493 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-27T09:49:18.664 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging 2023-03-27T09:49:18.666 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server 2023-03-27T09:49:18.667 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-03-27T09:49:18.691 INFO:teuthology.lock.ops:Node 'smithi097.front.sepia.ceph.com' reimaging is complete 2023-03-27T09:49:18.691 INFO:teuthology.lock.ops:Start node 'smithi142.front.sepia.ceph.com' reimaging 2023-03-27T09:49:18.692 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: reset os type and version on server 2023-03-27T09:49:18.692 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2023-03-27T09:49:18.721 INFO:teuthology.lock.ops:Node 'smithi142.front.sepia.ceph.com' reimaging is complete 2023-03-27T09:49:18.873 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.stream 2023-03-27T09:49:18.888 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 8.stream 2023-03-27T09:49:19.167 INFO:teuthology.orchestra.console:Power off smithi097 2023-03-27T09:49:19.222 INFO:teuthology.orchestra.console:Power off smithi142 2023-03-27T09:49:19.335 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-03-27T09:49:19.367 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2023-03-27T09:49:19.438 INFO:teuthology.orchestra.console:Power on smithi097 2023-03-27T09:49:19.477 INFO:teuthology.orchestra.console:Power on smithi142 2023-03-27T09:49:23.706 INFO:teuthology.orchestra.console:Power on for smithi097 completed 2023-03-27T09:49:23.807 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish 2023-03-27T09:49:27.852 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2023-03-27T09:49:27.953 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2023-03-27T09:52:12.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-03-27T09:52:16.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:52:21.865 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-03-27T09:52:25.861 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:52:30.917 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-03-27T09:52:34.917 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:52:39.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-03-27T09:52:43.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:52:49.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-03-27T09:52:53.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:52:58.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-03-27T09:53:02.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:53:07.141 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-03-27T09:53:26.533 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:53:35.593 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142 2023-03-27T09:53:42.295 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-27T09:54:02.795 INFO:teuthology.provision.fog.smithi142:Node is ready 2023-03-27T09:54:02.853 INFO:teuthology.orchestra.run.smithi142.stdout:smithi001 2023-03-27T09:54:02.910 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.1 smithi001 smithi001 2023-03-27T09:54:03.269 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2023-03-27T09:54:13.157 WARNING:teuthology.provision.fog:timed out 2023-03-27T09:54:19.588 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-27T09:54:19.846 INFO:teuthology.provision.fog.smithi097:Node is ready 2023-03-27T09:54:19.904 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com 2023-03-27T09:54:19.963 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097 2023-03-27T09:54:20.316 INFO:teuthology.provision.fog.smithi097:Deploy complete! 2023-03-27T09:54:20.513 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com 2023-03-27T09:54:20.514 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-27T09:54:20.539 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2023-03-27T09:54:20.539 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-27T09:54:20.565 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server 2023-03-27T09:54:20.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-03-27T09:54:20.754 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-03-27T09:54:20.811 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2023-03-27T09:54:20.812 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8" 2023-03-27T09:54:20.812 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2023-03-27T09:54:20.812 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2023-03-27T09:54:20.812 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2023-03-27T09:54:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2023-03-27T09:54:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-27T09:54:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2023-03-27T09:54:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-27T09:54:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2023-03-27T09:54:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-27T09:54:20.813 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-27T09:54:20.814 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-27T09:54:20.815 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-03-27T09:54:20.839 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2023-03-27T09:54:20.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2023-03-27T09:54:21.048 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2023-03-27T09:54:21.105 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2023-03-27T09:54:21.105 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8" 2023-03-27T09:54:21.105 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2023-03-27T09:54:21.105 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2023-03-27T09:54:21.105 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-27T09:54:21.106 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-27T09:54:21.107 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2023-03-27T09:54:21.325 INFO:teuthology.dispatcher.supervisor:Running job 7221299 2023-03-27T09:54:21.325 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_main/virtualenv/bin/teuthology -v --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2023-03-27_06:39:30-fs-wip-rishabh-cephman-6-testing-default-smithi/7221299 --name rishabh-2023-03-27_06:39:30-fs-wip-rishabh-cephman-6-testing-default-smithi --description fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/iozone}} -- /home/teuthworker/archive/rishabh-2023-03-27_06:39:30-fs-wip-rishabh-cephman-6-testing-default-smithi/7221299/orig.config.yaml 2023-03-27T09:54:21.328 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rishabh-2023-03-27_06:39:30-fs-wip-rishabh-cephman-6-testing-default-smithi/7221299 2023-03-27T09:54:21.329 INFO:teuthology.dispatcher.supervisor:Job PID: 10515 2023-03-27T09:54:21.329 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-03-27T21:55:15.633 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-27T21:55:15.834 INFO:teuthology.kill:Killing Pids: {10515} 2023-03-27T21:55:16.686 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-03-27T21:55:16.686 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['client.0'] 2023-03-27T21:55:16.687 INFO:teuthology.misc:Compressing logs... 2023-03-27T21:55:16.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-03-27T21:55:16.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2023-03-27T21:55:17.375 INFO:teuthology.misc:Compressing logs... 2023-03-27T21:55:17.437 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-03-27T21:55:17.440 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-03-27T21:55:17.505 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-03-27T21:55:17.505 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-03-27T21:55:17.567 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-03-27T21:55:17.568 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now 2023-03-27T21:55:17.570 INFO:teuthology.misc:Compressing logs... 2023-03-27T21:55:19.870 INFO:teuthology.kill:No teuthology processes running 2023-03-27T21:55:20.680 INFO:teuthology.kill:Nuking machines: ['smithi097', 'smithi142'] 2023-03-27T21:55:21.263 INFO:teuthology.kill:2023-03-27 21:55:21,263.263 INFO:teuthology.nuke:targets: 2023-03-27T21:55:21.264 INFO:teuthology.kill: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUl6eWKv/SpT5MRelHE/Tkb7CSoz+1YXxSVWpWgmJCmCwFVwCNnbL4/En4haoaQTWtLYXBmLWKWrmAPKnRnWgnxQNYywuHCFTJEW8cSHhbKD3HoT/GrqdegOT6FHg/R/Ni2sVDphOcQ8y1L7s85u37Zlvp4nEfcJTXoBMXyzgQ8ru1QEAbkxGqxamKv3F4ZRSccm6ilr3osMJZHGPn3E54r2WUkaNqrMifDQTQgcRZXrOVqA8fz9n0oT8MkjuMGpAQdAxzZPddvTfsIkRqVn2nxrsF2o2NRQDEP+Ch04MwkYCGhVpxevRXBrdejWKxK2A+bgBbsejn1YVM7tDmFObNwKtvPsXcklZ59JEx1RW8IWeyhlynJ6I8MPirshMS8m6crq3CNp83RczKF3fpELQlKMb3fshPu0C+xcdEF+WhYeNg9GUFOMqLMuEPYfqDLZttqRexv6zKCQiVcKaLdgWM9O8vh/NSz70uJDbivZFQ5xk4L+SoXLqNvwCn3It7N08= 2023-03-27T21:55:21.264 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIc0o0wZW9z9hCe3bUWeGe1R7CV5/x+dUBvY6YN3eK89VRn8XYcxzEhJhS+fymb1v5kqsPUsdPf5X/X53Ad3icw= 2023-03-27T21:55:21.264 INFO:teuthology.kill:2023-03-27 21:55:21,263.263 INFO:teuthology.nuke:Checking targets against current locks 2023-03-27T21:55:21.318 INFO:teuthology.kill:2023-03-27 21:55:21,318.318 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-27T21:55:21.342 INFO:teuthology.kill:2023-03-27 21:55:21,341.341 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-27T21:55:21.398 INFO:teuthology.kill:2023-03-27 21:55:21,397.397 INFO:teuthology.orchestra.console:Power off smithi097 2023-03-27T21:55:21.413 INFO:teuthology.kill:2023-03-27 21:55:21,413.413 INFO:teuthology.orchestra.console:Power off smithi142 2023-03-27T21:55:33.886 INFO:teuthology.kill:2023-03-27 21:55:33,885.885 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-03-27T21:55:33.892 INFO:teuthology.kill:2023-03-27 21:55:33,892.892 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2023-03-27T21:55:34.045 INFO:teuthology.kill:2023-03-27 21:55:34,045.045 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com 2023-03-27T21:55:34.053 INFO:teuthology.kill:2023-03-27 21:55:34,053.053 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com 2023-03-27T21:57:34.390 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-03-27T21:57:34.488 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi097 but it was locked by another job: /home/teuthworker/archive/lflores-2023-03-27_20:42:09-rados-wip-aclamk-bs-elastic-shared-blob-quincy-distro-default-smithi/7221559 2023-03-27T21:57:34.489 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/lflores-2023-03-27_20:42:09-rados-wip-aclamk-bs-elastic-shared-blob-quincy-distro-default-smithi/7221560