2022-03-26T17:09:05.345 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-26T17:09:05.479 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi017.front.sepia.ceph.com' reimaging 2022-03-26T17:09:05.481 INFO:teuthology.lock.ops:Updating [ubuntu@smithi017.front.sepia.ceph.com]: reset os type and version on server 2022-03-26T17:09:05.481 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2022-03-26T17:09:05.503 INFO:teuthology.lock.ops:Node 'ubuntu@smithi017.front.sepia.ceph.com' reimaging is complete 2022-03-26T17:09:05.503 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi022.front.sepia.ceph.com' reimaging 2022-03-26T17:09:05.503 INFO:teuthology.lock.ops:Updating [ubuntu@smithi022.front.sepia.ceph.com]: reset os type and version on server 2022-03-26T17:09:05.504 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2022-03-26T17:09:05.523 INFO:teuthology.lock.ops:Node 'ubuntu@smithi022.front.sepia.ceph.com' reimaging is complete 2022-03-26T17:09:05.703 INFO:teuthology.provision.fog.smithi017:Scheduling deploy of centos 8.stream 2022-03-26T17:09:05.719 INFO:teuthology.provision.fog.smithi022:Scheduling deploy of centos 8.stream 2022-03-26T17:09:06.005 INFO:teuthology.orchestra.console:Power off smithi017 2022-03-26T17:09:06.022 INFO:teuthology.orchestra.console:Power off smithi022 2022-03-26T17:09:14.384 INFO:teuthology.orchestra.console:Power off for smithi017 completed 2022-03-26T17:09:14.389 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2022-03-26T17:09:14.485 INFO:teuthology.orchestra.console:Power on smithi017 2022-03-26T17:09:14.493 INFO:teuthology.orchestra.console:Power on smithi022 2022-03-26T17:09:22.859 INFO:teuthology.orchestra.console:Power on for smithi017 completed 2022-03-26T17:09:22.864 INFO:teuthology.orchestra.console:Power on for smithi022 completed 2022-03-26T17:09:22.961 INFO:teuthology.provision.fog.smithi017:Waiting for deploy to finish 2022-03-26T17:09:22.965 INFO:teuthology.provision.fog.smithi022:Waiting for deploy to finish 2022-03-26T17:13:05.028 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-26T17:13:11.225 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-26T17:13:26.131 INFO:teuthology.provision.fog.smithi017:Node is ready 2022-03-26T17:13:26.152 INFO:teuthology.orchestra.run.smithi017.stdout:smithi107 2022-03-26T17:13:26.211 INFO:teuthology.orchestra.run.smithi017.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107 2022-03-26T17:13:26.697 INFO:teuthology.provision.fog.smithi017:Deploy complete! 2022-03-26T17:13:31.591 INFO:teuthology.provision.fog.smithi022:Node is ready 2022-03-26T17:13:31.615 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022.front.sepia.ceph.com 2022-03-26T17:13:31.674 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022 2022-03-26T17:13:32.041 INFO:teuthology.provision.fog.smithi022:Deploy complete! 2022-03-26T17:13:32.190 INFO:teuthology.lock.ops:Checking smithi017.front.sepia.ceph.com 2022-03-26T17:13:32.191 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-26T17:13:32.208 INFO:teuthology.lock.ops:Checking smithi022.front.sepia.ceph.com 2022-03-26T17:13:32.208 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-26T17:13:32.225 INFO:teuthology.lock.ops:Updating [smithi017.front.sepia.ceph.com]: set os type and version on server 2022-03-26T17:13:32.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2022-03-26T17:13:32.428 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2022-03-26T17:13:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2022-03-26T17:13:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="8" 2022-03-26T17:13:32.487 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2022-03-26T17:13:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2022-03-26T17:13:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="8" 2022-03-26T17:13:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el8" 2022-03-26T17:13:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-26T17:13:32.488 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2022-03-26T17:13:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-26T17:13:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2022-03-26T17:13:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-26T17:13:32.489 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-26T17:13:32.490 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-26T17:13:32.491 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2022-03-26T17:13:32.509 INFO:teuthology.lock.ops:Updating [smithi022.front.sepia.ceph.com]: set os type and version on server 2022-03-26T17:13:32.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2022-03-26T17:13:32.708 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2022-03-26T17:13:32.767 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2022-03-26T17:13:32.767 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="8" 2022-03-26T17:13:32.767 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2022-03-26T17:13:32.768 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2022-03-26T17:13:32.768 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="8" 2022-03-26T17:13:32.768 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el8" 2022-03-26T17:13:32.768 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-26T17:13:32.768 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2022-03-26T17:13:32.769 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-26T17:13:32.769 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2022-03-26T17:13:32.769 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-26T17:13:32.769 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-26T17:13:32.770 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-26T17:13:32.770 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2022-03-26T17:13:32.864 INFO:teuthology.dispatcher.supervisor:Running job 6757503 2022-03-26T17:13:32.864 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757503 --name vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757503/orig.config.yaml 2022-03-26T17:13:32.868 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757503 2022-03-26T17:13:32.869 INFO:teuthology.dispatcher.supervisor:Job PID: 24156 2022-03-26T17:13:32.870 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-26T23:44:00.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-26T23:44:01.138 INFO:teuthology.kill:Killing Pids: {24156} 2022-03-26T23:44:02.279 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-03-26T23:44:02.280 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-03-26T23:44:02.280 INFO:teuthology.misc:Compressing logs... 2022-03-26T23:44:02.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2022-03-26T23:44:02.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2022-03-26T23:44:04.235 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/dispatcher/supervisor.py", line 287, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/dispatcher/supervisor.py", line 345, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2022-03-26T23:44:04.316 INFO:teuthology.kill:No teuthology processes running 2022-03-26T23:44:05.456 INFO:teuthology.kill:Nuking machines: ['smithi017', 'smithi022'] 2022-03-26T23:44:06.437 INFO:teuthology.kill:2022-03-26 23:44:06,436.436 INFO:teuthology.nuke:targets: 2022-03-26T23:44:06.438 INFO:teuthology.kill: smithi017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgX3x77nf0onJ7HZl1QMM14t/qaRxOcXlKlXS5zMihp6EZ150dWyHGkjJHJ/dmfQKHP+uOI4S5i0FOZQqMWtgCUY3jYi9Bx1Iczubbak5SVHE/JpOJ81EIVGR/Z3h4m8fNeRztBmjCnOh/bkKCHYHNS1tjjHrKDN/g102SmP74wOQWXycGdhFVHiGzUfIFQgJlVvna11hw0hRKUVF6vVwLIeIGrYn9ldS6el6GTWpd5vgPEZ/7Bxm6C7iKR3nWrTrXbz8FEe12eFq4lg0bvhtfWMzRgqYVjM/tZCCiWxUp9zbHrLeMSg0zKd8ghVFe2SvmZUY2ypuJf/YcfJsKfKKXqj2gnTBaDPxYLAQrOAoXxzM9ZqtBJKU4uyxIk/K9ReYxo4jSDD5Fijbc3mbT/qQkP0jf3uuidnkdLmWdmD1FkCTposhvQtS5erGnibTqgcoLMaK9c9kytqnWpjUFtb8Xj1q+9P41Blx9tXj/CjjYRzI8QB7f9LcVNPmokxhiJ+0= 2022-03-26T23:44:06.438 INFO:teuthology.kill: smithi022.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7IkpEnpsSVo6m1mjrK2ukyXKQD051qiWV8jcqkadiuMugBL7iQJF8vkZilOpPqjbDLHtGCiItsgq8W97tMkN6HKN5dpVkofreT/QDUHGMxtBEdHzDEF5/UgKC/9suzb2vJz96N2LV8XdylXeREzneMhD1OZZTQk+v1+im9SL3nJ5rrIqyvT4My6yDjEzPZzAQcl2S49lvPRMcFSpuXpH0U6Ci+raMiNp+QgQjIY3pyYmZAv4KvRjW/Kz8T44f+xPnWmFZLuJT5OAocfUEeCkx3zP1qn2LsGkTfHhYlC5mdWWjkG/TdLvvUySqQsDK4RSGmUXZoctFNFApbK+kNONPFtoJRIAE9ukvNUJ7728OX8mdeMYYWYTE79Av26VnpJsAArIYT54GWLHFnsKu4Ig2EPIlLxg9uaeXdv5pgkw7N3oatR0dkuC88y7v9KPqJqnZGyoQ90HpHU6V7GrN1RvolHo5GU5/gzDdCt8oH21W0J3R1UOHIF2nDWSHYNfdzes= 2022-03-26T23:44:06.466 INFO:teuthology.kill:2022-03-26 23:44:06,466.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-26T23:44:06.469 INFO:teuthology.kill:2022-03-26 23:44:06,469.469 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-26T23:44:06.540 INFO:teuthology.kill:2022-03-26 23:44:06,540.540 INFO:teuthology.orchestra.console:Power off smithi022 2022-03-26T23:44:06.554 INFO:teuthology.kill:2022-03-26 23:44:06,554.554 INFO:teuthology.orchestra.console:Power off smithi017 2022-03-26T23:44:19.026 INFO:teuthology.kill:2022-03-26 23:44:19,026.026 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2022-03-26T23:44:19.032 INFO:teuthology.kill:2022-03-26 23:44:19,032.032 INFO:teuthology.orchestra.console:Power off for smithi017 completed 2022-03-26T23:44:19.175 INFO:teuthology.kill:2022-03-26 23:44:19,174.174 INFO:teuthology.lock.ops:unlocked: smithi022.front.sepia.ceph.com 2022-03-26T23:44:19.182 INFO:teuthology.kill:2022-03-26 23:44:19,182.182 INFO:teuthology.lock.ops:unlocked: smithi017.front.sepia.ceph.com 2022-03-26T23:46:19.485 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-26T23:46:19.604 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi017 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757595 2022-03-26T23:46:19.605 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi022 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757595