2022-03-04T12:05:40.529 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-04T12:05:40.657 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging 2022-03-04T12:05:40.659 INFO:teuthology.lock.ops:Updating [ubuntu@smithi072.front.sepia.ceph.com]: reset os type and version on server 2022-03-04T12:05:40.660 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2022-03-04T12:05:40.678 INFO:teuthology.lock.ops:Node 'ubuntu@smithi072.front.sepia.ceph.com' reimaging is complete 2022-03-04T12:05:40.679 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi185.front.sepia.ceph.com' reimaging 2022-03-04T12:05:40.679 INFO:teuthology.lock.ops:Updating [ubuntu@smithi185.front.sepia.ceph.com]: reset os type and version on server 2022-03-04T12:05:40.679 INFO:teuthology.lock.ops:Updating smithi185.front.sepia.ceph.com on lock server 2022-03-04T12:05:40.702 INFO:teuthology.lock.ops:Node 'ubuntu@smithi185.front.sepia.ceph.com' reimaging is complete 2022-03-04T12:05:40.898 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.stream 2022-03-04T12:05:40.908 INFO:teuthology.provision.fog.smithi185:Scheduling deploy of centos 8.stream 2022-03-04T12:05:41.223 INFO:teuthology.orchestra.console:Power off smithi185 2022-03-04T12:05:41.236 INFO:teuthology.orchestra.console:Power off smithi072 2022-03-04T12:05:41.365 INFO:teuthology.orchestra.console:Power off for smithi185 completed 2022-03-04T12:05:41.380 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2022-03-04T12:05:41.468 INFO:teuthology.orchestra.console:Power on smithi185 2022-03-04T12:05:41.481 INFO:teuthology.orchestra.console:Power on smithi072 2022-03-04T12:05:49.838 INFO:teuthology.orchestra.console:Power on for smithi185 completed 2022-03-04T12:05:49.844 INFO:teuthology.orchestra.console:Power on for smithi072 completed 2022-03-04T12:05:49.940 INFO:teuthology.provision.fog.smithi185:Waiting for deploy to finish 2022-03-04T12:05:49.945 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish 2022-03-04T12:10:37.072 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-04T12:10:37.318 INFO:teuthology.provision.fog.smithi185:Node is ready 2022-03-04T12:10:37.377 INFO:teuthology.orchestra.run.smithi185.stdout:smithi185.front.sepia.ceph.com 2022-03-04T12:10:37.435 INFO:teuthology.orchestra.run.smithi185.stdout:172.21.15.185 smithi185.front.sepia.ceph.com smithi185 2022-03-04T12:10:37.778 INFO:teuthology.provision.fog.smithi185:Deploy complete! 2022-03-04T12:10:52.061 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-04T12:10:52.384 INFO:teuthology.provision.fog.smithi072:Node is ready 2022-03-04T12:10:52.403 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com 2022-03-04T12:10:52.461 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072 2022-03-04T12:10:52.858 INFO:teuthology.provision.fog.smithi072:Deploy complete! 2022-03-04T12:10:53.012 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com 2022-03-04T12:10:53.013 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-04T12:10:53.032 INFO:teuthology.lock.ops:Checking smithi185.front.sepia.ceph.com 2022-03-04T12:10:53.032 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-04T12:10:53.050 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server 2022-03-04T12:10:53.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2022-03-04T12:10:53.245 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2022-03-04T12:10:53.301 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2022-03-04T12:10:53.302 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8" 2022-03-04T12:10:53.302 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2022-03-04T12:10:53.302 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2022-03-04T12:10:53.302 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8" 2022-03-04T12:10:53.303 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2022-03-04T12:10:53.303 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-04T12:10:53.303 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2022-03-04T12:10:53.303 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-04T12:10:53.304 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2022-03-04T12:10:53.304 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-04T12:10:53.304 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-04T12:10:53.304 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-04T12:10:53.305 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2022-03-04T12:10:53.326 INFO:teuthology.lock.ops:Updating [smithi185.front.sepia.ceph.com]: set os type and version on server 2022-03-04T12:10:53.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi185.front.sepia.ceph.com' 2022-03-04T12:10:53.573 INFO:teuthology.orchestra.run.smithi185.stdout:x86_64 2022-03-04T12:10:53.630 INFO:teuthology.orchestra.run.smithi185.stdout:NAME="CentOS Stream" 2022-03-04T12:10:53.630 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION="8" 2022-03-04T12:10:53.630 INFO:teuthology.orchestra.run.smithi185.stdout:ID="centos" 2022-03-04T12:10:53.631 INFO:teuthology.orchestra.run.smithi185.stdout:ID_LIKE="rhel fedora" 2022-03-04T12:10:53.631 INFO:teuthology.orchestra.run.smithi185.stdout:VERSION_ID="8" 2022-03-04T12:10:53.631 INFO:teuthology.orchestra.run.smithi185.stdout:PLATFORM_ID="platform:el8" 2022-03-04T12:10:53.631 INFO:teuthology.orchestra.run.smithi185.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-04T12:10:53.632 INFO:teuthology.orchestra.run.smithi185.stdout:ANSI_COLOR="0;31" 2022-03-04T12:10:53.632 INFO:teuthology.orchestra.run.smithi185.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-04T12:10:53.632 INFO:teuthology.orchestra.run.smithi185.stdout:HOME_URL="https://centos.org/" 2022-03-04T12:10:53.632 INFO:teuthology.orchestra.run.smithi185.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-04T12:10:53.632 INFO:teuthology.orchestra.run.smithi185.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-04T12:10:53.633 INFO:teuthology.orchestra.run.smithi185.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-04T12:10:53.633 INFO:teuthology.lock.ops:Updating smithi185.front.sepia.ceph.com on lock server 2022-03-04T12:10:53.744 INFO:teuthology.dispatcher.supervisor:Running job 6720176 2022-03-04T12:10:53.745 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology -v --owner scheduled_nshelke@teuthology --archive /home/teuthworker/archive/nshelke-2022-03-04_11:25:21-fs-wip-nshelke-skip-chmod-operation-testing-default-smithi/6720176 --name nshelke-2022-03-04_11:25:21-fs-wip-nshelke-skip-chmod-operation-testing-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/nshelke-2022-03-04_11:25:21-fs-wip-nshelke-skip-chmod-operation-testing-default-smithi/6720176/orig.config.yaml 2022-03-04T12:10:53.748 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nshelke-2022-03-04_11:25:21-fs-wip-nshelke-skip-chmod-operation-testing-default-smithi/6720176 2022-03-04T12:10:53.749 INFO:teuthology.dispatcher.supervisor:Job PID: 8001 2022-03-04T12:10:53.749 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-04T18:41:22.763 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-04T18:41:22.970 INFO:teuthology.kill:Killing Pids: {8001} 2022-03-04T18:41:24.331 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-03-04T18:41:24.332 INFO:teuthology.task.internal:roles: ubuntu@smithi185.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-03-04T18:41:24.332 INFO:teuthology.misc:Compressing logs... 2022-03-04T18:41:24.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2022-03-04T18:41:24.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi185.front.sepia.ceph.com' 2022-03-04T18:41:26.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/dispatcher/supervisor.py", line 287, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/dispatcher/supervisor.py", line 345, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2022-03-04T18:41:26.135 INFO:teuthology.kill:No teuthology processes running 2022-03-04T18:41:27.338 INFO:teuthology.kill:Nuking machines: ['smithi072', 'smithi185'] 2022-03-04T18:41:28.338 INFO:teuthology.kill:2022-03-04 18:41:28,338.338 INFO:teuthology.nuke:targets: 2022-03-04T18:41:28.339 INFO:teuthology.kill: smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwRbLpmAsDViF5ZsYoIh/Q0exway9PDKB/yk/8+XDxuNrBkgFH0ujHcRQLO/y467CnKhG3aUfgf+ySYkfxudqhxHaRZsctb0rgEOAe3IbiSy7yJAML7hDi3yZPsoaLSE1NrfQFjBskpCXgsuUVKv10QoHGUA0g1J6U4d1XvV2i+bv5nAbArjgGZZFKw+B9ehzYxYE6u2bUyk8OSE9POhqh1FiYgxEnD1aLg/i/53St8qwleSa0m7l+Wn4MqQe38V8xcLWAD6H0E8fyKMPGhrCq5FbGF8hHJNmCz+JAzWKGUc39/6IKoUMKGVIURMxLhYrIRcMeciubh+IbSa8n1XoHj9H9OixrzHUNZqtPySFJHgYR18alxv9E2zD5P1L57pTicN8ig5E44BVizysrGH9srM5MtFo7HveQPAYPgGNyhowBfkltAxJ/svn3UiN3wDZZ4nsjyQ4OiyCvOaLv33ifYjpNBWH1IM58xkULKG1E24bB02TaHuzAdxW4BMOFfgE= 2022-03-04T18:41:28.339 INFO:teuthology.kill: smithi185.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu2zISHVtICE6YvOSOl0blKGwtqBK8y0sBwE7N8F1GlER3Fs8M3Gx3EMfRsPtJjhbxPRWZKm8rbEDh/TyZmKAQL3bvy79jkDYnjKi4MkS18WrqAF3yPPTqS76jWfyQ7+QBu+akzvuv38DUTRyjDiSalbomOeOtF5nf2PsfyuzrmDgpdlVkY2+Yt+9Qwlqhz9Nm5BR8NqHCa8BJTbQbUC0GLuhONHOekfUhszTxOpUGj1sD8TTNUhthBe9EjdQdUVPdS1JgX1MG4ug1iiVZdiGy9ZN2uM7LkeJDitRip6fz6+BuDsaq2iEXkp0468a8v7gN6hKWg+Rh4JX6TFnoUFTG03QtsI7zWUEUq2qgZYCHKWNQtIAOgYkkOVeN/AcGNRw3SuT3Uq2ST8F0IPwGeLsXvhy9tfGR3ScaaJUItdjLZsSeTtgnUHhx0+nUeWZzDnLHjgGBDxOevEP0um2zqV1i2XqPE5zR7wZwwwFZdqOO6R0bgzqAuC9GzTWVILYHk8M= 2022-03-04T18:41:28.365 INFO:teuthology.kill:2022-03-04 18:41:28,365.365 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-04T18:41:28.368 INFO:teuthology.kill:2022-03-04 18:41:28,368.368 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-04T18:41:28.447 INFO:teuthology.kill:2022-03-04 18:41:28,447.447 INFO:teuthology.orchestra.console:Power off smithi072 2022-03-04T18:41:28.461 INFO:teuthology.kill:2022-03-04 18:41:28,461.461 INFO:teuthology.orchestra.console:Power off smithi185 2022-03-04T18:41:36.813 INFO:teuthology.kill:2022-03-04 18:41:36,812.812 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2022-03-04T18:41:36.973 INFO:teuthology.kill:2022-03-04 18:41:36,972.972 INFO:teuthology.lock.ops:unlocked: smithi072.front.sepia.ceph.com 2022-03-04T18:41:40.938 INFO:teuthology.kill:2022-03-04 18:41:40,938.938 INFO:teuthology.orchestra.console:Power off for smithi185 completed 2022-03-04T18:41:41.084 INFO:teuthology.kill:2022-03-04 18:41:41,084.084 INFO:teuthology.lock.ops:unlocked: smithi185.front.sepia.ceph.com 2022-03-04T18:43:41.437 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-04T18:43:41.550 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi072 but it was locked by another job: /home/teuthworker/archive/mrfreezeex-2022-03-04_16:43:54-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6720477