2023-05-03T09:02:11.726 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-03T09:02:11.883 INFO:teuthology.lock.ops:Start node 'smithi016.front.sepia.ceph.com' reimaging
2023-05-03T09:02:11.885 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: reset os type and version on server
2023-05-03T09:02:11.885 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server
2023-05-03T09:02:11.915 INFO:teuthology.lock.ops:Node 'smithi016.front.sepia.ceph.com' reimaging is complete
2023-05-03T09:02:11.916 INFO:teuthology.lock.ops:Start node 'smithi177.front.sepia.ceph.com' reimaging
2023-05-03T09:02:11.916 INFO:teuthology.lock.ops:Updating [smithi177.front.sepia.ceph.com]: reset os type and version on server
2023-05-03T09:02:11.916 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server
2023-05-03T09:02:11.939 INFO:teuthology.lock.ops:Node 'smithi177.front.sepia.ceph.com' reimaging is complete
2023-05-03T09:02:12.087 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of centos 8.stream
2023-05-03T09:02:12.090 INFO:teuthology.provision.fog.smithi177:Scheduling deploy of centos 8.stream
2023-05-03T09:02:12.386 INFO:teuthology.orchestra.console:Power off smithi177
2023-05-03T09:02:12.403 INFO:teuthology.orchestra.console:Power off smithi016
2023-05-03T09:02:12.536 INFO:teuthology.orchestra.console:Power off for smithi177 completed
2023-05-03T09:02:12.552 INFO:teuthology.orchestra.console:Power off for smithi016 completed
2023-05-03T09:02:12.638 INFO:teuthology.orchestra.console:Power on smithi177
2023-05-03T09:02:12.653 INFO:teuthology.orchestra.console:Power on smithi016
2023-05-03T09:02:16.901 INFO:teuthology.orchestra.console:Power on for smithi177 completed
2023-05-03T09:02:17.002 INFO:teuthology.provision.fog.smithi177:Waiting for deploy to finish
2023-05-03T09:02:21.024 INFO:teuthology.orchestra.console:Power on for smithi016 completed
2023-05-03T09:02:21.125 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish
2023-05-03T09:05:09.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-05-03T09:05:19.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-05-03T09:05:21.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177
2023-05-03T09:05:28.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-05-03T09:05:30.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177
2023-05-03T09:05:37.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-05-03T09:05:39.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177
2023-05-03T09:05:46.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-05-03T09:05:48.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177
2023-05-03T09:06:52.245 WARNING:teuthology.provision.fog:timed out
2023-05-03T09:06:54.365 WARNING:teuthology.provision.fog:timed out
2023-05-03T09:06:58.568 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-03T09:06:58.831 INFO:teuthology.provision.fog.smithi016:Node is ready
2023-05-03T09:06:58.849 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016.front.sepia.ceph.com
2023-05-03T09:06:58.909 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16 smithi016.front.sepia.ceph.com smithi016
2023-05-03T09:06:59.259 INFO:teuthology.provision.fog.smithi016:Deploy complete!
2023-05-03T09:07:00.435 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-03T09:07:00.567 INFO:teuthology.provision.fog.smithi177:Node is ready
2023-05-03T09:07:00.624 INFO:teuthology.orchestra.run.smithi177.stdout:smithi177.front.sepia.ceph.com
2023-05-03T09:07:00.682 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.15.177 smithi177.front.sepia.ceph.com smithi177
2023-05-03T09:07:01.020 INFO:teuthology.provision.fog.smithi177:Deploy complete!
2023-05-03T09:07:01.182 INFO:teuthology.lock.ops:Checking smithi016.front.sepia.ceph.com
2023-05-03T09:07:01.183 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-03T09:07:01.213 INFO:teuthology.lock.ops:Checking smithi177.front.sepia.ceph.com
2023-05-03T09:07:01.213 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-03T09:07:01.239 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: set os type and version on server
2023-05-03T09:07:01.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com'
2023-05-03T09:07:01.439 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64
2023-05-03T09:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream"
2023-05-03T09:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8"
2023-05-03T09:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos"
2023-05-03T09:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora"
2023-05-03T09:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8"
2023-05-03T09:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8"
2023-05-03T09:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-03T09:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31"
2023-05-03T09:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-03T09:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/"
2023-05-03T09:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-03T09:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-03T09:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-03T09:07:01.504 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server
2023-05-03T09:07:01.522 INFO:teuthology.lock.ops:Updating [smithi177.front.sepia.ceph.com]: set os type and version on server
2023-05-03T09:07:01.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com'
2023-05-03T09:07:01.715 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64
2023-05-03T09:07:01.772 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream"
2023-05-03T09:07:01.772 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="8"
2023-05-03T09:07:01.772 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos"
2023-05-03T09:07:01.772 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora"
2023-05-03T09:07:01.773 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="8"
2023-05-03T09:07:01.773 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el8"
2023-05-03T09:07:01.773 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-03T09:07:01.773 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31"
2023-05-03T09:07:01.773 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-03T09:07:01.774 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/"
2023-05-03T09:07:01.774 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-03T09:07:01.774 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-03T09:07:01.774 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-03T09:07:01.775 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server
2023-05-03T09:07:01.877 INFO:teuthology.dispatcher.supervisor:Running job 7256311
2023-05-03T09:07:01.877 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/bin/teuthology -v --owner scheduled_suriarte@teuthology --archive /home/teuthworker/archive/suriarte-2023-04-27_20:25:27-rgw-wip-d4n-distro-default-smithi/7256311 --name suriarte-2023-04-27_20:25:27-rgw-wip-d4n-distro-default-smithi --description rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides} -- /home/teuthworker/archive/suriarte-2023-04-27_20:25:27-rgw-wip-d4n-distro-default-smithi/7256311/orig.config.yaml
2023-05-03T09:07:01.881 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/suriarte-2023-04-27_20:25:27-rgw-wip-d4n-distro-default-smithi/7256311
2023-05-03T09:07:01.882 INFO:teuthology.dispatcher.supervisor:Job PID: 28374
2023-05-03T09:07:01.882 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-03T21:08:00.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-03T21:08:01.046 INFO:teuthology.kill:Killing Pids: {28374}
2023-05-03T21:08:01.993 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2023-05-03T21:08:01.994 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2023-05-03T21:08:01.994 INFO:teuthology.misc:Compressing logs...
2023-05-03T21:08:01.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com'
2023-05-03T21:08:02.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com'
2023-05-03T21:08:02.492 INFO:teuthology.misc:Compressing logs...
2023-05-03T21:08:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2023-05-03T21:08:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2023-05-03T21:08:02.537 INFO:teuthology.orchestra.run.smithi016.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	/home/ubuntu/cephtest/archive/syslog/kern.log:	 81.6% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2023-05-03T21:08:02.546 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2023-05-03T21:08:02.561 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2023-05-03T21:08:02.561 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-05-03T21:08:02.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --'
2023-05-03T21:08:02.635 INFO:teuthology.kill:No teuthology processes running
2023-05-03T21:08:03.473 INFO:teuthology.kill:Nuking machines: ['smithi016', 'smithi177']
2023-05-03T21:08:04.259 INFO:teuthology.kill:2023-05-03 21:08:04,259.259 INFO:teuthology.nuke:targets:
2023-05-03T21:08:04.260 INFO:teuthology.kill:  smithi016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC13s6t1plvwVilNR+B5DPLVwQFWAsd0Vq2oZV6hmKcR3Fd6Qq+p2/U/vsyWd6vF3yZVoWW/MVhDgJpgx1ZX80sGsbPkstHvPmwn8mLEIJvZhSNOLJuG66wSWo7Dt7YHvpd3bTR3MPp426Z3ljZQKmcxaQEmuAPu+KeOrV2/Szi2cuS1ExW6bVYD9+2NZ2ECgWiyE8e2wISs/fJCKxfuwHBbiZHT5b/IfZ361pCcIfXiHDx2dMkfrhO+E17vigB4lymPKEBVtda5Wup/RZ09js/KKJbICKryKhiV6pafzx/mj5KHHjsMtBoteBRRVSIMbC3kmmF/fhtHON7ot99vzRebJFkPTMFa7ktoM5RnOMefjkjqbPOX54U6UTVzNDYtwZFaQnN1+CyKY7uhrgh9yd5PzEWXy0nKahzymkFxTkn1BgQrrWGBq6JbtqROhMshMSG19wlZYWlTwDSfy41jnC4qkALejb5DxKDOLvLXrAYFa7l8FYNNVgFnZySJzuNwlM=
2023-05-03T21:08:04.260 INFO:teuthology.kill:  smithi177.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN9FTXhdrvOChPO2uZp4ky6TY3JCeLrNMaXrHXxQERZyMLFzXiX1pvjTOCbEuc3VD1731nYbawXsNjP+Mm7wuHYMbH9dYveFSlSCq4CTdror/D++tWXqYfAzBilhjUybCrQeWudJsTQ0C87GTDkYeK91ThwKVv+Yw9NYHR9BZ2BS/cb1tuGVBFX1XTpEXed8iSdER3dvl9OtjkTA/u+hwRD910b1QoIlbR5H1qdT87UQyo3M0orAe3KaNbugRBfhQcumTAA4FoP9ZewgJer+vHPRTHC+eV/o9NWw41ehTHE2wf0twCMBmEhXg4A9ni2eWZxnQ0oaYI7y0MI1l3iSFvzRmfiq7GK5jgoN9bdK5z4DMpVisv4adMlz+9ylo8cyO1WfkXwOQOmq61+D2dTIJ/3IBBpT0BKXUPOO7ewgckdzFrS+1SqSSKQPU4oQxDpCPJYOMtXed41O4HYHmGymskipPd4+QA8TqD55nxxiuhlPo+jc2MdoXXOf1tPjW4Tys=
2023-05-03T21:08:04.260 INFO:teuthology.kill:2023-05-03 21:08:04,259.259 INFO:teuthology.nuke:Checking targets against current locks
2023-05-03T21:08:04.323 INFO:teuthology.kill:2023-05-03 21:08:04,323.323 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-03T21:08:04.350 INFO:teuthology.kill:2023-05-03 21:08:04,350.350 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-03T21:08:04.409 INFO:teuthology.kill:2023-05-03 21:08:04,409.409 INFO:teuthology.orchestra.console:Power off smithi016
2023-05-03T21:08:04.453 INFO:teuthology.kill:2023-05-03 21:08:04,452.452 INFO:teuthology.orchestra.console:Power off smithi177
2023-05-03T21:08:12.816 INFO:teuthology.kill:2023-05-03 21:08:12,815.815 INFO:teuthology.orchestra.console:Power off for smithi016 completed
2023-05-03T21:08:12.978 INFO:teuthology.kill:2023-05-03 21:08:12,977.977 INFO:teuthology.lock.ops:unlocked: smithi016.front.sepia.ceph.com
2023-05-03T21:08:16.935 INFO:teuthology.kill:2023-05-03 21:08:16,934.934 INFO:teuthology.orchestra.console:Power off for smithi177 completed
2023-05-03T21:08:17.094 INFO:teuthology.kill:2023-05-03 21:08:17,093.093 INFO:teuthology.lock.ops:unlocked: smithi177.front.sepia.ceph.com
2023-05-03T21:10:17.469 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-03T21:10:17.577 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi016 but it was locked by another job: /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261208
2023-05-03T21:10:17.577 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi177 but it was locked by another job: /home/teuthworker/archive/vshankar-2023-05-03_12:59:37-fs-wip-vshankar-testing-20230503.090416-testing-default-smithi/7261208