2021-11-14T03:14:29.554 INFO:root:teuthology version: 1.1.0-d4737010
2021-11-14T03:14:29.681 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi125.front.sepia.ceph.com' reimaging
2021-11-14T03:14:29.682 INFO:teuthology.lock.ops:Updating [ubuntu@smithi125.front.sepia.ceph.com]: reset os type and version on server
2021-11-14T03:14:29.683 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server
2021-11-14T03:14:29.702 INFO:teuthology.lock.ops:Node 'ubuntu@smithi125.front.sepia.ceph.com' reimaging is complete
2021-11-14T03:14:29.703 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging
2021-11-14T03:14:29.703 INFO:teuthology.lock.ops:Updating [ubuntu@smithi165.front.sepia.ceph.com]: reset os type and version on server
2021-11-14T03:14:29.704 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-11-14T03:14:29.723 INFO:teuthology.lock.ops:Node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging is complete
2021-11-14T03:14:29.912 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of centos 8.stream
2021-11-14T03:14:29.937 INFO:teuthology.provision.fog.smithi165:Scheduling deploy of centos 8.stream
2021-11-14T03:14:30.217 INFO:teuthology.orchestra.console:Power off smithi125
2021-11-14T03:14:30.257 INFO:teuthology.orchestra.console:Power off smithi165
2021-11-14T03:14:38.589 INFO:teuthology.orchestra.console:Power off for smithi125 completed
2021-11-14T03:14:38.691 INFO:teuthology.orchestra.console:Power on smithi125
2021-11-14T03:14:42.728 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-11-14T03:14:42.830 INFO:teuthology.orchestra.console:Power on smithi165
2021-11-14T03:14:47.052 INFO:teuthology.orchestra.console:Power on for smithi125 completed
2021-11-14T03:14:47.079 INFO:teuthology.orchestra.console:Power on for smithi165 completed
2021-11-14T03:14:47.154 INFO:teuthology.provision.fog.smithi125:Waiting for deploy to finish
2021-11-14T03:14:47.180 INFO:teuthology.provision.fog.smithi165:Waiting for deploy to finish
2021-11-14T03:19:51.969 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-14T03:19:52.050 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-14T03:19:52.265 INFO:teuthology.provision.fog.smithi125:Node is ready
2021-11-14T03:19:52.282 INFO:teuthology.orchestra.run.smithi125.stdout:smithi125.front.sepia.ceph.com
2021-11-14T03:19:52.340 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125 smithi125.front.sepia.ceph.com smithi125
2021-11-14T03:19:52.348 INFO:teuthology.provision.fog.smithi165:Node is ready
2021-11-14T03:19:52.365 INFO:teuthology.orchestra.run.smithi165.stdout:smithi165.front.sepia.ceph.com
2021-11-14T03:19:52.422 INFO:teuthology.orchestra.run.smithi165.stdout:172.21.15.165 smithi165.front.sepia.ceph.com smithi165
2021-11-14T03:19:52.698 INFO:teuthology.provision.fog.smithi125:Deploy complete!
2021-11-14T03:19:52.781 INFO:teuthology.provision.fog.smithi165:Deploy complete!
2021-11-14T03:19:52.925 INFO:teuthology.lock.ops:Checking smithi125.front.sepia.ceph.com
2021-11-14T03:19:52.926 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-14T03:19:52.956 INFO:teuthology.lock.ops:Checking smithi165.front.sepia.ceph.com
2021-11-14T03:19:52.956 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-14T03:19:52.985 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: set os type and version on server
2021-11-14T03:19:52.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com'
2021-11-14T03:19:53.212 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64
2021-11-14T03:19:53.267 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="CentOS Stream"
2021-11-14T03:19:53.268 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="8"
2021-11-14T03:19:53.268 INFO:teuthology.orchestra.run.smithi125.stdout:ID="centos"
2021-11-14T03:19:53.268 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE="rhel fedora"
2021-11-14T03:19:53.269 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="8"
2021-11-14T03:19:53.269 INFO:teuthology.orchestra.run.smithi125.stdout:PLATFORM_ID="platform:el8"
2021-11-14T03:19:53.269 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-14T03:19:53.269 INFO:teuthology.orchestra.run.smithi125.stdout:ANSI_COLOR="0;31"
2021-11-14T03:19:53.269 INFO:teuthology.orchestra.run.smithi125.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-14T03:19:53.270 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://centos.org/"
2021-11-14T03:19:53.270 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-14T03:19:53.270 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-14T03:19:53.270 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-14T03:19:53.272 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server
2021-11-14T03:19:53.292 INFO:teuthology.lock.ops:Updating [smithi165.front.sepia.ceph.com]: set os type and version on server
2021-11-14T03:19:53.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com'
2021-11-14T03:19:53.516 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64
2021-11-14T03:19:53.571 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream"
2021-11-14T03:19:53.572 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="8"
2021-11-14T03:19:53.572 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos"
2021-11-14T03:19:53.572 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora"
2021-11-14T03:19:53.573 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="8"
2021-11-14T03:19:53.573 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el8"
2021-11-14T03:19:53.573 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-14T03:19:53.573 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31"
2021-11-14T03:19:53.574 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-14T03:19:53.574 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/"
2021-11-14T03:19:53.574 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-14T03:19:53.574 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-14T03:19:53.575 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-14T03:19:53.575 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-11-14T03:19:53.677 INFO:teuthology.dispatcher.supervisor:Running job 6501548
2021-11-14T03:19:53.678 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-13_15:31:06-rados-wip-yuriw-master-11.12.21-distro-basic-smithi/6501548 --name yuriw-2021-11-13_15:31:06-rados-wip-yuriw-master-11.12.21-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2021-11-13_15:31:06-rados-wip-yuriw-master-11.12.21-distro-basic-smithi/6501548/orig.config.yaml
2021-11-14T03:19:53.681 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-13_15:31:06-rados-wip-yuriw-master-11.12.21-distro-basic-smithi/6501548
2021-11-14T03:19:53.682 INFO:teuthology.dispatcher.supervisor:Job PID: 21519
2021-11-14T03:19:53.682 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-14T15:20:48.123 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-14T15:20:48.356 INFO:teuthology.kill:Killing Pids: {21519}
2021-11-14T15:20:49.603 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-14T15:20:49.604 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-14T15:20:49.604 INFO:teuthology.misc:Compressing logs...
2021-11-14T15:20:49.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com'
2021-11-14T15:20:49.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com'
2021-11-14T15:20:50.192 INFO:teuthology.misc:Compressing logs...
2021-11-14T15:20:50.231 INFO:teuthology.orchestra.run.smithi125.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-14T15:20:50.258 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-14T15:20:50.301 INFO:teuthology.orchestra.run.smithi125.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-14T15:20:50.301 INFO:teuthology.orchestra.run.smithi125.stderr:tar: Error is not recoverable: exiting now
2021-11-14T15:20:50.326 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-14T15:20:50.327 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now
2021-11-14T15:20:50.328 INFO:teuthology.misc:Compressing logs...
2021-11-14T15:20:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-14T15:20:50.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d4737010a85099043cf081dc05b4069d301b23fb/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-14T15:20:50.440 INFO:teuthology.kill:No teuthology processes running
2021-11-14T15:20:51.580 INFO:teuthology.kill:Nuking machines: ['smithi125', 'smithi165']
2021-11-14T15:20:52.567 INFO:teuthology.kill:2021-11-14 15:20:52,566.566 INFO:teuthology.nuke:targets:
2021-11-14T15:20:52.568 INFO:teuthology.kill:  smithi125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3Twab1DYGDhAfE2hWPMvbvl/SoJO2+J9A0Po3Uouveux5e0ESBYaYGHIMqcVQSOroZiENF6LDu6uIEAoZBIquJ8fjZSFF6G0Ff4/2ixVCffApJa3LVGMhEZPtamBEwkEjhYyk29Zs+uco6YJWeFJxD21ZjIBA1SdzDfuMrEl4aivBVp05oq3REUBq9CedT41MB468glIqWw5O1gq0kpQIspEQbM0U1qXrrIxrDiYrcIElNt0VZMA4iw5ULwFUVDQHhJiWeP9ruxw05/0bXbireOI6OjANKNtPzTeIXwFxjYzwDveco9cDzyITEe7rRP8znfTsFEC+vGt8pCvXGH+NYcpszNSDXPGD8svh7T2E2KMNFrHsrJBzDgommEB5kVQD3H8jWb5V4eC3ClDaDPPJymljt73i6SbsxBDFyJva2WxCrnuM0ywHvRr0tAreCwXdSsIH9mVjmHbcbab26m0fS+/pt9eqB4TVCQjZiPXhOmoqv0AUQAELeRzmfe+XVK8=
2021-11-14T15:20:52.568 INFO:teuthology.kill:  smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6GozbZyWV5+uOGxRn9OCM4Jn1TEc2lr9X2j690LbQRtNqa+e84zKiW+gU0eMOHkJEvWO2Pdmi1p4zFiVe+/5I8KFGOBviRO5Dnzyxo3if5ECWhyPsZsuzb/AoPx1eA3hi7gslSpGgEfxYgsieAIb2IvsDZCDWkgPtQ2LKb28vLffh+E7aJNgcwfn2oslIwDqSIpjKwQrqUKo+AItYq4JAIj+57x2yUImvKJVM1bTVOp/liAC79en7f87/DBzfZRV0INkc+di+JNG4v81xD0LybCFyUvDE9ys0FiBWkNaAUJYyFk2pEoaRjiGfkPGopmKc7rP1H/bijj3FmcovrNjJ7Ra75ogsvDihgAeYs+vkq8pHzRHgWXtp5MLTSTDEIMO5m/fn5+KETWSJBcjZu5Lm3B/irywY707+wYbVjcFB5Uz8xXIGqr5JZ5F2EDDCr8+t4LEw94fPbeEGBvQ8xIllPBIkqsxtMfHGI7SzQhHZjo4WcngbIiwdQYTL/t8chF0=
2021-11-14T15:20:52.594 INFO:teuthology.kill:2021-11-14 15:20:52,593.593 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-14T15:20:52.598 INFO:teuthology.kill:2021-11-14 15:20:52,598.598 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-14T15:20:52.677 INFO:teuthology.kill:2021-11-14 15:20:52,677.677 INFO:teuthology.orchestra.console:Power off smithi165
2021-11-14T15:20:52.690 INFO:teuthology.kill:2021-11-14 15:20:52,690.690 INFO:teuthology.orchestra.console:Power off smithi125
2021-11-14T15:21:05.159 INFO:teuthology.kill:2021-11-14 15:21:05,159.159 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-11-14T15:21:05.165 INFO:teuthology.kill:2021-11-14 15:21:05,165.165 INFO:teuthology.orchestra.console:Power off for smithi125 completed
2021-11-14T15:21:05.329 INFO:teuthology.kill:2021-11-14 15:21:05,328.328 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com
2021-11-14T15:21:05.330 INFO:teuthology.kill:2021-11-14 15:21:05,330.330 INFO:teuthology.lock.ops:unlocked: smithi165.front.sepia.ceph.com
2021-11-14T15:23:05.644 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-14T15:23:05.775 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi125 but it was locked by another job: /home/teuthworker/archive/rfriedma-2021-11-14_15:03:09-rados-wip-ronenf-unique-scrub-distro-basic-smithi/6503212
2021-11-14T15:23:05.775 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi165 but it was locked by another job: /home/teuthworker/archive/rfriedma-2021-11-14_15:03:09-rados-wip-ronenf-unique-scrub-distro-basic-smithi/6503212