2021-11-16T15:15:31.312 INFO:root:teuthology version: 1.1.0-6cfddd5e
2021-11-16T15:15:31.451 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi146.front.sepia.ceph.com' reimaging
2021-11-16T15:15:31.452 INFO:teuthology.lock.ops:Updating [ubuntu@smithi146.front.sepia.ceph.com]: reset os type and version on server
2021-11-16T15:15:31.453 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2021-11-16T15:15:31.483 INFO:teuthology.lock.ops:Node 'ubuntu@smithi146.front.sepia.ceph.com' reimaging is complete
2021-11-16T15:15:31.484 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging
2021-11-16T15:15:31.484 INFO:teuthology.lock.ops:Updating [ubuntu@smithi162.front.sepia.ceph.com]: reset os type and version on server
2021-11-16T15:15:31.485 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server
2021-11-16T15:15:31.517 INFO:teuthology.lock.ops:Node 'ubuntu@smithi162.front.sepia.ceph.com' reimaging is complete
2021-11-16T15:15:31.701 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of centos 8.stream
2021-11-16T15:15:31.727 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of centos 8.stream
2021-11-16T15:15:32.051 INFO:teuthology.orchestra.console:Power off smithi146
2021-11-16T15:15:32.067 INFO:teuthology.orchestra.console:Power off smithi162
2021-11-16T15:15:40.425 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2021-11-16T15:15:40.527 INFO:teuthology.orchestra.console:Power on smithi146
2021-11-16T15:15:44.539 INFO:teuthology.orchestra.console:Power off for smithi162 completed
2021-11-16T15:15:44.641 INFO:teuthology.orchestra.console:Power on smithi162
2021-11-16T15:15:48.906 INFO:teuthology.orchestra.console:Power on for smithi162 completed
2021-11-16T15:15:48.996 INFO:teuthology.orchestra.console:Power on for smithi146 completed
2021-11-16T15:15:49.011 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish
2021-11-16T15:15:49.097 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish
2021-11-16T15:20:18.491 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-16T15:20:39.055 INFO:teuthology.provision.fog.smithi162:Node is ready
2021-11-16T15:20:39.076 INFO:teuthology.orchestra.run.smithi162.stdout:smithi162.front.sepia.ceph.com
2021-11-16T15:20:39.133 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.15.162 smithi162.front.sepia.ceph.com smithi162
2021-11-16T15:20:39.494 INFO:teuthology.provision.fog.smithi162:Deploy complete!
2021-11-16T15:20:53.750 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-16T15:20:54.017 INFO:teuthology.provision.fog.smithi146:Node is ready
2021-11-16T15:20:54.035 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com
2021-11-16T15:20:54.092 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146
2021-11-16T15:20:54.425 INFO:teuthology.provision.fog.smithi146:Deploy complete!
2021-11-16T15:20:54.603 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com
2021-11-16T15:20:54.604 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-16T15:20:54.627 INFO:teuthology.lock.ops:Checking smithi162.front.sepia.ceph.com
2021-11-16T15:20:54.628 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-16T15:20:54.648 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server
2021-11-16T15:20:54.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2021-11-16T15:20:54.832 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64
2021-11-16T15:20:54.887 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream"
2021-11-16T15:20:54.888 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8"
2021-11-16T15:20:54.888 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos"
2021-11-16T15:20:54.888 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora"
2021-11-16T15:20:54.888 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8"
2021-11-16T15:20:54.889 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8"
2021-11-16T15:20:54.889 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-16T15:20:54.889 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31"
2021-11-16T15:20:54.890 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-16T15:20:54.890 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/"
2021-11-16T15:20:54.890 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-16T15:20:54.891 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-16T15:20:54.891 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-16T15:20:54.894 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2021-11-16T15:20:54.918 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: set os type and version on server
2021-11-16T15:20:54.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com'
2021-11-16T15:20:55.168 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64
2021-11-16T15:20:55.225 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream"
2021-11-16T15:20:55.227 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8"
2021-11-16T15:20:55.227 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos"
2021-11-16T15:20:55.227 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora"
2021-11-16T15:20:55.228 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8"
2021-11-16T15:20:55.228 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8"
2021-11-16T15:20:55.228 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 8"
2021-11-16T15:20:55.228 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31"
2021-11-16T15:20:55.229 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-16T15:20:55.229 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/"
2021-11-16T15:20:55.229 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-11-16T15:20:55.229 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-11-16T15:20:55.229 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-11-16T15:20:55.230 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server
2021-11-16T15:20:55.332 INFO:teuthology.dispatcher.supervisor:Running job 6506501
2021-11-16T15:20:55.332 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6506501 --name yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6506501/orig.config.yaml
2021-11-16T15:20:55.337 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-16_13:07:14-rados-wip-yuri4-testing-2021-11-15-1306-distro-basic-smithi/6506501
2021-11-16T15:20:55.338 INFO:teuthology.dispatcher.supervisor:Job PID: 13486
2021-11-16T15:20:55.338 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-17T03:21:49.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-17T03:21:49.824 INFO:teuthology.kill:Killing Pids: {13486}
2021-11-17T03:21:51.089 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-17T03:21:51.090 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-17T03:21:51.091 INFO:teuthology.misc:Compressing logs...
2021-11-17T03:21:51.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2021-11-17T03:21:51.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com'
2021-11-17T03:21:51.619 INFO:teuthology.misc:Compressing logs...
2021-11-17T03:21:51.656 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-17T03:21:51.685 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-17T03:21:51.724 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-17T03:21:51.725 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2021-11-17T03:21:51.750 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-17T03:21:51.750 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now
2021-11-17T03:21:51.751 INFO:teuthology.misc:Compressing logs...
2021-11-17T03:21:51.793 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-17T03:21:51.795 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-17T03:21:51.863 INFO:teuthology.kill:No teuthology processes running
2021-11-17T03:21:52.994 INFO:teuthology.kill:Nuking machines: ['smithi146', 'smithi162']
2021-11-17T03:21:53.955 INFO:teuthology.kill:2021-11-17 03:21:53,954.954 INFO:teuthology.nuke:targets:
2021-11-17T03:21:53.956 INFO:teuthology.kill:  smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpWaqWOwFO0ql37kwGutPyYx73yNNcChxzAXsurmP4ix/DSGw3iRpiUEYrFTp2tsqox2BcDDyXflhn47tiKzwH5om1K1Wj72MwyhiAfd+mJHP6bxAZWAgIT+LoFbWVXrqFXySf+Iqyi9mbCBsB9rjJp3v1PxyAbIrGNTLYQCct8ppoHgaTiFOXyf+o9REcT/pncsr4wmwfU8/g6XhIb1YKaJ+Wg06uFGUCWITMvV6Gvd/BFGNATQ6Lg0So071Ba0dJ2Dwk+Z+Wa9va1Z4iyVab99nlYxmI8BsqvrWd99dZHLqMvp88eJEHW4uUrS4n55/x0VLpQLNSluy5e4lB9a2jm0WUUOfa4i9OwbnxRJftFWxYGb0ZJTVdhsIqrDq05zD45Fgh6+jX4TuVvrIv5SMY/Mq7JCEFuKpVqkPrV2rSCjPk2rutmVp9Ygz6jp9FqCTocAwbQhXO4PKwshovb2iD52LCntr9XswVn5U+g99XQ0ej+Tl8K6EEEYHhWcdEFT8=
2021-11-17T03:21:53.956 INFO:teuthology.kill:  smithi162.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSnViabuJwQ2gkeg/NqHscY2infcLRxLc4dHN6qXaDI1xc2xTaUrAVh9Gk7gJi9mjViMdwMDMIHE0RGcAL/ngApLmbwI7Tziodq4D71rDNkgrxyrIf3HxUo2B0raGdBsNIJj/8u0lY8/Bt9MjOY7FhJecsEoFFFo+hwYQl0usODMp7Rch+jLqWsFEFsevIpYf4KYH63y6b6elkI+BlhlEDXnVgPSCH+IcfxPFKjAHYKCHatm8wF9o6nDkve6punSA7lT6WOUwVOYmk4zre5T94Fxe7R6ImKBYPwz2zsCFPcZI2KdxVnVmnu7+aBzlhp/TWefhWNjMKlkIWih+lFWjwZOYWkaVFTtXQHq5yAw8hoUJxbgMoH8PUOXkADVOUM0h5P8k+jPIIO0IdxmEeo5eeFGXlGJt7WluYYFkCgZz46WJowq+LgwwXoyLAVltf/ops/PuYdfuHhkvV5wTmflM0U3uSwpdQRbd3EPxV6gxVFCINl9K7qgFHPno5/KyIirE=
2021-11-17T03:21:53.980 INFO:teuthology.kill:2021-11-17 03:21:53,980.980 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-17T03:21:53.983 INFO:teuthology.kill:2021-11-17 03:21:53,983.983 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-17T03:21:54.056 INFO:teuthology.kill:2021-11-17 03:21:54,056.056 INFO:teuthology.orchestra.console:Power off smithi162
2021-11-17T03:21:54.069 INFO:teuthology.kill:2021-11-17 03:21:54,069.069 INFO:teuthology.orchestra.console:Power off smithi146
2021-11-17T03:22:02.434 INFO:teuthology.kill:2021-11-17 03:22:02,433.433 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2021-11-17T03:22:02.578 INFO:teuthology.kill:2021-11-17 03:22:02,578.578 INFO:teuthology.lock.ops:unlocked: smithi146.front.sepia.ceph.com
2021-11-17T03:22:06.542 INFO:teuthology.kill:2021-11-17 03:22:06,542.542 INFO:teuthology.orchestra.console:Power off for smithi162 completed
2021-11-17T03:22:06.687 INFO:teuthology.kill:2021-11-17 03:22:06,687.687 INFO:teuthology.lock.ops:unlocked: smithi162.front.sepia.ceph.com
2021-11-17T03:24:07.016 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-17T03:24:07.128 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi146 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6507831
2021-11-17T03:24:07.129 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi162 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-16_20:03:49-rados-wip-yuri6-testing-2021-11-16-0926-pacific-distro-basic-smithi/6507830