2021-10-29T11:07:23.574 INFO:root:teuthology version: 1.1.0-c56135d1
2021-10-29T11:07:23.703 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging
2021-10-29T11:07:23.705 INFO:teuthology.lock.ops:Updating [ubuntu@smithi159.front.sepia.ceph.com]: reset os type and version on server
2021-10-29T11:07:23.705 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server
2021-10-29T11:07:23.732 INFO:teuthology.lock.ops:Node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging is complete
2021-10-29T11:07:23.732 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging
2021-10-29T11:07:23.733 INFO:teuthology.lock.ops:Updating [ubuntu@smithi164.front.sepia.ceph.com]: reset os type and version on server
2021-10-29T11:07:23.733 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2021-10-29T11:07:23.752 INFO:teuthology.lock.ops:Node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging is complete
2021-10-29T11:07:23.941 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of centos 8.3
2021-10-29T11:07:23.949 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of centos 8.3
2021-10-29T11:07:24.267 INFO:teuthology.orchestra.console:Power off smithi159
2021-10-29T11:07:24.307 INFO:teuthology.orchestra.console:Power off smithi164
2021-10-29T11:07:32.665 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2021-10-29T11:07:32.766 INFO:teuthology.orchestra.console:Power on smithi164
2021-10-29T11:07:36.755 INFO:teuthology.orchestra.console:Power off for smithi159 completed
2021-10-29T11:07:36.856 INFO:teuthology.orchestra.console:Power on smithi159
2021-10-29T11:07:41.111 INFO:teuthology.orchestra.console:Power on for smithi159 completed
2021-10-29T11:07:41.176 INFO:teuthology.orchestra.console:Power on for smithi164 completed
2021-10-29T11:07:41.213 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish
2021-10-29T11:07:41.277 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish
2021-10-29T11:12:08.414 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-29T11:12:24.154 INFO:teuthology.provision.fog.smithi159:Node is ready
2021-10-29T11:12:24.172 INFO:teuthology.orchestra.run.smithi159.stdout:smithi133
2021-10-29T11:12:24.228 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.15.133 smithi133.front.sepia.ceph.com smithi133
2021-10-29T11:12:24.573 INFO:teuthology.provision.fog.smithi159:Deploy complete!
2021-10-29T11:12:45.925 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-10-29T11:12:46.238 INFO:teuthology.provision.fog.smithi164:Node is ready
2021-10-29T11:12:46.293 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com
2021-10-29T11:12:46.348 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164
2021-10-29T11:12:46.668 INFO:teuthology.provision.fog.smithi164:Deploy complete!
2021-10-29T11:12:46.849 INFO:teuthology.lock.ops:Checking smithi159.front.sepia.ceph.com
2021-10-29T11:12:46.850 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-29T11:12:46.872 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com
2021-10-29T11:12:46.872 INFO:teuthology.lock.ops:New key found. Updating...
2021-10-29T11:12:46.891 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: set os type and version on server
2021-10-29T11:12:46.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2021-10-29T11:12:47.160 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64
2021-10-29T11:12:47.214 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Linux"
2021-10-29T11:12:47.214 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8"
2021-10-29T11:12:47.215 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos"
2021-10-29T11:12:47.215 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora"
2021-10-29T11:12:47.215 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8"
2021-10-29T11:12:47.215 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8"
2021-10-29T11:12:47.215 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Linux 8"
2021-10-29T11:12:47.216 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31"
2021-10-29T11:12:47.216 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-10-29T11:12:47.216 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/"
2021-10-29T11:12:47.216 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-10-29T11:12:47.217 INFO:teuthology.orchestra.run.smithi159.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-10-29T11:12:47.217 INFO:teuthology.orchestra.run.smithi159.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-10-29T11:12:47.218 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server
2021-10-29T11:12:47.237 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server
2021-10-29T11:12:47.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2021-10-29T11:12:47.440 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64
2021-10-29T11:12:47.494 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Linux"
2021-10-29T11:12:47.495 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8"
2021-10-29T11:12:47.495 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos"
2021-10-29T11:12:47.495 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora"
2021-10-29T11:12:47.495 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8"
2021-10-29T11:12:47.496 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8"
2021-10-29T11:12:47.496 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Linux 8"
2021-10-29T11:12:47.496 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31"
2021-10-29T11:12:47.496 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-10-29T11:12:47.496 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/"
2021-10-29T11:12:47.497 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-10-29T11:12:47.497 INFO:teuthology.orchestra.run.smithi164.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-10-29T11:12:47.497 INFO:teuthology.orchestra.run.smithi164.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-10-29T11:12:47.498 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2021-10-29T11:12:47.612 INFO:teuthology.dispatcher.supervisor:Running job 6466144
2021-10-29T11:12:47.612 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-10-28_23:09:03-rados-wip-yuri2-testing-2021-10-28-1343-distro-basic-smithi/6466144 --name yuriw-2021-10-28_23:09:03-rados-wip-yuri2-testing-2021-10-28-1343-distro-basic-smithi --description rados/cephadm/smoke/{0-nvme-loop distro/centos_8.3_container_tools_3.0 fixed-2 mon_election/classic start} -- /home/teuthworker/archive/yuriw-2021-10-28_23:09:03-rados-wip-yuri2-testing-2021-10-28-1343-distro-basic-smithi/6466144/orig.config.yaml
2021-10-29T11:12:47.616 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-10-28_23:09:03-rados-wip-yuri2-testing-2021-10-28-1343-distro-basic-smithi/6466144
2021-10-29T11:12:47.617 INFO:teuthology.dispatcher.supervisor:Job PID: 15910
2021-10-29T11:12:47.617 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-10-29T23:13:41.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-10-29T23:13:41.314 INFO:teuthology.kill:Killing Pids: {15910}
2021-10-29T23:13:42.416 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a']
2021-10-29T23:13:42.417 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a']
2021-10-29T23:13:42.417 INFO:teuthology.misc:Compressing logs...
2021-10-29T23:13:42.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2021-10-29T23:13:42.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2021-10-29T23:13:42.667 INFO:teuthology.orchestra.run.smithi159.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-10-29T23:13:42.872 INFO:teuthology.orchestra.run.smithi164.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-10-29T23:13:42.902 INFO:teuthology.orchestra.run.smithi159.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T23:13:42.902 INFO:teuthology.orchestra.run.smithi159.stderr:tar: Error is not recoverable: exiting now
2021-10-29T23:13:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-10-29T23:13:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:tar: Error is not recoverable: exiting now
2021-10-29T23:13:42.931 INFO:teuthology.misc:Compressing logs...
2021-10-29T23:13:43.099 INFO:teuthology.misc:Compressing logs...
2021-10-29T23:13:43.124 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-10-29T23:13:43.137 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-10-29T23:13:43.181 INFO:teuthology.kill:No teuthology processes running
2021-10-29T23:13:44.259 INFO:teuthology.kill:Nuking machines: ['smithi159', 'smithi164']
2021-10-29T23:13:45.254 INFO:teuthology.kill:2021-10-29 23:13:45,254.254 INFO:teuthology.nuke:targets:
2021-10-29T23:13:45.255 INFO:teuthology.kill:  smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkgMVZYp8yz6HGG5QfSt1peZA/qC/idWh+cmfaK7DKAhgvApAoa9eawdKPwvFJXghEo76BgSJhG36OwI719285ylLlYjzNz2uEgiy4vNctcfiwK7MsjhKfxiEXsZmQ59nmctaWojrMGmOfeGyDdUKuEtCD7yFEuKVEIk1P18wsxjAUnU//Nhc2UfhiJBlMnPj2eME/H+hpyViyQXTgy3vYb61BX5WAGCtRpITkbRVfvHL8t+4nZPwRD/0xWtySgFAkHIgz/wzpNjofC5GYt1GrsiZGWWGEEPLY51kCxgPihQ3PuxoqK64vtmmIVsHypBsCKMyGQ8K1v3NFsWD0z9B9v5AooUYoCPD08Hc7VBWVQmTzKM4g5zR+wSFXs8Jh0D+0iuVrEuwbgJgMqhIXWFvDJXF9sb8l5FwCuTaQhzg+RUKCm5cs8QuMq/8AbzaODVvJKqDzmZDFf1oycdVHpwelg6sPUYnrpCVyKggs6uUgiQnEyDz0dxMI1k2zbNGqUfc=
2021-10-29T23:13:45.255 INFO:teuthology.kill:  smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKQpc99hultil/KFAeJcvDf3i2nRuHQHLsLqHRzYlDE+m+fG6a42rncF6H2IjdjMvHOcBztoiZI+ydCiibJR2s0TzwqrSP/jjIvlisxpuCQIu7kybiFSkF2nYTSTtEwpE/+ZLvpwDnt/2vnz0daaAU2UNC++WRZnTv5ThKFTNFQtO42hfbJDiA6UIL6CReacgTTEdokKd3eGY7kge/i6m1IXTHGJxUQMDA8zOGRAQggkGhpg7W6fWor/TmJuNEyirJ663BjrbJmEWJrgt5TzuedBig1jXf9LCcN0aqxQ+qOh32PrdWnjEVXU0PpXRizS9eDaqxLszllgcWakaAaSvC0xiRHj+L6MMGk3LxPSGxsi495GSeT+3WotBOGytv0qLJpixbmNui2RI1l2JeOyc/snlNN6yPkg7COlaDFUcpdcRxD+KhiCZ6NXebauNx53wfXQIpmcrVI4ejQ1ytHnJdVKw6mGE4PwgPdMIMQAMwj4QQ1dRH8o7PWDi1++6f09s=
2021-10-29T23:13:45.281 INFO:teuthology.kill:2021-10-29 23:13:45,280.280 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T23:13:45.283 INFO:teuthology.kill:2021-10-29 23:13:45,283.283 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-10-29T23:13:45.357 INFO:teuthology.kill:2021-10-29 23:13:45,357.357 INFO:teuthology.orchestra.console:Power off smithi159
2021-10-29T23:13:45.370 INFO:teuthology.kill:2021-10-29 23:13:45,370.370 INFO:teuthology.orchestra.console:Power off smithi164
2021-10-29T23:13:57.841 INFO:teuthology.kill:2021-10-29 23:13:57,841.841 INFO:teuthology.orchestra.console:Power off for smithi159 completed
2021-10-29T23:13:57.847 INFO:teuthology.kill:2021-10-29 23:13:57,846.846 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2021-10-29T23:13:57.994 INFO:teuthology.kill:2021-10-29 23:13:57,994.994 INFO:teuthology.lock.ops:unlocked: smithi159.front.sepia.ceph.com
2021-10-29T23:13:57.996 INFO:teuthology.kill:2021-10-29 23:13:57,996.996 INFO:teuthology.lock.ops:unlocked: smithi164.front.sepia.ceph.com
2021-10-29T23:15:58.304 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-10-29T23:15:58.393 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi159 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467551
2021-10-29T23:15:58.394 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi164 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-10-29_17:42:41-rados-wip-yuri7-testing-2021-10-28-1307-distro-basic-smithi/6467551