2022-03-05T03:25:01.242 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-05T03:25:01.365 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging
2022-03-05T03:25:01.366 INFO:teuthology.lock.ops:Updating [ubuntu@smithi039.front.sepia.ceph.com]: reset os type and version on server
2022-03-05T03:25:01.367 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server
2022-03-05T03:25:01.385 INFO:teuthology.lock.ops:Node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging is complete
2022-03-05T03:25:01.385 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi125.front.sepia.ceph.com' reimaging
2022-03-05T03:25:01.386 INFO:teuthology.lock.ops:Updating [ubuntu@smithi125.front.sepia.ceph.com]: reset os type and version on server
2022-03-05T03:25:01.386 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server
2022-03-05T03:25:01.408 INFO:teuthology.lock.ops:Node 'ubuntu@smithi125.front.sepia.ceph.com' reimaging is complete
2022-03-05T03:25:01.608 INFO:teuthology.provision.fog.smithi039:Scheduling deploy of centos 8.stream
2022-03-05T03:25:01.631 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of centos 8.stream
2022-03-05T03:25:02.059 INFO:teuthology.orchestra.console:Power off smithi125
2022-03-05T03:25:02.080 INFO:teuthology.orchestra.console:Power off smithi039
2022-03-05T03:25:14.549 INFO:teuthology.orchestra.console:Power off for smithi125 completed
2022-03-05T03:25:14.554 INFO:teuthology.orchestra.console:Power off for smithi039 completed
2022-03-05T03:25:14.651 INFO:teuthology.orchestra.console:Power on smithi125
2022-03-05T03:25:14.659 INFO:teuthology.orchestra.console:Power on smithi039
2022-03-05T03:25:23.019 INFO:teuthology.orchestra.console:Power on for smithi125 completed
2022-03-05T03:25:23.026 INFO:teuthology.orchestra.console:Power on for smithi039 completed
2022-03-05T03:25:23.121 INFO:teuthology.provision.fog.smithi125:Waiting for deploy to finish
2022-03-05T03:25:23.128 INFO:teuthology.provision.fog.smithi039:Waiting for deploy to finish
2022-03-05T03:29:22.368 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-05T03:29:47.339 INFO:teuthology.provision.fog.smithi125:Node is ready
2022-03-05T03:29:47.362 INFO:teuthology.orchestra.run.smithi125.stdout:smithi125.front.sepia.ceph.com
2022-03-05T03:29:47.422 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125 smithi125.front.sepia.ceph.com smithi125
2022-03-05T03:29:47.824 INFO:teuthology.provision.fog.smithi125:Deploy complete!
2022-03-05T03:30:05.982 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-05T03:30:26.311 INFO:teuthology.provision.fog.smithi039:Node is ready
2022-03-05T03:30:26.334 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039.front.sepia.ceph.com
2022-03-05T03:30:26.394 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39 smithi039.front.sepia.ceph.com smithi039
2022-03-05T03:30:26.772 INFO:teuthology.provision.fog.smithi039:Deploy complete!
2022-03-05T03:30:26.921 INFO:teuthology.lock.ops:Checking smithi039.front.sepia.ceph.com
2022-03-05T03:30:26.922 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-05T03:30:26.942 INFO:teuthology.lock.ops:Checking smithi125.front.sepia.ceph.com
2022-03-05T03:30:26.942 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-05T03:30:26.962 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: set os type and version on server
2022-03-05T03:30:26.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com'
2022-03-05T03:30:27.162 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64
2022-03-05T03:30:27.221 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream"
2022-03-05T03:30:27.221 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8"
2022-03-05T03:30:27.221 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos"
2022-03-05T03:30:27.221 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora"
2022-03-05T03:30:27.222 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8"
2022-03-05T03:30:27.222 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8"
2022-03-05T03:30:27.222 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-05T03:30:27.222 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31"
2022-03-05T03:30:27.223 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-05T03:30:27.223 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/"
2022-03-05T03:30:27.223 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-05T03:30:27.223 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-05T03:30:27.224 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-05T03:30:27.225 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server
2022-03-05T03:30:27.245 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: set os type and version on server
2022-03-05T03:30:27.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com'
2022-03-05T03:30:27.489 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64
2022-03-05T03:30:27.546 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="CentOS Stream"
2022-03-05T03:30:27.546 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="8"
2022-03-05T03:30:27.546 INFO:teuthology.orchestra.run.smithi125.stdout:ID="centos"
2022-03-05T03:30:27.547 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE="rhel fedora"
2022-03-05T03:30:27.547 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="8"
2022-03-05T03:30:27.547 INFO:teuthology.orchestra.run.smithi125.stdout:PLATFORM_ID="platform:el8"
2022-03-05T03:30:27.547 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-05T03:30:27.548 INFO:teuthology.orchestra.run.smithi125.stdout:ANSI_COLOR="0;31"
2022-03-05T03:30:27.548 INFO:teuthology.orchestra.run.smithi125.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-05T03:30:27.548 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://centos.org/"
2022-03-05T03:30:27.548 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-05T03:30:27.548 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-05T03:30:27.549 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-05T03:30:27.549 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server
2022-03-05T03:30:27.648 INFO:teuthology.dispatcher.supervisor:Running job 6721637
2022-03-05T03:30:27.648 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721637 --name yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721637/orig.config.yaml
2022-03-05T03:30:27.651 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721637
2022-03-05T03:30:27.652 INFO:teuthology.dispatcher.supervisor:Job PID: 17404
2022-03-05T03:30:27.652 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-05T10:00:57.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-05T10:00:57.454 INFO:teuthology.kill:Killing Pids: {17404}
2022-03-05T10:00:58.615 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2022-03-05T10:00:58.616 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2022-03-05T10:00:58.616 INFO:teuthology.misc:Compressing logs...
2022-03-05T10:00:58.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com'
2022-03-05T10:00:58.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com'
2022-03-05T10:00:59.070 INFO:teuthology.misc:Compressing logs...
2022-03-05T10:00:59.669 INFO:teuthology.orchestra.run.smithi125.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-03-05T10:00:59.673 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-03-05T10:01:00.041 INFO:teuthology.misc:Compressing logs...
2022-03-05T10:01:00.071 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/ceph-mon.a.log.gz already exists;	not overwritten
2022-03-05T10:01:00.101 INFO:teuthology.orchestra.run.smithi125.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists;	not overwritten
2022-03-05T10:01:13.457 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/dispatcher/supervisor.py", line 287, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/dispatcher/supervisor.py", line 345, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2022-03-05T10:01:13.550 INFO:teuthology.kill:No teuthology processes running
2022-03-05T10:01:14.717 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi125']
2022-03-05T10:01:15.687 INFO:teuthology.kill:2022-03-05 10:01:15,687.687 INFO:teuthology.nuke:targets:
2022-03-05T10:01:15.688 INFO:teuthology.kill:  smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPN6nM9eikrbndXZla/bt2QaRe/kxJO+bj+FcEE/uSQBwJovzXJtXPqAPI2rQeAH3oTcQuuwMh/L9rXuShMQhwPHbMYNkqn3VT8c5LuAnP0S7ei7+p1T+otcfxu3Db1lt0RdU/Rybm4wq5CiBkZ7x63QtrPvC+IHFlC3fmDuJlnGO/lL06FM7X+V2Klca2E6jxm4WSxZ1BL9VdPodDK3Nqs6jI+13wSN4jsYcLmMTvKw/zRNBrhS0EapKnmP421uykpOUFwpdlZquKaWsEmHWoHU4RrjMqVSOL/EpcDGO458MoV8ixos4EUIFjmZE9Z1rmT0Ge5UlDv47jl++d4d6MFEI1fT75Fp7dTlyuQJ9mwgKC8mcyhAzG1BZhY+lnNOfC3F11Fwg2BuCqAg6FcdpGW7n5QSQqMLMefDsG3LOHEFQdceftPWsRqxX8t5DCouMSmOnl5KQXewHS3dCuZ4ROqDhBdJRPf4/zsEsEvm5iCmp0wMMMnAY3193cUVFxamM=
2022-03-05T10:01:15.688 INFO:teuthology.kill:  smithi125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNkW4LnTYN+fRKUVZwpCnOKnS+2hOkvHh9rT1nzhO/T4CZac7UN07rSMELAOaLljPAK6QySlaN2AuF8EPjStPBDnAdAGhwpaZtITQ3E9x1lZJcO6N5YV9sKL5+oK7BgRt/i2VioysG5aY0LA53nDMJnF7XDNhzkyNItsg4JSHUXs65wANbei8yedsgnd1ho/5M2E6nn8eeY8UIEPOpnDrIirQBxNlwzY++oJkVOLlL9zfsxUofgWib+8vOZmnDmTUkPwln+Is3LQ4SbFS+/SqWjsDZffLDRDilEfSDHUEX/35MplLr64BfPwjIUDgI7b35trqon3AdpcOaTJHY1URUiqvxwrk0wiH9UYp2z5LNAcY4Y47sXJHRMWHx+fMi+cNrWQAgLMve3MksfWiiYoI6kbIetvy20c5rHsDOAtqEDV//RH/LdZIGDlLFOFsr2IvxnXtGGujZdL4r4IwEBrH5Bri4dKrReAoLBySwLgEje32Ojj981wQqPNRxCDSaDo8=
2022-03-05T10:01:15.716 INFO:teuthology.kill:2022-03-05 10:01:15,716.716 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-05T10:01:15.719 INFO:teuthology.kill:2022-03-05 10:01:15,719.719 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-05T10:01:15.797 INFO:teuthology.kill:2022-03-05 10:01:15,796.796 INFO:teuthology.orchestra.console:Power off smithi125
2022-03-05T10:01:15.810 INFO:teuthology.kill:2022-03-05 10:01:15,809.809 INFO:teuthology.orchestra.console:Power off smithi039
2022-03-05T10:01:24.174 INFO:teuthology.kill:2022-03-05 10:01:24,174.174 INFO:teuthology.orchestra.console:Power off for smithi039 completed
2022-03-05T10:01:24.321 INFO:teuthology.kill:2022-03-05 10:01:24,320.320 INFO:teuthology.lock.ops:unlocked: smithi039.front.sepia.ceph.com
2022-03-05T10:01:28.282 INFO:teuthology.kill:2022-03-05 10:01:28,281.281 INFO:teuthology.orchestra.console:Power off for smithi125 completed
2022-03-05T10:01:28.425 INFO:teuthology.kill:2022-03-05 10:01:28,425.425 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com
2022-03-05T10:03:28.737 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-05T10:03:28.839 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi039 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-04_21:59:11-rados-master-distro-default-smithi/6721522
2022-03-05T10:03:28.839 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi125 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-04_21:59:11-rados-master-distro-default-smithi/6721522