2021-11-24T00:37:56.549 INFO:root:teuthology version: 1.1.0-6cfddd5e
2021-11-24T00:37:56.690 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi045.front.sepia.ceph.com' reimaging
2021-11-24T00:37:56.691 INFO:teuthology.lock.ops:Updating [ubuntu@smithi045.front.sepia.ceph.com]: reset os type and version on server
2021-11-24T00:37:56.691 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2021-11-24T00:37:56.710 INFO:teuthology.lock.ops:Node 'ubuntu@smithi045.front.sepia.ceph.com' reimaging is complete
2021-11-24T00:37:56.710 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi077.front.sepia.ceph.com' reimaging
2021-11-24T00:37:56.711 INFO:teuthology.lock.ops:Updating [ubuntu@smithi077.front.sepia.ceph.com]: reset os type and version on server
2021-11-24T00:37:56.711 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server
2021-11-24T00:37:56.730 INFO:teuthology.lock.ops:Node 'ubuntu@smithi077.front.sepia.ceph.com' reimaging is complete
2021-11-24T00:37:56.927 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of centos 8.3
2021-11-24T00:37:56.933 INFO:teuthology.provision.fog.smithi077:Scheduling deploy of centos 8.3
2021-11-24T00:37:57.230 INFO:teuthology.orchestra.console:Power off smithi045
2021-11-24T00:37:57.247 INFO:teuthology.orchestra.console:Power off smithi077
2021-11-24T00:37:57.380 INFO:teuthology.orchestra.console:Power off for smithi045 completed
2021-11-24T00:37:57.481 INFO:teuthology.orchestra.console:Power on smithi045
2021-11-24T00:38:01.733 INFO:teuthology.orchestra.console:Power on for smithi045 completed
2021-11-24T00:38:01.835 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish
2021-11-24T00:38:05.617 INFO:teuthology.orchestra.console:Power off for smithi077 completed
2021-11-24T00:38:05.719 INFO:teuthology.orchestra.console:Power on smithi077
2021-11-24T00:38:14.196 INFO:teuthology.orchestra.console:Power on for smithi077 completed
2021-11-24T00:38:14.297 INFO:teuthology.provision.fog.smithi077:Waiting for deploy to finish
2021-11-24T00:42:36.049 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-24T00:42:36.325 INFO:teuthology.provision.fog.smithi045:Node is ready
2021-11-24T00:42:36.341 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045.front.sepia.ceph.com
2021-11-24T00:42:36.400 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45 smithi045.front.sepia.ceph.com smithi045
2021-11-24T00:42:36.733 INFO:teuthology.provision.fog.smithi045:Deploy complete!
2021-11-24T00:43:03.662 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-24T00:43:03.967 INFO:teuthology.provision.fog.smithi077:Node is ready
2021-11-24T00:43:03.982 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077.front.sepia.ceph.com
2021-11-24T00:43:04.039 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77 smithi077.front.sepia.ceph.com smithi077
2021-11-24T00:43:04.415 INFO:teuthology.provision.fog.smithi077:Deploy complete!
2021-11-24T00:43:04.713 INFO:teuthology.lock.ops:Checking smithi045.front.sepia.ceph.com
2021-11-24T00:43:04.714 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-24T00:43:04.739 INFO:teuthology.lock.ops:Checking smithi077.front.sepia.ceph.com
2021-11-24T00:43:04.739 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-24T00:43:04.771 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: set os type and version on server
2021-11-24T00:43:04.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2021-11-24T00:43:05.052 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64
2021-11-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Linux"
2021-11-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8"
2021-11-24T00:43:05.108 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos"
2021-11-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora"
2021-11-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8"
2021-11-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8"
2021-11-24T00:43:05.109 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-24T00:43:05.110 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31"
2021-11-24T00:43:05.111 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-24T00:43:05.111 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/"
2021-11-24T00:43:05.111 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-24T00:43:05.111 INFO:teuthology.orchestra.run.smithi045.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-24T00:43:05.112 INFO:teuthology.orchestra.run.smithi045.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-24T00:43:05.113 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2021-11-24T00:43:05.134 INFO:teuthology.lock.ops:Updating [smithi077.front.sepia.ceph.com]: set os type and version on server
2021-11-24T00:43:05.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com'
2021-11-24T00:43:05.338 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64
2021-11-24T00:43:05.396 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Linux"
2021-11-24T00:43:05.396 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8"
2021-11-24T00:43:05.397 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos"
2021-11-24T00:43:05.397 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora"
2021-11-24T00:43:05.397 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8"
2021-11-24T00:43:05.398 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8"
2021-11-24T00:43:05.398 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-24T00:43:05.398 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31"
2021-11-24T00:43:05.399 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-24T00:43:05.399 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/"
2021-11-24T00:43:05.399 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-24T00:43:05.400 INFO:teuthology.orchestra.run.smithi077.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-24T00:43:05.402 INFO:teuthology.orchestra.run.smithi077.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-24T00:43:05.403 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server
2021-11-24T00:43:05.510 INFO:teuthology.dispatcher.supervisor:Running job 6522012
2021-11-24T00:43:05.510 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology -v --owner scheduled_ifed01@teuthology --archive /home/teuthworker/archive/ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi/6522012 --name ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi/6522012/orig.config.yaml
2021-11-24T00:43:05.514 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ifed01-2021-11-23_11:51:25-rados-wip-ifed-better-daemonperf-distro-default-smithi/6522012
2021-11-24T00:43:05.515 INFO:teuthology.dispatcher.supervisor:Job PID: 18166
2021-11-24T00:43:05.515 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-24T12:44:02.174 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-24T12:44:02.386 INFO:teuthology.kill:Killing Pids: {18166}
2021-11-24T12:44:03.640 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-24T12:44:03.641 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-24T12:44:03.642 INFO:teuthology.misc:Compressing logs...
2021-11-24T12:44:03.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2021-11-24T12:44:03.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com'
2021-11-24T12:44:04.224 INFO:teuthology.misc:Compressing logs...
2021-11-24T12:44:04.255 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-24T12:44:04.287 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2021-11-24T12:44:04.323 INFO:teuthology.orchestra.run.smithi045.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-24T12:44:04.323 INFO:teuthology.orchestra.run.smithi045.stderr:tar: Error is not recoverable: exiting now
2021-11-24T12:44:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2021-11-24T12:44:04.353 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now
2021-11-24T12:44:04.354 INFO:teuthology.misc:Compressing logs...
2021-11-24T12:44:04.391 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2021-11-24T12:44:04.396 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 242, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/dispatcher/supervisor.py", line 300, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2021-11-24T12:44:04.468 INFO:teuthology.kill:No teuthology processes running
2021-11-24T12:44:05.753 INFO:teuthology.kill:Nuking machines: ['smithi045', 'smithi077']
2021-11-24T12:44:06.804 INFO:teuthology.kill:2021-11-24 12:44:06,803.803 INFO:teuthology.nuke:targets:
2021-11-24T12:44:06.805 INFO:teuthology.kill:  smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC776suYKtGF0Txze33ez8N+/IQWalC9hqDl9A+nn+Pjo8z4fVWX6zAElzaSR4e8GYw5y6F1Majz0t1hdBttyqKtbF0yUMonJx1R7eKZTDUYPEuu08yk2HeQ+C04z+wuXFUi1+XEO+HVZ7pI3Ye/bXjp7szDwGL565vJ6xo5Rv5UdyMX+1TdKVClvVDsVDmZu6y7vjZcJpl53OAfuKPpgxsJRxRvcrgmRiPs5tDmhtoh/77QHXPASJ0IKUulVAqIRno0mtHDZ2/+3agzLFc3br3+dpv43I08Mv1tQ++Tqaouex6qTKIHuBM4HHvydwsXkw/O9Rtbsu++8Ozr2/JD4SumnQVs4NENGoE4koimiz3lhfkh+hIQ4EkwpqpUMqeIH1K/JVWSS4OLbfkYHMA7MzxVoNWGF6M9heR9ABOKiZJBm8kE33eMLOnLw6WhtEdZ41MHfVXylBYfM59lT87BKw8BvJUD99sge7us+XL2NIIr50YgnzXBl4jAEQKowx0xKM=
2021-11-24T12:44:06.806 INFO:teuthology.kill:  smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkTnSAzLHek9cI6t6t/m7bZstfDaWcQuc72zoDW2pgEslel/7pucvIE9f9qqsMrhQi4J9iHEEo29BJVnc8PwDaDwo600cXo0uMIUkY7CQkKckm66afRnEpIFuh6vMuts1l6/RDskOnPA3x/jIQyoG4GNoAEkc3WD1LVPzIGzMKIFvt9vgmm0GBuT9rDvJwZziSRXtvSQ36Nl8FumsLaCqIhZWyB72W3BIPRjPHyfeEZMtUqjIj2rO3lpX1FwsFEed+9cXbu4GPaJgPyof5GHElhkYGM/mMhv2Jad23SD+hw0hJyLfLrnCor2iD4Dd35Mjcvuk5H1bIlBBRo3feYCZmR5u0+1J6nk9ZVu+/gugFg5W21dz/EsT+slGomtiWPa1UoiNzPDMr34CfVidKnVaXejVVvwzSv2inyI5wP+DBebTKxC4aSEgp8wXvYZ5Nc5QQ1Je0HxLayYqZisMVO1SDlmatm3iYmO7A6rzb2c6XhNOOJ31Lw69IA5jgatQIEiM=
2021-11-24T12:44:06.831 INFO:teuthology.kill:2021-11-24 12:44:06,830.830 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-24T12:44:06.838 INFO:teuthology.kill:2021-11-24 12:44:06,837.837 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-24T12:44:06.915 INFO:teuthology.kill:2021-11-24 12:44:06,914.914 INFO:teuthology.orchestra.console:Power off smithi077
2021-11-24T12:44:06.928 INFO:teuthology.kill:2021-11-24 12:44:06,928.928 INFO:teuthology.orchestra.console:Power off smithi045
2021-11-24T12:44:15.301 INFO:teuthology.kill:2021-11-24 12:44:15,301.301 INFO:teuthology.orchestra.console:Power off for smithi045 completed
2021-11-24T12:44:15.460 INFO:teuthology.kill:2021-11-24 12:44:15,459.459 INFO:teuthology.lock.ops:unlocked: smithi045.front.sepia.ceph.com
2021-11-24T12:44:19.400 INFO:teuthology.kill:2021-11-24 12:44:19,399.399 INFO:teuthology.orchestra.console:Power off for smithi077 completed
2021-11-24T12:44:19.545 INFO:teuthology.kill:2021-11-24 12:44:19,545.545 INFO:teuthology.lock.ops:unlocked: smithi077.front.sepia.ceph.com
2021-11-24T12:46:19.891 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-24T12:46:20.013 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi045 but it was locked by another job: /home/teuthworker/archive/vshankar-2021-11-24_07:14:27-fs-wip-vshankar-testing-20211124-094330-testing-default-smithi/6524018
2021-11-24T12:46:20.013 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi077 but it was locked by another job: /home/teuthworker/archive/vshankar-2021-11-24_07:14:27-fs-wip-vshankar-testing-20211124-094330-testing-default-smithi/6524018