2023-04-08T02:44:05.247 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-08T02:44:05.380 INFO:teuthology.lock.ops:Start node 'smithi084.front.sepia.ceph.com' reimaging 2023-04-08T02:44:05.381 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: reset os type and version on server 2023-04-08T02:44:05.381 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2023-04-08T02:44:05.410 INFO:teuthology.lock.ops:Node 'smithi084.front.sepia.ceph.com' reimaging is complete 2023-04-08T02:44:05.411 INFO:teuthology.lock.ops:Start node 'smithi148.front.sepia.ceph.com' reimaging 2023-04-08T02:44:05.411 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: reset os type and version on server 2023-04-08T02:44:05.411 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-04-08T02:44:05.435 INFO:teuthology.lock.ops:Node 'smithi148.front.sepia.ceph.com' reimaging is complete 2023-04-08T02:44:05.567 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of ubuntu 20.04 2023-04-08T02:44:05.585 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of ubuntu 20.04 2023-04-08T02:44:05.779 INFO:teuthology.orchestra.console:Power off smithi148 2023-04-08T02:44:05.798 INFO:teuthology.orchestra.console:Power off smithi084 2023-04-08T02:44:14.162 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2023-04-08T02:44:14.263 INFO:teuthology.orchestra.console:Power on smithi084 2023-04-08T02:44:18.266 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2023-04-08T02:44:18.368 INFO:teuthology.orchestra.console:Power on smithi148 2023-04-08T02:44:18.516 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2023-04-08T02:44:18.617 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2023-04-08T02:44:22.622 INFO:teuthology.orchestra.console:Power on for smithi148 completed 2023-04-08T02:44:22.723 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2023-04-08T02:48:23.049 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-04-08T02:48:27.141 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-04-08T02:48:32.137 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-04-08T02:48:36.201 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-04-08T02:48:41.221 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-04-08T02:48:45.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-04-08T02:48:50.277 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.84 2023-04-08T02:48:54.309 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-04-08T02:49:03.365 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-04-08T02:49:12.421 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-04-08T02:49:56.311 WARNING:teuthology.provision.fog:timed out 2023-04-08T02:50:02.537 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-04-08T02:50:03.025 INFO:teuthology.provision.fog.smithi084:Node is ready 2023-04-08T02:50:03.084 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2023-04-08T02:50:03.091 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2023-04-08T02:50:03.311 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2023-04-08T02:50:18.445 WARNING:teuthology.provision.fog:timed out 2023-04-08T02:50:24.671 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-04-08T02:50:25.148 INFO:teuthology.provision.fog.smithi148:Node is ready 2023-04-08T02:50:25.155 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com 2023-04-08T02:50:25.161 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148 2023-04-08T02:50:25.364 INFO:teuthology.provision.fog.smithi148:Deploy complete! 2023-04-08T02:50:25.635 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2023-04-08T02:50:25.635 INFO:teuthology.lock.ops:New key found. Updating... 2023-04-08T02:50:25.660 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com 2023-04-08T02:50:25.660 INFO:teuthology.lock.ops:New key found. Updating... 2023-04-08T02:50:25.686 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2023-04-08T02:50:25.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-04-08T02:50:25.941 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2023-04-08T02:50:25.948 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Ubuntu" 2023-04-08T02:50:25.948 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-04-08T02:50:25.948 INFO:teuthology.orchestra.run.smithi084.stdout:ID=ubuntu 2023-04-08T02:50:25.948 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE=debian 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="20.04" 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.ubuntu.com/" 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_CODENAME=focal 2023-04-08T02:50:25.949 INFO:teuthology.orchestra.run.smithi084.stdout:UBUNTU_CODENAME=focal 2023-04-08T02:50:25.950 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2023-04-08T02:50:25.976 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server 2023-04-08T02:50:25.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-04-08T02:50:26.199 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2023-04-08T02:50:26.206 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Ubuntu" 2023-04-08T02:50:26.206 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-04-08T02:50:26.206 INFO:teuthology.orchestra.run.smithi148.stdout:ID=ubuntu 2023-04-08T02:50:26.206 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE=debian 2023-04-08T02:50:26.206 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-04-08T02:50:26.206 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="20.04" 2023-04-08T02:50:26.207 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.ubuntu.com/" 2023-04-08T02:50:26.207 INFO:teuthology.orchestra.run.smithi148.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-04-08T02:50:26.207 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-04-08T02:50:26.207 INFO:teuthology.orchestra.run.smithi148.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-04-08T02:50:26.207 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_CODENAME=focal 2023-04-08T02:50:26.207 INFO:teuthology.orchestra.run.smithi148.stdout:UBUNTU_CODENAME=focal 2023-04-08T02:50:26.208 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-04-08T02:50:26.413 INFO:teuthology.dispatcher.supervisor:Running job 7235184 2023-04-08T02:50:26.414 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/bin/teuthology -v --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2023-04-07_15:39:07-rgw-wip-cbodley-testing-distro-default-smithi/7235184 --name cbodley-2023-04-07_15:39:07-rgw-wip-cbodley-testing-distro-default-smithi --description rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides} -- /home/teuthworker/archive/cbodley-2023-04-07_15:39:07-rgw-wip-cbodley-testing-distro-default-smithi/7235184/orig.config.yaml 2023-04-08T02:50:26.418 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/cbodley-2023-04-07_15:39:07-rgw-wip-cbodley-testing-distro-default-smithi/7235184 2023-04-08T02:50:26.419 INFO:teuthology.dispatcher.supervisor:Job PID: 31404 2023-04-08T02:50:26.419 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-04-08T14:51:16.982 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-04-08T14:51:17.149 INFO:teuthology.kill:Killing Pids: {31404} 2023-04-08T14:51:18.008 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-04-08T14:51:18.009 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-04-08T14:51:18.009 INFO:teuthology.misc:Compressing logs... 2023-04-08T14:51:18.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2023-04-08T14:51:18.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-04-08T14:51:18.577 INFO:teuthology.misc:Compressing logs... 2023-04-08T14:51:18.593 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2023-04-08T14:51:18.594 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2023-04-08T14:51:18.594 INFO:teuthology.orchestra.run.smithi084.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 76.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-04-08T14:51:18.606 INFO:teuthology.orchestra.run.smithi084.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 88.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-04-08T14:51:18.637 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2023-04-08T14:51:18.638 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2023-04-08T14:51:18.638 INFO:teuthology.orchestra.run.smithi148.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 76.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-04-08T14:51:18.650 INFO:teuthology.orchestra.run.smithi148.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 88.3% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-04-08T14:51:18.692 INFO:teuthology.misc:Compressing logs... 2023-04-08T14:51:18.728 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34181.log 2023-04-08T14:51:18.729 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33750.log 2023-04-08T14:51:18.730 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34181.log.gz 2023-04-08T14:51:18.730 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-04-08T14:51:18.730 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33954.log 2023-04-08T14:51:18.731 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33750.log.gz 2023-04-08T14:51:18.731 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33965.log 2023-04-08T14:51:18.731 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33964.log 2023-04-08T14:51:18.732 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33954.log.gz 2023-04-08T14:51:18.732 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2023-04-08T14:51:18.733 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33753.log 2023-04-08T14:51:18.733 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33965.log.gz 2023-04-08T14:51:18.733 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33964.log.gz 2023-04-08T14:51:18.733 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34183.log 2023-04-08T14:51:18.733 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2023-04-08T14:51:18.734 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33753.log.gz 2023-04-08T14:51:18.735 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-04-08T14:51:18.735 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34373.log 2023-04-08T14:51:18.735 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34183.log.gz 2023-04-08T14:51:18.735 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-04-08T14:51:18.735 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.34373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34178.log 2023-04-08T14:51:18.735 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34373.log.gz 2023-04-08T14:51:18.737 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-04-08T14:51:18.737 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34178.log: gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2023-04-08T14:51:18.737 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34178.log.gz 2023-04-08T14:51:18.737 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34531.log 2023-04-08T14:51:18.738 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/rgw.ceph.client.0.log: 93.0% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2023-04-08T14:51:18.738 INFO:teuthology.orchestra.run.smithi084.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-04-08T14:51:18.738 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33644.log 2023-04-08T14:51:18.739 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33698.log 2023-04-08T14:51:18.739 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34531.log.gz 2023-04-08T14:51:18.739 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33952.log 2023-04-08T14:51:18.739 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33644.log.gz 2023-04-08T14:51:18.740 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33698.log: /var/log/ceph/ceph-client.admin.33959.log 2023-04-08T14:51:18.740 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33698.log.gz 2023-04-08T14:51:18.741 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33617.log 2023-04-08T14:51:18.741 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33952.log.gz 2023-04-08T14:51:18.741 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33564.log 2023-04-08T14:51:18.741 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33959.log.gz 2023-04-08T14:51:18.742 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34427.log 2023-04-08T14:51:18.742 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33617.log.gz 2023-04-08T14:51:18.742 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 /var/log/ceph/ceph-client.admin.33564.log: --verbose -- /var/log/ceph/ceph-client.admin.34504.log 2023-04-08T14:51:18.742 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33564.log.gz 2023-04-08T14:51:18.742 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34427.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.25168.log 2023-04-08T14:51:18.743 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34427.log.gz 2023-04-08T14:51:18.744 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34452.log 2023-04-08T14:51:18.744 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34504.log.gz 2023-04-08T14:51:18.744 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-04-08T14:51:18.744 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.tmp-client.admin.25168.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.25168.log.gz 2023-04-08T14:51:18.744 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33751.log 2023-04-08T14:51:18.744 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34452.log.gz 2023-04-08T14:51:18.744 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34185.log 2023-04-08T14:51:18.745 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2023-04-08T14:51:18.746 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33751.log: /var/log/ceph/ceph-client.admin.34185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33751.log.gz 2023-04-08T14:51:18.746 INFO:teuthology.orchestra.run.smithi084.stderr:gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.34185.log.gz--verbose 2023-04-08T14:51:18.746 INFO:teuthology.orchestra.run.smithi084.stderr:-- /var/log/ceph/ceph-client.admin.33749.log 2023-04-08T14:51:18.746 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34182.log 2023-04-08T14:51:18.746 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz 2023-04-08T14:51:18.746 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33749.log.gz 2023-04-08T14:51:18.747 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30231.log 2023-04-08T14:51:18.748 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33678.log 2023-04-08T14:51:18.749 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-04-08T14:51:18.749 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.30231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30231.log.gz 2023-04-08T14:51:18.752 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.33678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30124.log 2023-04-08T14:51:18.752 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33678.log.gz 2023-04-08T14:51:18.752 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30150.log 2023-04-08T14:51:18.752 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.30124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30124.log.gz 2023-04-08T14:51:18.752 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-04-08T14:51:18.753 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.30150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30150.log.gz 2023-04-08T14:51:18.753 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-04-08T14:51:18.753 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33747.log 2023-04-08T14:51:18.754 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2023-04-08T14:51:18.754 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34182.log: /var/log/ceph/ceph-client.admin.33747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34182.log.gz 2023-04-08T14:51:18.754 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33671.log 2023-04-08T14:51:18.754 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33747.log.gz 2023-04-08T14:51:18.754 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34180.log 2023-04-08T14:51:18.754 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.30316.log: /var/log/ceph/ceph-client.admin.33671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33953.log 2023-04-08T14:51:18.754 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33671.log.gz 0.0% 2023-04-08T14:51:18.755 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2023-04-08T14:51:18.755 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30019.log 2023-04-08T14:51:18.755 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34180.log.gz 2023-04-08T14:51:18.755 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33953.log.gz 2023-04-08T14:51:18.755 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-04-08T14:51:18.755 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34179.log 2023-04-08T14:51:18.758 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.30019.log: /var/log/ceph/ceph.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30019.log.gz 2023-04-08T14:51:18.758 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-04-08T14:51:18.758 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30186.log 2023-04-08T14:51:18.758 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34179.log.gz 2023-04-08T14:51:18.759 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33744.log 2023-04-08T14:51:18.759 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.30186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30186.log.gz 2023-04-08T14:51:18.759 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33746.log 2023-04-08T14:51:18.759 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33951.log 2023-04-08T14:51:18.759 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33744.log: 0.0%/var/log/ceph/ceph-client.admin.33746.log: -- replaced with /var/log/ceph/ceph-client.admin.33744.log.gz 2023-04-08T14:51:18.759 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33746.log.gz 2023-04-08T14:51:18.761 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-04-08T14:51:18.762 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30047.log 2023-04-08T14:51:18.762 INFO:teuthology.orchestra.run.smithi148.stderr:gzip /var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-04-08T14:51:18.762 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.30047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30047.log.gz 2023-04-08T14:51:18.762 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.1.log 2023-04-08T14:51:18.762 INFO:teuthology.orchestra.run.smithi148.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-04-08T14:51:18.762 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30073.log 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/rgw.ceph.client.1.log: 93.0% -- replaced with /var/log/ceph/rgw.ceph.client.1.log.gz 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30175.log 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.30073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33653.log 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30073.log.gz 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.30175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33703.log 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30175.log.gz 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.33653.log: -- /var/log/ceph/ceph-client.admin.30098.log 2023-04-08T14:51:18.763 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33653.log.gz 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.33703.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33703.log.gz 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr:gzip /var/log/ceph/ceph-client.admin.30098.log: -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30098.log.gz 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.log: gzip /var/log/ceph/ceph-osd.4.log: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.33728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2023-04-08T14:51:18.764 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30203.log 2023-04-08T14:51:18.766 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34479.log 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33951.log.gz 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33969.log 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34479.log.gz 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.33969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34400.log 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33969.log.gz 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-04-08T14:51:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34400.log.gz 2023-04-08T14:51:18.768 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33590.log 2023-04-08T14:51:18.768 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-client.admin.30203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30203.log.gz 2023-04-08T14:51:18.778 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.33590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33590.log.gz 2023-04-08T14:51:18.811 INFO:teuthology.orchestra.run.smithi148.stderr: 91.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-04-08T14:51:18.832 INFO:teuthology.orchestra.run.smithi084.stderr: 95.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-04-08T14:51:19.158 INFO:teuthology.orchestra.run.smithi084.stderr: 95.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-04-08T14:51:21.770 INFO:teuthology.orchestra.run.smithi148.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-04-08T14:51:27.799 INFO:teuthology.orchestra.run.smithi148.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-04-08T14:51:28.527 INFO:teuthology.orchestra.run.smithi084.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-04-08T14:51:35.923 INFO:teuthology.orchestra.run.smithi084.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-04-08T14:51:36.958 INFO:teuthology.orchestra.run.smithi148.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-04-08T14:51:37.234 INFO:teuthology.orchestra.run.smithi084.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-04-08T14:51:37.351 INFO:teuthology.orchestra.run.smithi084.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-04-08T14:51:38.170 INFO:teuthology.orchestra.run.smithi148.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-04-08T14:51:39.353 INFO:teuthology.orchestra.run.smithi148.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-04-08T14:51:41.177 INFO:teuthology.orchestra.run.smithi148.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-04-08T14:51:44.077 INFO:teuthology.orchestra.run.smithi084.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-04-08T14:51:46.720 INFO:teuthology.orchestra.run.smithi084.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-04-08T14:52:09.198 INFO:teuthology.kill:No teuthology processes running 2023-04-08T14:52:10.013 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi148'] 2023-04-08T14:52:10.605 INFO:teuthology.kill:2023-04-08 14:52:10,604.604 INFO:teuthology.nuke:targets: 2023-04-08T14:52:10.606 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= 2023-04-08T14:52:10.606 INFO:teuthology.kill: smithi148.front.sepia.ceph.com: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw 2023-04-08T14:52:10.606 INFO:teuthology.kill:2023-04-08 14:52:10,605.605 INFO:teuthology.nuke:Checking targets against current locks 2023-04-08T14:52:10.662 INFO:teuthology.kill:2023-04-08 14:52:10,662.662 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-04-08T14:52:10.684 INFO:teuthology.kill:2023-04-08 14:52:10,683.683 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-04-08T14:52:10.733 INFO:teuthology.kill:2023-04-08 14:52:10,733.733 INFO:teuthology.orchestra.console:Power off smithi084 2023-04-08T14:52:10.773 INFO:teuthology.kill:2023-04-08 14:52:10,772.772 INFO:teuthology.orchestra.console:Power off smithi148 2023-04-08T14:52:23.221 INFO:teuthology.kill:2023-04-08 14:52:23,221.221 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2023-04-08T14:52:23.245 INFO:teuthology.kill:2023-04-08 14:52:23,244.244 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2023-04-08T14:52:23.379 INFO:teuthology.kill:2023-04-08 14:52:23,378.378 INFO:teuthology.lock.ops:unlocked: smithi084.front.sepia.ceph.com 2023-04-08T14:52:23.394 INFO:teuthology.kill:2023-04-08 14:52:23,394.394 INFO:teuthology.lock.ops:unlocked: smithi148.front.sepia.ceph.com 2023-04-08T14:54:23.711 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-04-08T14:54:23.800 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi084 but it was locked by another job: /home/teuthworker/archive/teuthology-2023-04-08_14:23:09-upgrade:pacific-x-reef-distro-default-smithi/7235437 2023-04-08T14:54:23.800 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi148 but it was locked by another job: /home/teuthworker/archive/teuthology-2023-04-08_14:23:09-upgrade:quincy-x-reef-distro-default-smithi/7235438