2023-01-14T14:36:33.296 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2023-01-14T14:36:33.415 INFO:teuthology.lock.ops:Start node 'smithi103.front.sepia.ceph.com' reimaging
2023-01-14T14:36:33.416 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T14:36:33.416 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-01-14T14:36:33.432 INFO:teuthology.lock.ops:Node 'smithi103.front.sepia.ceph.com' reimaging is complete
2023-01-14T14:36:33.432 INFO:teuthology.lock.ops:Start node 'smithi169.front.sepia.ceph.com' reimaging
2023-01-14T14:36:33.432 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T14:36:33.433 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server
2023-01-14T14:36:33.449 INFO:teuthology.lock.ops:Node 'smithi169.front.sepia.ceph.com' reimaging is complete
2023-01-14T14:36:33.592 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of ubuntu 22.04
2023-01-14T14:36:33.594 INFO:teuthology.provision.fog.smithi169:Scheduling deploy of ubuntu 22.04
2023-01-14T14:36:33.824 INFO:teuthology.orchestra.console:Power off smithi169
2023-01-14T14:36:33.839 INFO:teuthology.orchestra.console:Power off smithi103
2023-01-14T14:36:33.969 INFO:teuthology.orchestra.console:Power off for smithi169 completed
2023-01-14T14:36:33.981 INFO:teuthology.orchestra.console:Power off for smithi103 completed
2023-01-14T14:36:34.070 INFO:teuthology.orchestra.console:Power on smithi169
2023-01-14T14:36:34.081 INFO:teuthology.orchestra.console:Power on smithi103
2023-01-14T14:36:38.333 INFO:teuthology.orchestra.console:Power on for smithi169 completed
2023-01-14T14:36:38.434 INFO:teuthology.provision.fog.smithi169:Waiting for deploy to finish
2023-01-14T14:36:42.454 INFO:teuthology.orchestra.console:Power on for smithi103 completed
2023-01-14T14:36:42.555 INFO:teuthology.provision.fog.smithi103:Waiting for deploy to finish
2023-01-14T14:39:58.872 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-01-14T14:40:28.108 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169
2023-01-14T14:40:34.508 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:40:37.164 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169
2023-01-14T14:40:43.564 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:40:46.220 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169
2023-01-14T14:40:52.620 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:40:55.276 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169
2023-01-14T14:41:01.676 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:41:04.332 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169
2023-01-14T14:41:10.732 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:41:48.240 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:41:54.244 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:42:00.246 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:42:06.249 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:42:10.335 WARNING:teuthology.provision.fog:timed out
2023-01-14T14:42:12.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-01-14T14:42:16.536 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T14:42:16.962 INFO:teuthology.provision.fog.smithi169:Node is ready
2023-01-14T14:42:17.008 INFO:teuthology.orchestra.run.smithi169.stdout:smithi169.front.sepia.ceph.com
2023-01-14T14:42:17.061 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169 smithi169.front.sepia.ceph.com smithi169
2023-01-14T14:42:17.384 INFO:teuthology.provision.fog.smithi169:Deploy complete!
2023-01-14T14:42:18.474 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T14:42:19.119 INFO:teuthology.provision.fog.smithi103:Node is ready
2023-01-14T14:42:19.165 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103.front.sepia.ceph.com
2023-01-14T14:42:19.214 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103 smithi103.front.sepia.ceph.com smithi103
2023-01-14T14:42:19.524 INFO:teuthology.provision.fog.smithi103:Deploy complete!
2023-01-14T14:42:19.696 INFO:teuthology.lock.ops:Checking smithi103.front.sepia.ceph.com
2023-01-14T14:42:19.697 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-14T14:42:19.713 INFO:teuthology.lock.ops:Checking smithi169.front.sepia.ceph.com
2023-01-14T14:42:19.714 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-14T14:42:19.729 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: set os type and version on server
2023-01-14T14:42:19.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-01-14T14:42:19.957 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64
2023-01-14T14:42:20.005 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-01-14T14:42:20.005 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="Ubuntu"
2023-01-14T14:42:20.005 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="22.04"
2023-01-14T14:42:20.006 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-01-14T14:42:20.006 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_CODENAME=jammy
2023-01-14T14:42:20.006 INFO:teuthology.orchestra.run.smithi103.stdout:ID=ubuntu
2023-01-14T14:42:20.006 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE=debian
2023-01-14T14:42:20.007 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://www.ubuntu.com/"
2023-01-14T14:42:20.007 INFO:teuthology.orchestra.run.smithi103.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-01-14T14:42:20.007 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-01-14T14:42:20.007 INFO:teuthology.orchestra.run.smithi103.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-01-14T14:42:20.007 INFO:teuthology.orchestra.run.smithi103.stdout:UBUNTU_CODENAME=jammy
2023-01-14T14:42:20.008 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-01-14T14:42:20.023 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: set os type and version on server
2023-01-14T14:42:20.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com'
2023-01-14T14:42:20.264 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64
2023-01-14T14:42:20.312 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-01-14T14:42:20.313 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="Ubuntu"
2023-01-14T14:42:20.313 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="22.04"
2023-01-14T14:42:20.313 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-01-14T14:42:20.313 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_CODENAME=jammy
2023-01-14T14:42:20.314 INFO:teuthology.orchestra.run.smithi169.stdout:ID=ubuntu
2023-01-14T14:42:20.314 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE=debian
2023-01-14T14:42:20.314 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://www.ubuntu.com/"
2023-01-14T14:42:20.314 INFO:teuthology.orchestra.run.smithi169.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-01-14T14:42:20.315 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-01-14T14:42:20.315 INFO:teuthology.orchestra.run.smithi169.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-01-14T14:42:20.315 INFO:teuthology.orchestra.run.smithi169.stdout:UBUNTU_CODENAME=jammy
2023-01-14T14:42:20.315 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server
2023-01-14T14:42:20.411 INFO:teuthology.dispatcher.supervisor:Running job 7142106
2023-01-14T14:42:20.411 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_aemerson@teuthology --archive /home/teuthworker/archive/aemerson-2023-01-13_23:20:40-rgw-wip-rgw-multisite-reshard-reef-distro-default-smithi/7142106 --name aemerson-2023-01-13_23:20:40-rgw-wip-rgw-multisite-reshard-reef-distro-default-smithi --description rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} -- /home/teuthworker/archive/aemerson-2023-01-13_23:20:40-rgw-wip-rgw-multisite-reshard-reef-distro-default-smithi/7142106/orig.config.yaml
2023-01-14T14:42:20.415 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/aemerson-2023-01-13_23:20:40-rgw-wip-rgw-multisite-reshard-reef-distro-default-smithi/7142106
2023-01-14T14:42:20.416 INFO:teuthology.dispatcher.supervisor:Job PID: 12288
2023-01-14T14:42:20.416 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-01-15T02:43:42.551 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-15T02:43:42.690 INFO:teuthology.kill:Killing Pids: {12288}
2023-01-15T02:43:43.355 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2023-01-15T02:43:43.356 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2023-01-15T02:43:43.356 INFO:teuthology.misc:Compressing logs...
2023-01-15T02:43:43.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-01-15T02:43:43.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com'
2023-01-15T02:43:43.911 INFO:teuthology.misc:Compressing logs...
2023-01-15T02:43:44.050 INFO:teuthology.misc:Compressing logs...
2023-01-15T02:43:48.294 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping
2023-01-15T02:44:06.462 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping
2023-01-15T02:44:11.913 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping
2023-01-15T02:44:25.557 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping
2023-01-15T02:44:29.746 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping
2023-01-15T02:44:48.308 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping
2023-01-15T02:44:51.035 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping
2023-01-15T02:44:56.716 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping
2023-01-15T02:45:16.011 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping
2023-01-15T02:45:19.603 INFO:teuthology.orchestra.run.smithi169.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping
2023-01-15T02:45:38.183 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping
2023-01-15T02:45:48.213 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping