2022-02-18T14:30:35.773 INFO:root:teuthology version: 1.1.0-eea25212 2022-02-18T14:30:35.896 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi181.front.sepia.ceph.com' reimaging 2022-02-18T14:30:35.897 INFO:teuthology.lock.ops:Updating [ubuntu@smithi181.front.sepia.ceph.com]: reset os type and version on server 2022-02-18T14:30:35.897 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2022-02-18T14:30:35.916 INFO:teuthology.lock.ops:Node 'ubuntu@smithi181.front.sepia.ceph.com' reimaging is complete 2022-02-18T14:30:35.917 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi204.front.sepia.ceph.com' reimaging 2022-02-18T14:30:35.917 INFO:teuthology.lock.ops:Updating [ubuntu@smithi204.front.sepia.ceph.com]: reset os type and version on server 2022-02-18T14:30:35.917 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2022-02-18T14:30:35.939 INFO:teuthology.lock.ops:Node 'ubuntu@smithi204.front.sepia.ceph.com' reimaging is complete 2022-02-18T14:30:36.129 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of ubuntu 20.04 2022-02-18T14:30:36.138 INFO:teuthology.provision.fog.smithi204:Scheduling deploy of ubuntu 20.04 2022-02-18T14:30:36.428 INFO:teuthology.orchestra.console:Power off smithi181 2022-02-18T14:30:36.443 INFO:teuthology.orchestra.console:Power off smithi204 2022-02-18T14:30:36.572 INFO:teuthology.orchestra.console:Power off for smithi181 completed 2022-02-18T14:30:36.584 INFO:teuthology.orchestra.console:Power off for smithi204 completed 2022-02-18T14:30:36.674 INFO:teuthology.orchestra.console:Power on smithi181 2022-02-18T14:30:36.686 INFO:teuthology.orchestra.console:Power on smithi204 2022-02-18T14:30:45.052 INFO:teuthology.orchestra.console:Power on for smithi204 completed 2022-02-18T14:30:45.108 INFO:teuthology.orchestra.console:Power on for smithi181 completed 2022-02-18T14:30:45.154 INFO:teuthology.provision.fog.smithi204:Waiting for deploy to finish 2022-02-18T14:30:45.209 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish 2022-02-18T14:33:46.540 ERROR:teuthology.orchestra.connection:Error authenticating with smithi204.front.sepia.ceph.com: Authentication failed. 2022-02-18T14:35:43.434 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-18T14:35:43.950 INFO:teuthology.provision.fog.smithi181:Node is ready 2022-02-18T14:35:43.957 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com 2022-02-18T14:35:43.964 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181 2022-02-18T14:35:44.187 INFO:teuthology.provision.fog.smithi181:Deploy complete! 2022-02-18T14:35:59.722 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-18T14:36:00.135 INFO:teuthology.provision.fog.smithi204:Node is ready 2022-02-18T14:36:00.142 INFO:teuthology.orchestra.run.smithi204.stdout:smithi204.front.sepia.ceph.com 2022-02-18T14:36:00.149 INFO:teuthology.orchestra.run.smithi204.stdout:172.21.15.204 smithi204.front.sepia.ceph.com smithi204 2022-02-18T14:36:00.318 INFO:teuthology.provision.fog.smithi204:Deploy complete! 2022-02-18T14:36:00.543 INFO:teuthology.lock.ops:Checking smithi181.front.sepia.ceph.com 2022-02-18T14:36:00.544 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-18T14:36:00.564 INFO:teuthology.lock.ops:Checking smithi204.front.sepia.ceph.com 2022-02-18T14:36:00.565 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-18T14:36:00.582 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: set os type and version on server 2022-02-18T14:36:00.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2022-02-18T14:36:00.758 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2022-02-18T14:36:00.765 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Ubuntu" 2022-02-18T14:36:00.766 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-18T14:36:00.766 INFO:teuthology.orchestra.run.smithi181.stdout:ID=ubuntu 2022-02-18T14:36:00.766 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE=debian 2022-02-18T14:36:00.766 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-18T14:36:00.767 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="20.04" 2022-02-18T14:36:00.767 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-18T14:36:00.767 INFO:teuthology.orchestra.run.smithi181.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-18T14:36:00.767 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-18T14:36:00.768 INFO:teuthology.orchestra.run.smithi181.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-18T14:36:00.768 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_CODENAME=focal 2022-02-18T14:36:00.768 INFO:teuthology.orchestra.run.smithi181.stdout:UBUNTU_CODENAME=focal 2022-02-18T14:36:00.769 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2022-02-18T14:36:00.788 INFO:teuthology.lock.ops:Updating [smithi204.front.sepia.ceph.com]: set os type and version on server 2022-02-18T14:36:00.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2022-02-18T14:36:00.962 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2022-02-18T14:36:01.012 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2022-02-18T14:36:01.013 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-02-18T14:36:01.013 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2022-02-18T14:36:01.013 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2022-02-18T14:36:01.013 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-02-18T14:36:01.014 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="20.04" 2022-02-18T14:36:01.014 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2022-02-18T14:36:01.014 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-02-18T14:36:01.014 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-02-18T14:36:01.014 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-02-18T14:36:01.015 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=focal 2022-02-18T14:36:01.015 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=focal 2022-02-18T14:36:01.015 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2022-02-18T14:36:01.115 INFO:teuthology.dispatcher.supervisor:Running job 6694194 2022-02-18T14:36:01.116 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eea2521245e542c7c1a063d296779f572aa0255a/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-02-18_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6694194 --name teuthology-2022-02-18_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi --description upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} -- /home/teuthworker/archive/teuthology-2022-02-18_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6694194/orig.config.yaml 2022-02-18T14:36:01.119 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-02-18_14:22:03-upgrade:octopus-x-quincy-distro-default-smithi/6694194 2022-02-18T14:36:01.120 INFO:teuthology.dispatcher.supervisor:Job PID: 25588 2022-02-18T14:36:01.120 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-18T21:06:11.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-18T21:06:11.814 INFO:teuthology.kill:Killing Pids: {25588} 2022-02-18T21:06:12.930 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2022-02-18T21:06:12.931 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2022-02-18T21:06:12.931 INFO:teuthology.misc:Compressing logs... 2022-02-18T21:06:12.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2022-02-18T21:06:13.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2022-02-18T21:06:13.503 INFO:teuthology.misc:Compressing logs... 2022-02-18T21:06:13.785 INFO:teuthology.misc:Compressing logs... 2022-02-18T21:06:17.559 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-mgr.y.log: file size changed while zipping 2022-02-18T21:06:58.795 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-02-18T21:07:17.075 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-02-18T21:07:21.300 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-mon.c.log: file size changed while zipping 2022-02-18T21:07:34.513 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-mon.a.log: file size changed while zipping 2022-02-18T21:07:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-02-18T21:07:44.288 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-mon.b.log: file size changed while zipping 2022-02-18T21:08:23.425 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-02-18T21:08:35.391 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-02-18T21:09:17.797 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-02-18T21:09:27.227 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-02-18T21:10:16.582 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/3fc215e6-90c9-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-02-18T21:11:55.402 INFO:teuthology.kill:No teuthology processes running 2022-02-18T21:11:56.461 INFO:teuthology.kill:Nuking machines: ['smithi181', 'smithi204'] 2022-02-18T21:11:57.420 INFO:teuthology.kill:2022-02-18 21:11:57,420.420 INFO:teuthology.nuke:targets: 2022-02-18T21:11:57.421 INFO:teuthology.kill: smithi181.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZwqFHa0YCHv0DLRMPT6L8lWHMLVNoHA7q3om+N9yBXbwhnLgeReF8phA8nAZIGqy58Nx+4D1MbBlEzpBFPUuqD756qoMRxLG9KSvtkOQPW8ZgR10S3guPZ7tkxoQERYcx9dJb7Um+G8OOUzv0/eEPDw+HXbT6QsaRulxKOInEdwkKmCMYQaWjbd+P+744qZLa6bfrnpkJiixbdML89/w3XMug/B6rb31dOFSvhluR0hWMNg+bo6Ip4WCZgpoHgBiCLqC7Hy2fgodKAyvcgI/rpswnlgYj8ci800bGRhGJEs3UVYhdDFl/rXs6HB56tRor3MvAoMhT153QfbbPhlb6pItJjj6X4pZaAythNyY6SBN4B4TsOVYke7jC5cihEktJUyngwzNqyTqCmBQzJZRgp26XcWDDjo5uBU+yTFfjlLt5IrCxz6TSOj4+nkSF18+Kz9kaU72SxBHneSVQLinjRZYLjAomLV2otToK1d41inYtJ8yG7yX9rVU70Swv9pc= 2022-02-18T21:11:57.421 INFO:teuthology.kill: smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi0EXgMyK9K8rhSJrv+Ex+rfmN9ufnY2F9naSPTIYwl2kB3vqNXQ+NVbItau0byHNw5JnW2COLXthUn+ZXH3ll0l6pbDZQT4iAX5qpqU8GmWaRkYYw4PbmGh4+fTwY2gQe/aqcl7Da8b/JHhKLJ8OmKBhQXVSthmVyC9wGPcCi9Ed2OhEBsdDSpG2ZdyYLjemuLxz8AXFMqn1fbP/ItPt3/WX+iiJUKYhT5KmFpdMcixh7L9sHea+vQOgHyHnEU5hMbIuSATI96DUZfyAi/vVZsCLedxc7FjMBZjrHnGQMhHzVA3Z0jP/KNMhFI5NWQ4k/EZXhyHm5OHi28DrzLwKYpmYg7iMRGRj8cxPzJa79isUMfho6iOSDlB+v+j7SIfL3Wh1s9fCMPSu6Oa4edw0DgBMO92ZuFPwpSnmz8MZLLQBkF6FxTS3ExdMts/9/KHkTlmzavsDoVEIpPLPKAwTgFJNVrkI2uDLCmqG+WuFGabtOetMSh8vyH/3ykx/N6j8= 2022-02-18T21:11:57.446 INFO:teuthology.kill:2022-02-18 21:11:57,446.446 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-18T21:11:57.449 INFO:teuthology.kill:2022-02-18 21:11:57,449.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-18T21:11:57.524 INFO:teuthology.kill:2022-02-18 21:11:57,523.523 INFO:teuthology.orchestra.console:Power off smithi204 2022-02-18T21:11:57.537 INFO:teuthology.kill:2022-02-18 21:11:57,537.537 INFO:teuthology.orchestra.console:Power off smithi181 2022-02-18T21:12:10.010 INFO:teuthology.kill:2022-02-18 21:12:10,009.009 INFO:teuthology.orchestra.console:Power off for smithi204 completed 2022-02-18T21:12:10.016 INFO:teuthology.kill:2022-02-18 21:12:10,016.016 INFO:teuthology.orchestra.console:Power off for smithi181 completed 2022-02-18T21:12:10.152 INFO:teuthology.kill:2022-02-18 21:12:10,152.152 INFO:teuthology.lock.ops:unlocked: smithi204.front.sepia.ceph.com 2022-02-18T21:12:10.156 INFO:teuthology.kill:2022-02-18 21:12:10,156.156 INFO:teuthology.lock.ops:unlocked: smithi181.front.sepia.ceph.com 2022-02-18T21:14:10.359 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-18T21:14:10.523 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi181 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-18_20:48:16-upgrade-master-distro-default-smithi/6694427 2022-02-18T21:14:10.524 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi204 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-18_20:48:16-upgrade-master-distro-default-smithi/6694427