2021-11-01T21:25:22.040 INFO:root:teuthology version: 1.1.0-c56135d1
2021-11-01T21:25:22.177 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi036.front.sepia.ceph.com' reimaging
2021-11-01T21:25:22.178 INFO:teuthology.lock.ops:Updating [ubuntu@smithi036.front.sepia.ceph.com]: reset os type and version on server
2021-11-01T21:25:22.179 INFO:teuthology.lock.ops:Updating smithi036.front.sepia.ceph.com on lock server
2021-11-01T21:25:22.198 INFO:teuthology.lock.ops:Node 'ubuntu@smithi036.front.sepia.ceph.com' reimaging is complete
2021-11-01T21:25:22.198 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi123.front.sepia.ceph.com' reimaging
2021-11-01T21:25:22.199 INFO:teuthology.lock.ops:Updating [ubuntu@smithi123.front.sepia.ceph.com]: reset os type and version on server
2021-11-01T21:25:22.199 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2021-11-01T21:25:22.217 INFO:teuthology.lock.ops:Node 'ubuntu@smithi123.front.sepia.ceph.com' reimaging is complete
2021-11-01T21:25:22.416 INFO:teuthology.provision.fog.smithi036:Scheduling deploy of centos 8.3
2021-11-01T21:25:22.424 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of centos 8.3
2021-11-01T21:25:22.791 INFO:teuthology.orchestra.console:Power off smithi036
2021-11-01T21:25:22.829 INFO:teuthology.orchestra.console:Power off smithi123
2021-11-01T21:25:31.189 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2021-11-01T21:25:31.290 INFO:teuthology.orchestra.console:Power on smithi123
2021-11-01T21:25:35.275 INFO:teuthology.orchestra.console:Power off for smithi036 completed
2021-11-01T21:25:35.376 INFO:teuthology.orchestra.console:Power on smithi036
2021-11-01T21:25:39.658 INFO:teuthology.orchestra.console:Power on for smithi123 completed
2021-11-01T21:25:39.760 INFO:teuthology.provision.fog.smithi123:Waiting for deploy to finish
2021-11-01T21:25:43.746 INFO:teuthology.orchestra.console:Power on for smithi036 completed
2021-11-01T21:25:43.848 INFO:teuthology.provision.fog.smithi036:Waiting for deploy to finish
2021-11-01T21:28:15.049 ERROR:teuthology.orchestra.connection:Error authenticating with smithi036.front.sepia.ceph.com: Authentication failed.
2021-11-01T21:30:29.432 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-01T21:30:29.757 INFO:teuthology.provision.fog.smithi123:Node is ready
2021-11-01T21:30:29.774 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123.front.sepia.ceph.com
2021-11-01T21:30:29.832 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123 smithi123.front.sepia.ceph.com smithi123
2021-11-01T21:30:30.160 INFO:teuthology.provision.fog.smithi123:Deploy complete!
2021-11-01T21:30:46.864 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-01T21:30:47.169 INFO:teuthology.provision.fog.smithi036:Node is ready
2021-11-01T21:30:47.185 INFO:teuthology.orchestra.run.smithi036.stdout:smithi036.front.sepia.ceph.com
2021-11-01T21:30:47.242 INFO:teuthology.orchestra.run.smithi036.stdout:172.21.15.36 smithi036.front.sepia.ceph.com smithi036
2021-11-01T21:30:47.575 INFO:teuthology.provision.fog.smithi036:Deploy complete!
2021-11-01T21:30:47.742 INFO:teuthology.lock.ops:Checking smithi036.front.sepia.ceph.com
2021-11-01T21:30:47.743 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-01T21:30:47.763 INFO:teuthology.lock.ops:Checking smithi123.front.sepia.ceph.com
2021-11-01T21:30:47.763 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-01T21:30:47.783 INFO:teuthology.lock.ops:Updating [smithi036.front.sepia.ceph.com]: set os type and version on server
2021-11-01T21:30:47.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi036.front.sepia.ceph.com'
2021-11-01T21:30:47.986 INFO:teuthology.orchestra.run.smithi036.stdout:x86_64
2021-11-01T21:30:48.042 INFO:teuthology.orchestra.run.smithi036.stdout:NAME="CentOS Linux"
2021-11-01T21:30:48.042 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION="8"
2021-11-01T21:30:48.043 INFO:teuthology.orchestra.run.smithi036.stdout:ID="centos"
2021-11-01T21:30:48.043 INFO:teuthology.orchestra.run.smithi036.stdout:ID_LIKE="rhel fedora"
2021-11-01T21:30:48.044 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION_ID="8"
2021-11-01T21:30:48.044 INFO:teuthology.orchestra.run.smithi036.stdout:PLATFORM_ID="platform:el8"
2021-11-01T21:30:48.044 INFO:teuthology.orchestra.run.smithi036.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-01T21:30:48.045 INFO:teuthology.orchestra.run.smithi036.stdout:ANSI_COLOR="0;31"
2021-11-01T21:30:48.046 INFO:teuthology.orchestra.run.smithi036.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-01T21:30:48.046 INFO:teuthology.orchestra.run.smithi036.stdout:HOME_URL="https://centos.org/"
2021-11-01T21:30:48.047 INFO:teuthology.orchestra.run.smithi036.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-01T21:30:48.047 INFO:teuthology.orchestra.run.smithi036.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-01T21:30:48.047 INFO:teuthology.orchestra.run.smithi036.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-01T21:30:48.049 INFO:teuthology.lock.ops:Updating smithi036.front.sepia.ceph.com on lock server
2021-11-01T21:30:48.071 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: set os type and version on server
2021-11-01T21:30:48.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com'
2021-11-01T21:30:48.346 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64
2021-11-01T21:30:48.402 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Linux"
2021-11-01T21:30:48.402 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8"
2021-11-01T21:30:48.402 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos"
2021-11-01T21:30:48.402 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora"
2021-11-01T21:30:48.403 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8"
2021-11-01T21:30:48.403 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8"
2021-11-01T21:30:48.403 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-01T21:30:48.403 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31"
2021-11-01T21:30:48.404 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-01T21:30:48.404 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/"
2021-11-01T21:30:48.404 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-01T21:30:48.404 INFO:teuthology.orchestra.run.smithi123.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-01T21:30:48.405 INFO:teuthology.orchestra.run.smithi123.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-01T21:30:48.405 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2021-11-01T21:30:48.542 INFO:teuthology.dispatcher.supervisor:Running job 6473898
2021-11-01T21:30:48.543 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-11-01_21:14:30-orch:cephadm-wip-sage2-testing-2021-11-01-1529-distro-basic-smithi/6473898 --name sage-2021-11-01_21:14:30-orch:cephadm-wip-sage2-testing-2021-11-01-1529-distro-basic-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/sage-2021-11-01_21:14:30-orch:cephadm-wip-sage2-testing-2021-11-01-1529-distro-basic-smithi/6473898/orig.config.yaml
2021-11-01T21:30:48.546 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-11-01_21:14:30-orch:cephadm-wip-sage2-testing-2021-11-01-1529-distro-basic-smithi/6473898
2021-11-01T21:30:48.547 INFO:teuthology.dispatcher.supervisor:Job PID: 11305
2021-11-01T21:30:48.547 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-02T09:31:09.891 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-02T09:31:10.122 INFO:teuthology.kill:Killing Pids: {11305}
2021-11-02T09:31:11.273 INFO:teuthology.task.internal:roles: ubuntu@smithi036.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-02T09:31:11.274 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-02T09:31:11.274 INFO:teuthology.misc:Compressing logs...
2021-11-02T09:31:11.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi036.front.sepia.ceph.com'
2021-11-02T09:31:11.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com'
2021-11-02T09:31:11.764 INFO:teuthology.misc:Compressing logs...
2021-11-02T09:31:12.112 INFO:teuthology.misc:Compressing logs...
2021-11-02T09:31:28.710 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-02T09:31:32.628 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi123.bsudvl.log: file size changed while zipping
2021-11-02T09:31:37.589 INFO:teuthology.orchestra.run.smithi036.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-mon.smithi036.log: file size changed while zipping
2021-11-02T09:31:40.710 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-mon.smithi123.log: file size changed while zipping
2021-11-02T09:31:49.746 INFO:teuthology.orchestra.run.smithi036.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi036.fmcgfn.log: file size changed while zipping
2021-11-02T09:32:00.515 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-02T09:32:15.014 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-02T09:32:35.818 INFO:teuthology.orchestra.run.smithi036.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi036.pebuqk.log: file size changed while zipping
2021-11-02T09:32:45.364 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-mds.cephfs.smithi123.hknsmu.log: file size changed while zipping
2021-11-02T09:32:52.838 INFO:teuthology.orchestra.run.smithi036.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2021-11-02T09:33:05.252 INFO:teuthology.orchestra.run.smithi036.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-mgr.smithi036.fixsbi.log: file size changed while zipping
2021-11-02T09:33:20.241 INFO:teuthology.orchestra.run.smithi036.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-02T09:33:38.430 INFO:teuthology.orchestra.run.smithi036.stderr:gzip: /var/log/ceph/c02e7534-3b5c-11ec-8c28-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-11-02T09:34:26.852 INFO:teuthology.kill:No teuthology processes running
2021-11-02T09:34:27.986 INFO:teuthology.kill:Nuking machines: ['smithi036', 'smithi123']
2021-11-02T09:34:28.967 INFO:teuthology.kill:2021-11-02 09:34:28,967.967 INFO:teuthology.nuke:targets:
2021-11-02T09:34:28.968 INFO:teuthology.kill:  smithi036.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsWioJHb2ISsffyToqDPDWadl4gY0g8OiWTP5E8sNC9Ey/s3gM1F6L19bKn1pZVPYzV95en7vN3VNUnDTsryejFf2XBxWkfd0hvkjEgyVKHnGApO2e8WdpmiOm3gVYtX1NHmLCxLBj/rq8x2LCWjbi47+cRTvx+6H3njXcMkmJGkeGe6/1RvEWC4/Z2mZrF+CU212O2tB56dwxkjNqqxo98An8GHThHcnQkoOSU5dMDBSjZN46lyUDpyyZRuOWKx9Z1CXwiQ38cQl1xoBkfzZMeaLU1YXBUrxwzl8QD0xZ0KxMJOQLE1/kWBUf97pNx39knTAcHsBXCx3Y8IiSrrTBjksxoM491HyljFYUYIRZRqmOLQbAB3BF+tvLKSYbAkfeId0LfZuqpTFUsOhmL2bTMzT2p7g1GFp2e0aeK/gIimsKDaiM8WCYs+NrBNtbEfcyKbDKbPbx5OsSdzqGgTJO3cDlTfKiGNVd4k6sF1EY4C5qzG89qcZz/Ze//moa5eE=
2021-11-02T09:34:28.969 INFO:teuthology.kill:  smithi123.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC15mugMWU9RV0mbltP/I1v0nYR6CMc785LYFV2oNj98vdqj4E4AlbbVRSJcJ8hUYzFtph5YR/xDFPA5KMLcB5ubNq7stVWf0SbMSlLaSnPRnN2QPnDLU/TM1KeTKXs0s7tiM9V0has8hneMFc8iWVaGFQ9+8lYTPqM+OX+aZTm8XdTfZxRcutjFIZFyl1hbZSXS9n8g7IwLPBpspuEj2n2jKBR44QDI7EOF2+NVgb06yNQTfGqVVo0dI7OBEl1CLvZeiPLsQVvE5I1P2VwJ1JgwHCYVh03oOOdzET6Dg0VQijE14AjAktt31e+7J4yo301XR429goNdxi038OZzcWxiIWntmtwPvg4AUzbbHNlApxOiZYSWa4KKVDOiWyqhU/2AHI7L/BjUed70RnWrzaUvmMjpZ14imPbSr0dVK1y7tJx7tSwsIJTBQKlAKxW5foiehL8XgBPJqRuOOHiMc5TuS0mAZwjPR7cj5gsZokQew1W4/PMn7QepfR4SUHue5s=
2021-11-02T09:34:28.994 INFO:teuthology.kill:2021-11-02 09:34:28,994.994 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-02T09:34:28.997 INFO:teuthology.kill:2021-11-02 09:34:28,997.997 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-02T09:34:29.073 INFO:teuthology.kill:2021-11-02 09:34:29,073.073 INFO:teuthology.orchestra.console:Power off smithi123
2021-11-02T09:34:29.086 INFO:teuthology.kill:2021-11-02 09:34:29,085.085 INFO:teuthology.orchestra.console:Power off smithi036
2021-11-02T09:34:37.441 INFO:teuthology.kill:2021-11-02 09:34:37,441.441 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2021-11-02T09:34:37.589 INFO:teuthology.kill:2021-11-02 09:34:37,588.588 INFO:teuthology.lock.ops:unlocked: smithi123.front.sepia.ceph.com
2021-11-02T09:34:41.558 INFO:teuthology.kill:2021-11-02 09:34:41,558.558 INFO:teuthology.orchestra.console:Power off for smithi036 completed
2021-11-02T09:34:41.704 INFO:teuthology.kill:2021-11-02 09:34:41,703.703 INFO:teuthology.lock.ops:unlocked: smithi036.front.sepia.ceph.com
2021-11-02T09:36:41.906 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-02T09:36:42.038 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi036 but it was locked by another job: /home/teuthworker/archive/pdiazbou-2021-11-02_04:20:02-orch-wip-pereman2-testing-2021-11-01-2310-distro-basic-smithi/6474786
2021-11-02T09:36:42.039 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi123 but it was locked by another job: /home/teuthworker/archive/pdiazbou-2021-11-02_04:20:02-orch-wip-pereman2-testing-2021-11-01-2310-distro-basic-smithi/6474786