2021-11-04T15:53:23.175 INFO:root:teuthology version: 1.1.0-c56135d1
2021-11-04T15:53:23.294 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi157.front.sepia.ceph.com' reimaging
2021-11-04T15:53:23.295 INFO:teuthology.lock.ops:Updating [ubuntu@smithi157.front.sepia.ceph.com]: reset os type and version on server
2021-11-04T15:53:23.295 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2021-11-04T15:53:23.314 INFO:teuthology.lock.ops:Node 'ubuntu@smithi157.front.sepia.ceph.com' reimaging is complete
2021-11-04T15:53:23.314 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi171.front.sepia.ceph.com' reimaging
2021-11-04T15:53:23.315 INFO:teuthology.lock.ops:Updating [ubuntu@smithi171.front.sepia.ceph.com]: reset os type and version on server
2021-11-04T15:53:23.315 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server
2021-11-04T15:53:23.334 INFO:teuthology.lock.ops:Node 'ubuntu@smithi171.front.sepia.ceph.com' reimaging is complete
2021-11-04T15:53:23.526 INFO:teuthology.provision.fog.smithi157:Scheduling deploy of centos 8.3
2021-11-04T15:53:23.537 INFO:teuthology.provision.fog.smithi171:Scheduling deploy of centos 8.3
2021-11-04T15:53:23.902 INFO:teuthology.orchestra.console:Power off smithi171
2021-11-04T15:53:23.917 INFO:teuthology.orchestra.console:Power off smithi157
2021-11-04T15:53:32.277 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2021-11-04T15:53:32.379 INFO:teuthology.orchestra.console:Power on smithi157
2021-11-04T15:53:36.386 INFO:teuthology.orchestra.console:Power off for smithi171 completed
2021-11-04T15:53:36.488 INFO:teuthology.orchestra.console:Power on smithi171
2021-11-04T15:53:40.745 INFO:teuthology.orchestra.console:Power on for smithi171 completed
2021-11-04T15:53:40.753 INFO:teuthology.orchestra.console:Power on for smithi157 completed
2021-11-04T15:53:40.847 INFO:teuthology.provision.fog.smithi171:Waiting for deploy to finish
2021-11-04T15:53:40.854 INFO:teuthology.provision.fog.smithi157:Waiting for deploy to finish
2021-11-04T15:58:45.789 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-04T15:58:45.843 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-04T15:58:46.111 INFO:teuthology.provision.fog.smithi157:Node is ready
2021-11-04T15:58:46.126 INFO:teuthology.orchestra.run.smithi157.stdout:smithi157.front.sepia.ceph.com
2021-11-04T15:58:46.134 INFO:teuthology.provision.fog.smithi171:Node is ready
2021-11-04T15:58:46.149 INFO:teuthology.orchestra.run.smithi171.stdout:smithi171.front.sepia.ceph.com
2021-11-04T15:58:46.185 INFO:teuthology.orchestra.run.smithi157.stdout:172.21.15.157 smithi157.front.sepia.ceph.com smithi157
2021-11-04T15:58:46.204 INFO:teuthology.orchestra.run.smithi171.stdout:172.21.15.171 smithi171.front.sepia.ceph.com smithi171
2021-11-04T15:58:46.509 INFO:teuthology.provision.fog.smithi157:Deploy complete!
2021-11-04T15:58:46.528 INFO:teuthology.provision.fog.smithi171:Deploy complete!
2021-11-04T15:58:46.713 INFO:teuthology.lock.ops:Checking smithi157.front.sepia.ceph.com
2021-11-04T15:58:46.714 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-04T15:58:46.733 INFO:teuthology.lock.ops:Checking smithi171.front.sepia.ceph.com
2021-11-04T15:58:46.733 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-04T15:58:46.755 INFO:teuthology.lock.ops:Updating [smithi157.front.sepia.ceph.com]: set os type and version on server
2021-11-04T15:58:46.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2021-11-04T15:58:47.011 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64
2021-11-04T15:58:47.065 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Linux"
2021-11-04T15:58:47.065 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8"
2021-11-04T15:58:47.066 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos"
2021-11-04T15:58:47.066 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora"
2021-11-04T15:58:47.066 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8"
2021-11-04T15:58:47.066 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8"
2021-11-04T15:58:47.067 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-04T15:58:47.067 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31"
2021-11-04T15:58:47.067 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-04T15:58:47.067 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/"
2021-11-04T15:58:47.068 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-04T15:58:47.068 INFO:teuthology.orchestra.run.smithi157.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-04T15:58:47.068 INFO:teuthology.orchestra.run.smithi157.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-04T15:58:47.069 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server
2021-11-04T15:58:47.088 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: set os type and version on server
2021-11-04T15:58:47.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com'
2021-11-04T15:58:47.351 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64
2021-11-04T15:58:47.406 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Linux"
2021-11-04T15:58:47.406 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="8"
2021-11-04T15:58:47.406 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos"
2021-11-04T15:58:47.406 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora"
2021-11-04T15:58:47.407 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="8"
2021-11-04T15:58:47.407 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el8"
2021-11-04T15:58:47.407 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-04T15:58:47.407 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31"
2021-11-04T15:58:47.408 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-04T15:58:47.408 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/"
2021-11-04T15:58:47.408 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-04T15:58:47.408 INFO:teuthology.orchestra.run.smithi171.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-04T15:58:47.409 INFO:teuthology.orchestra.run.smithi171.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-04T15:58:47.409 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server
2021-11-04T15:58:47.509 INFO:teuthology.dispatcher.supervisor:Running job 6485114
2021-11-04T15:58:47.510 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/6485114 --name pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/6485114/orig.config.yaml
2021-11-04T15:58:47.515 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/6485114
2021-11-04T15:58:47.516 INFO:teuthology.dispatcher.supervisor:Job PID: 3440
2021-11-04T15:58:47.516 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-04T23:07:20.092 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-11-04T23:07:21.191 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-11-04T23:07:23.145 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-11-04T23:07:25.994 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-11-04T23:07:29.739 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-11-04T23:07:34.380 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-11-04T23:07:39.917 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/pdonnell-2021-11-04_15:43:53-fs-wip-pdonnell-testing-20211103.023355-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-11-05T04:00:06.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-05T04:00:06.218 INFO:teuthology.kill:Killing Pids: {3440}
2021-11-05T04:00:07.355 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2021-11-05T04:00:07.356 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2021-11-05T04:00:07.356 INFO:teuthology.misc:Compressing logs...
2021-11-05T04:00:07.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com'
2021-11-05T04:00:07.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com'
2021-11-05T04:00:07.857 INFO:teuthology.misc:Compressing logs...
2021-11-05T04:00:08.191 INFO:teuthology.misc:Compressing logs...
2021-11-05T04:00:08.910 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-mon.smithi157.log: file size changed while zipping
2021-11-05T04:00:09.050 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-05T04:00:09.290 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2021-11-05T04:00:09.658 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-05T04:00:09.983 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-mgr.smithi157.fooxuq.log: file size changed while zipping
2021-11-05T04:00:10.059 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-05T04:00:10.404 INFO:teuthology.orchestra.run.smithi157.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-11-05T04:00:11.263 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/d4f6eec2-3d89-11ec-8c28-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-05T04:00:35.273 INFO:teuthology.orchestra.run.smithi157.stderr:tar: .: file changed as we read it
2021-11-05T04:00:50.351 INFO:teuthology.kill:No teuthology processes running
2021-11-05T04:00:51.389 INFO:teuthology.kill:Nuking machines: ['smithi157', 'smithi171']
2021-11-05T04:00:52.287 INFO:teuthology.kill:2021-11-05 04:00:52,287.287 INFO:teuthology.nuke:targets:
2021-11-05T04:00:52.288 INFO:teuthology.kill: smithi157.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEDuRfi9eRsAp/sUohnDxBtoK0c+D27Yaj+XFdDjaom7h0OBIVHVFHQiGbmoVNJNqBy1Prp0QPHgtfE42Xi6Y4z75Zu6y8vXptFf7I1Fvum+mQVgkNxx55yyXlYDfyCqBce+hdBLrhXcAAZ8jWQ1jZfTx2DQG81Hd55fvqeQVlYK7qgn7Kr3Jkj0ND3fgD3hCK2csBgig7lfeHVDe036iz85kLTK+UMSBxHHgGz/jNrfCsxuwP01XLW8dQwsF8D2OauZ4xpED3nQsUvWciWIby4ByPUXcZx8IrV/ECsdxIjGPaKVwZBnI1noVxRBmsDbD68M8g705VAGLS3oKMBwlmfiJa++vKhSQP8q6w/oXTZi+22k5cMDLj1UsWoW+2H1LNXFQm5nXf3yynt4HmhzZo7ORcxc2757Lsl7WIV2yB8lR7HSfeWAvkCXw+xOSyI5G0WAHn2m8p/S4MfI+qYcqDT+NSnq/mwWla2F3sq2eYkyv0z1Oa/BjtXSZd3/Tee38=
2021-11-05T04:00:52.288 INFO:teuthology.kill: smithi171.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7vuMKaNTRNODwdavwCUwipCkH1LfVcFGwwXjLcW1eMPB0sYVFYYwrflr1Jl8ECnK4PXxA3sRAr0HiOxpdSVyAYuu/O1f5CZ6IsuHPvaDzn7Ep79wehfa1RJmklASdjvb4spRjKLCg1e72PyhHqwuST0+2i+kTdRN4itojMNhxvwvvjs4QrMniebTRYzkp5njiRZuhOvAnXUfKs2TSdmW6vAOzvAmMxnoHOGuliIAw6bCgXkaGVCLxavFOUDU9iv07Ys2XvUgJjsaL1UiRlfBb9kgWu1Z9CqWGde9bGnvmcSetL+TJgTwrBP3SsYAH72mPWCmc/Z47HD/bgi0dx7oVuMhoF6jKuBhnQO+F29UMKFSup8kJMAXs9ZLf4iIIK2HF9Z4jiuS6GEkR34Ft1E03SKdPuvzLeTgaUtbkWUouMsvdi98r28loq2mB1JRz+6tl04UQ61MI53Delrq+1NDMcQwCcYKycfmFfhb4ICj82xG2cec+QWN/rZL9FRZG2ZM=
2021-11-05T04:00:52.313 INFO:teuthology.kill:2021-11-05 04:00:52,313.313 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-05T04:00:52.316 INFO:teuthology.kill:2021-11-05 04:00:52,316.316 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-05T04:00:52.389 INFO:teuthology.kill:2021-11-05 04:00:52,388.388 INFO:teuthology.orchestra.console:Power off smithi157
2021-11-05T04:00:52.402 INFO:teuthology.kill:2021-11-05 04:00:52,402.402 INFO:teuthology.orchestra.console:Power off smithi171
2021-11-05T04:01:00.762 INFO:teuthology.kill:2021-11-05 04:01:00,762.762 INFO:teuthology.orchestra.console:Power off for smithi157 completed
2021-11-05T04:01:00.906 INFO:teuthology.kill:2021-11-05 04:01:00,906.906 INFO:teuthology.lock.ops:unlocked: smithi157.front.sepia.ceph.com
2021-11-05T04:01:04.880 INFO:teuthology.kill:2021-11-05 04:01:04,880.880 INFO:teuthology.orchestra.console:Power off for smithi171 completed
2021-11-05T04:01:05.027 INFO:teuthology.kill:2021-11-05 04:01:05,026.026 INFO:teuthology.lock.ops:unlocked: smithi171.front.sepia.ceph.com
2021-11-05T04:03:05.344 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15