2022-03-13T06:26:35.372 INFO:root:teuthology version: 1.1.0-fbfc7b38
2022-03-13T06:26:35.516 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging
2022-03-13T06:26:35.517 INFO:teuthology.lock.ops:Updating [ubuntu@smithi057.front.sepia.ceph.com]: reset os type and version on server
2022-03-13T06:26:35.517 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-03-13T06:26:35.538 INFO:teuthology.lock.ops:Node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging is complete
2022-03-13T06:26:35.538 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging
2022-03-13T06:26:35.538 INFO:teuthology.lock.ops:Updating [ubuntu@smithi104.front.sepia.ceph.com]: reset os type and version on server
2022-03-13T06:26:35.539 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2022-03-13T06:26:35.559 INFO:teuthology.lock.ops:Node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging is complete
2022-03-13T06:26:35.749 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of centos 8.stream
2022-03-13T06:26:35.774 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of centos 8.stream
2022-03-13T06:26:36.237 INFO:teuthology.orchestra.console:Power off smithi057
2022-03-13T06:26:36.254 INFO:teuthology.orchestra.console:Power off smithi104
2022-03-13T06:26:36.387 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2022-03-13T06:26:36.401 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2022-03-13T06:26:36.489 INFO:teuthology.orchestra.console:Power on smithi057
2022-03-13T06:26:36.503 INFO:teuthology.orchestra.console:Power on smithi104
2022-03-13T06:26:40.760 INFO:teuthology.orchestra.console:Power on for smithi104 completed
2022-03-13T06:26:40.861 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish
2022-03-13T06:26:44.866 INFO:teuthology.orchestra.console:Power on for smithi057 completed
2022-03-13T06:26:44.967 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish
2022-03-13T06:30:47.745 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-13T06:31:13.432 INFO:teuthology.provision.fog.smithi104:Node is ready
2022-03-13T06:31:13.455 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104.front.sepia.ceph.com
2022-03-13T06:31:13.515 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104 smithi104.front.sepia.ceph.com smithi104
2022-03-13T06:31:13.887 INFO:teuthology.provision.fog.smithi104:Deploy complete!
2022-03-13T06:31:17.213 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-13T06:31:17.492 INFO:teuthology.provision.fog.smithi057:Node is ready
2022-03-13T06:31:17.510 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com
2022-03-13T06:31:17.571 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057
2022-03-13T06:31:17.924 INFO:teuthology.provision.fog.smithi057:Deploy complete!
2022-03-13T06:31:18.113 INFO:teuthology.lock.ops:Checking smithi057.front.sepia.ceph.com
2022-03-13T06:31:18.114 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-13T06:31:18.135 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com
2022-03-13T06:31:18.135 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-13T06:31:18.155 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: set os type and version on server
2022-03-13T06:31:18.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-03-13T06:31:18.346 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64
2022-03-13T06:31:18.404 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream"
2022-03-13T06:31:18.405 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8"
2022-03-13T06:31:18.405 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos"
2022-03-13T06:31:18.405 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora"
2022-03-13T06:31:18.406 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8"
2022-03-13T06:31:18.406 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8"
2022-03-13T06:31:18.406 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-13T06:31:18.406 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31"
2022-03-13T06:31:18.406 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-13T06:31:18.407 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/"
2022-03-13T06:31:18.407 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-13T06:31:18.407 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-13T06:31:18.407 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-13T06:31:18.409 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-03-13T06:31:18.427 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server
2022-03-13T06:31:18.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2022-03-13T06:31:18.627 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64
2022-03-13T06:31:18.686 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream"
2022-03-13T06:31:18.687 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8"
2022-03-13T06:31:18.687 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos"
2022-03-13T06:31:18.687 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora"
2022-03-13T06:31:18.687 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8"
2022-03-13T06:31:18.688 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8"
2022-03-13T06:31:18.688 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-13T06:31:18.688 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31"
2022-03-13T06:31:18.688 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-13T06:31:18.689 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/"
2022-03-13T06:31:18.689 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-13T06:31:18.689 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-13T06:31:18.689 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-13T06:31:18.690 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2022-03-13T06:31:18.790 INFO:teuthology.dispatcher.supervisor:Running job 6734236
2022-03-13T06:31:18.790 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734236 --name teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734236/orig.config.yaml
2022-03-13T06:31:18.795 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6734236
2022-03-13T06:31:18.795 INFO:teuthology.dispatcher.supervisor:Job PID: 312
2022-03-13T06:31:18.796 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-13T13:01:48.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-13T13:01:48.680 INFO:teuthology.kill:Killing Pids: {312}
2022-03-13T13:01:49.946 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2022-03-13T13:01:49.947 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2022-03-13T13:01:49.948 INFO:teuthology.misc:Compressing logs...
2022-03-13T13:01:49.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-03-13T13:01:50.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2022-03-13T13:01:50.188 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-13T13:01:50.369 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-03-13T13:01:50.412 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-13T13:01:50.413 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2022-03-13T13:01:50.469 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-03-13T13:01:50.470 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now
2022-03-13T13:01:50.471 INFO:teuthology.misc:Compressing logs...
2022-03-13T13:01:50.905 INFO:teuthology.misc:Compressing logs...
2022-03-13T13:01:54.384 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-mon.smithi104.log: file size changed while zipping
2022-03-13T13:02:09.978 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-03-13T13:02:12.720 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-mgr.smithi057.mwwmai.log: file size changed while zipping
2022-03-13T13:02:15.306 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-03-13T13:02:25.523 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-03-13T13:02:38.195 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-mon.smithi057.log: file size changed while zipping
2022-03-13T13:02:39.702 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-03-13T13:02:57.247 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-03-13T13:02:59.909 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-03-13T13:03:15.448 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-03-13T13:03:23.171 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/5ae83ee6-a298-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-03-13T13:03:58.900 INFO:teuthology.kill:No teuthology processes running
2022-03-13T13:04:00.092 INFO:teuthology.kill:Nuking machines: ['smithi057', 'smithi104']
2022-03-13T13:04:01.162 INFO:teuthology.kill:2022-03-13 13:04:01,161.161 INFO:teuthology.nuke:targets:
2022-03-13T13:04:01.163 INFO:teuthology.kill:  smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCylN3GThzdYrhBRRYV38T/KxixxCLX2i0Od0cRXgTeObb5KkM/J6pxpXboaHTZ8FdDglsIWwBqXQiz0S/LPpPSPVyWsSOmZu20fuhvwNaW7XlqNHcWryMWVUJbSo9EQfnaerQ+SMEPBlHFwOHD8nxyF/aDMpzN0Bg3JVZ//wDEw6sV2gngeXmehultsQ0iJP8UmoCdn1ZLievja6JEyiDovhH7WjAF6UzIMSwnKn65+c59bNcBqJCapdEEmZqaErfI0IsL8kSFeN/rMFZLof2TIR0zcrSf4Msed7wFbQhyNIF1p09AZdjTnKaRXVvz0eL5A2wmg5IeFS04n7ChrtQ0kfYP3iGG+o11mkRC6G+zKQUyEG59ep2OkPJzz4bj7eNo9cWtyIl+C8JGNXvOVviytzoEW8Mm7FSC4HBjPGYVam+lnaqeJ8zH8LuR5xMjtBWAVvfid0QDzFGpiDlFR1e7UwYghSaP1iAoIIy/yqDfIc0lsAN/j7xtR6nXGYTaTGk=
2022-03-13T13:04:01.163 INFO:teuthology.kill:  smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGthHODKgdBWfpxt8J4iyEJpnMXLpf7LQLDEtfB8jL4kZtud2pPXEFBd9i+/l9xM1xlqE43+5SxFliJ81kL2TW/8874yn1noxzKpG1VGM0Lb4l4QP8ofKGjPl+SnNuq1shiPS1XDT5rX9n1LrreZv6qAznoZhdDcKRDAQiutWtqT2GZoaeydXGMvutNH30Co/nWDkhJY+nc6rcKgGdsr8JggDQ7f4vrP634BPw5yDuUVcqluf7oJOa+h3ghTRyTxHJ5KKUSxIDo6X1TAk5IwBOJWzq5aZBQzZwLaCh/CQGewL3p2qiAeI8YzToInVq/GVKY8zC979yd+Iknkj2xTWoxNfnjQNdGDvT6Lj536gOp2HKvAuEfsFvi5uLN7O3yf19ESPoX213laY5KNbxV3IEHwZPnHflzCq5yWfL1kSbdIhBjufv3F07/tuVY0nHiKLcWjj6ympddVQADty8ZWeH7rmMIdZGDFK+AAbu0mxRrsL3uUus/4HZT5KYNRW86P0=
2022-03-13T13:04:01.191 INFO:teuthology.kill:2022-03-13 13:04:01,191.191 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-13T13:04:01.197 INFO:teuthology.kill:2022-03-13 13:04:01,197.197 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-13T13:04:01.286 INFO:teuthology.kill:2022-03-13 13:04:01,286.286 INFO:teuthology.orchestra.console:Power off smithi057
2022-03-13T13:04:01.306 INFO:teuthology.kill:2022-03-13 13:04:01,305.305 INFO:teuthology.orchestra.console:Power off smithi104
2022-03-13T13:04:09.678 INFO:teuthology.kill:2022-03-13 13:04:09,677.677 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2022-03-13T13:04:09.832 INFO:teuthology.kill:2022-03-13 13:04:09,831.831 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com
2022-03-13T13:04:13.797 INFO:teuthology.kill:2022-03-13 13:04:13,796.796 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2022-03-13T13:04:13.945 INFO:teuthology.kill:2022-03-13 13:04:13,945.945 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com
2022-03-13T13:06:14.290 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-13T13:06:14.390 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi057 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-03-13_12:58:02-rados:thrash-wip-rf-42714-pacific-distro-basic-smithi/6734979
2022-03-13T13:06:14.391 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi104 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-03-13_12:58:02-rados:thrash-wip-rf-42714-pacific-distro-basic-smithi/6734981