2022-04-05T08:03:26.426 INFO:root:teuthology version: 1.1.0-d877b023 2022-04-05T08:03:26.562 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi174.front.sepia.ceph.com' reimaging 2022-04-05T08:03:26.563 INFO:teuthology.lock.ops:Updating [ubuntu@smithi174.front.sepia.ceph.com]: reset os type and version on server 2022-04-05T08:03:26.563 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2022-04-05T08:03:26.589 INFO:teuthology.lock.ops:Node 'ubuntu@smithi174.front.sepia.ceph.com' reimaging is complete 2022-04-05T08:03:26.589 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi186.front.sepia.ceph.com' reimaging 2022-04-05T08:03:26.590 INFO:teuthology.lock.ops:Updating [ubuntu@smithi186.front.sepia.ceph.com]: reset os type and version on server 2022-04-05T08:03:26.590 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-04-05T08:03:26.613 INFO:teuthology.lock.ops:Node 'ubuntu@smithi186.front.sepia.ceph.com' reimaging is complete 2022-04-05T08:03:26.803 INFO:teuthology.provision.fog.smithi174:Scheduling deploy of centos 8.stream 2022-04-05T08:03:26.822 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of centos 8.stream 2022-04-05T08:03:27.108 INFO:teuthology.orchestra.console:Power off smithi174 2022-04-05T08:03:27.129 INFO:teuthology.orchestra.console:Power off smithi186 2022-04-05T08:03:35.499 INFO:teuthology.orchestra.console:Power off for smithi174 completed 2022-04-05T08:03:35.600 INFO:teuthology.orchestra.console:Power on smithi174 2022-04-05T08:03:39.621 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2022-04-05T08:03:39.723 INFO:teuthology.orchestra.console:Power on smithi186 2022-04-05T08:03:43.970 INFO:teuthology.orchestra.console:Power on for smithi174 completed 2022-04-05T08:03:44.073 INFO:teuthology.provision.fog.smithi174:Waiting for deploy to finish 2022-04-05T08:03:48.090 INFO:teuthology.orchestra.console:Power on for smithi186 completed 2022-04-05T08:03:48.191 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish 2022-04-05T08:07:41.011 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-05T08:08:05.884 INFO:teuthology.provision.fog.smithi174:Node is ready 2022-04-05T08:08:05.904 INFO:teuthology.orchestra.run.smithi174.stdout:smithi174.front.sepia.ceph.com 2022-04-05T08:08:05.961 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.15.174 smithi174.front.sepia.ceph.com smithi174 2022-04-05T08:08:06.317 INFO:teuthology.provision.fog.smithi174:Deploy complete! 2022-04-05T08:08:28.693 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-05T08:08:28.966 INFO:teuthology.provision.fog.smithi186:Node is ready 2022-04-05T08:08:28.985 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186.front.sepia.ceph.com 2022-04-05T08:08:29.045 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.186 smithi186.front.sepia.ceph.com smithi186 2022-04-05T08:08:29.410 INFO:teuthology.provision.fog.smithi186:Deploy complete! 2022-04-05T08:08:29.602 INFO:teuthology.lock.ops:Checking smithi174.front.sepia.ceph.com 2022-04-05T08:08:29.603 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-05T08:08:29.624 INFO:teuthology.lock.ops:Checking smithi186.front.sepia.ceph.com 2022-04-05T08:08:29.625 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-05T08:08:29.644 INFO:teuthology.lock.ops:Updating [smithi174.front.sepia.ceph.com]: set os type and version on server 2022-04-05T08:08:29.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2022-04-05T08:08:29.890 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2022-04-05T08:08:29.947 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2022-04-05T08:08:29.948 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="8" 2022-04-05T08:08:29.948 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2022-04-05T08:08:29.948 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2022-04-05T08:08:29.949 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="8" 2022-04-05T08:08:29.949 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el8" 2022-04-05T08:08:29.949 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-05T08:08:29.949 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2022-04-05T08:08:29.950 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-05T08:08:29.950 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2022-04-05T08:08:29.950 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-05T08:08:29.950 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-05T08:08:29.951 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-05T08:08:29.952 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2022-04-05T08:08:29.973 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: set os type and version on server 2022-04-05T08:08:29.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-04-05T08:08:30.168 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2022-04-05T08:08:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2022-04-05T08:08:30.228 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8" 2022-04-05T08:08:30.228 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2022-04-05T08:08:30.229 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2022-04-05T08:08:30.229 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8" 2022-04-05T08:08:30.229 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2022-04-05T08:08:30.229 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-05T08:08:30.230 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2022-04-05T08:08:30.230 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-05T08:08:30.230 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2022-04-05T08:08:30.230 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-05T08:08:30.230 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-05T08:08:30.231 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-05T08:08:30.231 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-04-05T08:08:30.333 INFO:teuthology.dispatcher.supervisor:Running job 6777361 2022-04-05T08:08:30.333 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-04-05_04:17:02-fs-pacific-distro-default-smithi/6777361 --name teuthology-2022-04-05_04:17:02-fs-pacific-distro-default-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} -- /home/teuthworker/archive/teuthology-2022-04-05_04:17:02-fs-pacific-distro-default-smithi/6777361/orig.config.yaml 2022-04-05T08:08:30.337 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-04-05_04:17:02-fs-pacific-distro-default-smithi/6777361 2022-04-05T08:08:30.338 INFO:teuthology.dispatcher.supervisor:Job PID: 1321 2022-04-05T08:08:30.338 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-04-05T14:38:59.051 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-04-05T14:38:59.315 INFO:teuthology.kill:Killing Pids: {1321} 2022-04-05T14:39:00.501 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-04-05T14:39:00.502 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-04-05T14:39:00.502 INFO:teuthology.misc:Compressing logs... 2022-04-05T14:39:00.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2022-04-05T14:39:00.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-04-05T14:39:01.146 INFO:teuthology.misc:Compressing logs... 2022-04-05T14:42:04.051 INFO:teuthology.misc:Compressing logs... 2022-04-05T14:44:37.413 INFO:teuthology.orchestra.run.smithi174.stderr:tar: .: file changed as we read it 2022-04-05T14:45:06.938 INFO:teuthology.kill:No teuthology processes running 2022-04-05T14:45:08.094 INFO:teuthology.kill:Nuking machines: ['smithi174', 'smithi186'] 2022-04-05T14:45:09.121 INFO:teuthology.kill:2022-04-05 14:45:09,121.121 INFO:teuthology.nuke:targets: 2022-04-05T14:45:09.122 INFO:teuthology.kill: smithi174.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA25YNgxwmBEEMGmak1vstUdGlblS76xVzIro4zMAhOTGD4jHa9m4N9T6PV2Ho56xOPz04Yn9QbZfh6etvPnqKbTPOX3soUYQRm2druVqvK163q+QTmEValuG6k//OkoN53lh+DZHS/LnV82wJBVrlG7hNuXuSy8yvJVFOKVUUPqD+cLGy2te1rl8cmZubuZegV5IJloWQqgWoQv2p81a4b/NJIZtEfURF87AyX6RqJG50wBbWsIUO1tsrq2qALfpQj5pZBgvlI00GvuyBC9E9oxJw5Zg3bvXSzQ8z4mQvoC1Mq0P8dI4ALnYsU6zsn7X8+a5ppKnsMhwZ/4Tv5ntfJgOC0tCU8XYziEw7PlWNO3NzFTZ7cbDBMB3hlF82TGgLQzc6jaWjKzunrtUpGhrJGHgRAMu1uxrSotCL9GPorITSauYZQPDFMXjzwQUAHEDBqy/0uIDi58R3VEBBI164EC05uAXFllpNyua0AW23WF5xFOjIvHxB9Vea54p2/tE= 2022-04-05T14:45:09.123 INFO:teuthology.kill: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDut0K1xD5QTufGhfgcDYrJUaYW9z2nYlFC4Xl1iwgRYEEWBNiQ3ZvTIs4aKgGsbMrk9bFLm00xVAdhOj6qC88WOTfQZx1lWRT5+eyhp3XZ8pDXsK5uIhAMPJxOhy6ytXvQbJ9nV3rvdc06Um2A1crwsS174S0Cx2QdXvgEoMY5ab5S7IsNiM5C9CNRhgEMYh1KkjCdzuT5jNjREDFBnYQHW445a+wOmxeZ8ka41NropaySLXm2wJ8jmeZDDcezIbekFv0aMgNo4AQCu9ZaPpsOr7sPdtFIUhZSzg02Pr/IslV7EKbj+dAAi3VBA3RwaEmaEcs46UjLxOgKY+fhGLOikbPceGx4p20AHKIESIdzQxewTIcPPojULCgSo/3gVReenYUktGXSIIArRZdvbUn2VCPj3cVL+XeD+6D0vK44VgalDH5zhVake0jyq3j56g6NecfU220IjqHgFK9LlKVR+OCbzvG9oVIWURi5ZHD5zU/6F5k2K0F9oB/9Uxb5Wa0= 2022-04-05T14:45:09.149 INFO:teuthology.kill:2022-04-05 14:45:09,148.148 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-05T14:45:09.152 INFO:teuthology.kill:2022-04-05 14:45:09,152.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-05T14:45:09.226 INFO:teuthology.kill:2022-04-05 14:45:09,226.226 INFO:teuthology.orchestra.console:Power off smithi186 2022-04-05T14:45:09.240 INFO:teuthology.kill:2022-04-05 14:45:09,240.240 INFO:teuthology.orchestra.console:Power off smithi174 2022-04-05T14:45:17.599 INFO:teuthology.kill:2022-04-05 14:45:17,599.599 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2022-04-05T14:45:17.752 INFO:teuthology.kill:2022-04-05 14:45:17,751.751 INFO:teuthology.lock.ops:unlocked: smithi186.front.sepia.ceph.com 2022-04-05T14:45:21.720 INFO:teuthology.kill:2022-04-05 14:45:21,719.719 INFO:teuthology.orchestra.console:Power off for smithi174 completed 2022-04-05T14:45:21.866 INFO:teuthology.kill:2022-04-05 14:45:21,865.865 INFO:teuthology.lock.ops:unlocked: smithi174.front.sepia.ceph.com 2022-04-05T14:47:22.196 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-04-05T14:47:22.312 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi174 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-04-05_14:34:27-rgw-wip-cbodley-testing-distro-default-smithi/6777905 2022-04-05T14:47:22.313 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi186 but it was locked by another job: /home/teuthworker/archive/cbodley-2022-04-05_14:34:27-rgw-wip-cbodley-testing-distro-default-smithi/6777905