2023-04-22T18:36:54.214 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-22T18:36:54.371 INFO:teuthology.lock.ops:Start node 'smithi029.front.sepia.ceph.com' reimaging
2023-04-22T18:36:54.373 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: reset os type and version on server
2023-04-22T18:36:54.373 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2023-04-22T18:36:54.395 INFO:teuthology.lock.ops:Node 'smithi029.front.sepia.ceph.com' reimaging is complete
2023-04-22T18:36:54.395 INFO:teuthology.lock.ops:Start node 'smithi096.front.sepia.ceph.com' reimaging
2023-04-22T18:36:54.396 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: reset os type and version on server
2023-04-22T18:36:54.396 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server
2023-04-22T18:36:54.419 INFO:teuthology.lock.ops:Node 'smithi096.front.sepia.ceph.com' reimaging is complete
2023-04-22T18:36:54.554 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of rhel 8.6
2023-04-22T18:36:54.601 INFO:teuthology.provision.fog.smithi096:Scheduling deploy of rhel 8.6
2023-04-22T18:36:54.798 INFO:teuthology.orchestra.console:Power off smithi029
2023-04-22T18:36:54.821 INFO:teuthology.orchestra.console:Power off smithi096
2023-04-22T18:37:03.190 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2023-04-22T18:37:03.196 INFO:teuthology.orchestra.console:Power off for smithi096 completed
2023-04-22T18:37:03.293 INFO:teuthology.orchestra.console:Power on smithi029
2023-04-22T18:37:03.303 INFO:teuthology.orchestra.console:Power on smithi096
2023-04-22T18:37:11.660 INFO:teuthology.orchestra.console:Power on for smithi029 completed
2023-04-22T18:37:11.670 INFO:teuthology.orchestra.console:Power on for smithi096 completed
2023-04-22T18:37:11.762 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish
2023-04-22T18:37:11.772 INFO:teuthology.provision.fog.smithi096:Waiting for deploy to finish
2023-04-22T18:40:00.773 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.96
2023-04-22T18:40:09.829 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.96
2023-04-22T18:40:18.885 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.96
2023-04-22T18:40:27.941 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.96
2023-04-22T18:40:30.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.29
2023-04-22T18:40:36.997 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.96
2023-04-22T18:40:40.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.29
2023-04-22T18:40:46.053 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.96
2023-04-22T18:40:49.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.29
2023-04-22T18:40:58.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.29
2023-04-22T18:41:07.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.29
2023-04-22T18:41:52.097 WARNING:teuthology.provision.fog:timed out
2023-04-22T18:41:58.439 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-22T18:41:58.726 INFO:teuthology.provision.fog.smithi096:Node is ready
2023-04-22T18:41:58.744 INFO:teuthology.orchestra.run.smithi096.stdout:smithi096
2023-04-22T18:41:58.802 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96 smithi096.front.sepia.ceph.com smithi096
2023-04-22T18:41:59.153 INFO:teuthology.provision.fog.smithi096:Deploy complete!
2023-04-22T18:42:13.190 WARNING:teuthology.provision.fog:timed out
2023-04-22T18:42:20.198 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-22T18:42:20.522 INFO:teuthology.provision.fog.smithi029:Node is ready
2023-04-22T18:42:20.540 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com
2023-04-22T18:42:20.598 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029
2023-04-22T18:42:20.967 INFO:teuthology.provision.fog.smithi029:Deploy complete!
2023-04-22T18:42:21.236 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com
2023-04-22T18:42:21.238 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-22T18:42:21.268 INFO:teuthology.lock.ops:Checking smithi096.front.sepia.ceph.com
2023-04-22T18:42:21.269 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-22T18:42:21.294 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server
2023-04-22T18:42:21.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2023-04-22T18:42:21.488 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64
2023-04-22T18:42:21.550 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Red Hat Enterprise Linux"
2023-04-22T18:42:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8.6 (Ootpa)"
2023-04-22T18:42:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:ID="rhel"
2023-04-22T18:42:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="fedora"
2023-04-22T18:42:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8.6"
2023-04-22T18:42:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8"
2023-04-22T18:42:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-04-22T18:42:21.551 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31"
2023-04-22T18:42:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-04-22T18:42:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.redhat.com/"
2023-04-22T18:42:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-04-22T18:42:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-22T18:42:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:
2023-04-22T18:42:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-22T18:42:21.552 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-04-22T18:42:21.553 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-04-22T18:42:21.553 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-04-22T18:42:21.554 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2023-04-22T18:42:21.576 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: set os type and version on server
2023-04-22T18:42:21.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com'
2023-04-22T18:42:21.832 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64
2023-04-22T18:42:21.888 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="Red Hat Enterprise Linux"
2023-04-22T18:42:21.889 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="8.6 (Ootpa)"
2023-04-22T18:42:21.889 INFO:teuthology.orchestra.run.smithi096.stdout:ID="rhel"
2023-04-22T18:42:21.889 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="fedora"
2023-04-22T18:42:21.889 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="8.6"
2023-04-22T18:42:21.889 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el8"
2023-04-22T18:42:21.889 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31"
2023-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.redhat.com/"
2023-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi096.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi096.stdout:
2023-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-22T18:42:21.891 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-04-22T18:42:21.891 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-04-22T18:42:21.891 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-04-22T18:42:21.891 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server
2023-04-22T18:42:22.007 INFO:teuthology.dispatcher.supervisor:Running job 7248248
2023-04-22T18:42:22.008 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-04-21_18:21:42-fs-wip-pdonnell-testing-20230421.153328-distro-default-smithi/7248248 --name pdonnell-2023-04-21_18:21:42-fs-wip-pdonnell-testing-20230421.153328-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} -- /home/teuthworker/archive/pdonnell-2023-04-21_18:21:42-fs-wip-pdonnell-testing-20230421.153328-distro-default-smithi/7248248/orig.config.yaml
2023-04-22T18:42:22.011 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2023-04-21_18:21:42-fs-wip-pdonnell-testing-20230421.153328-distro-default-smithi/7248248
2023-04-22T18:42:22.012 INFO:teuthology.dispatcher.supervisor:Job PID: 23055
2023-04-22T18:42:22.012 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-04-23T06:43:19.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-04-23T06:43:20.007 INFO:teuthology.kill:Killing Pids: {23055}
2023-04-23T06:43:21.028 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2']
2023-04-23T06:43:21.029 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']
2023-04-23T06:43:21.029 INFO:teuthology.misc:Compressing logs...
2023-04-23T06:43:21.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2023-04-23T06:43:21.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com'
2023-04-23T06:43:22.122 INFO:teuthology.misc:Compressing logs...
2023-04-23T06:43:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2023-04-23T06:43:22.180 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2023-04-23T06:43:22.202 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2023-04-23T06:43:22.202 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2023-04-23T06:43:22.203 INFO:teuthology.orchestra.run.smithi096.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	/home/ubuntu/cephtest/archive/syslog/misc.log:	 80.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2023-04-23T06:43:22.246 INFO:teuthology.orchestra.run.smithi096.stderr: 88.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2023-04-23T06:43:22.274 INFO:teuthology.orchestra.run.smithi029.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	/home/ubuntu/cephtest/archive/syslog/misc.log:	 89.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2023-04-23T06:43:22.293 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2023-04-23T06:43:22.365 INFO:teuthology.misc:Compressing logs...
2023-04-23T07:14:57.014 INFO:teuthology.kill:No teuthology processes running
2023-04-23T07:14:57.990 INFO:teuthology.kill:Nuking machines: ['smithi029', 'smithi096']
2023-04-23T07:14:58.674 INFO:teuthology.kill:2023-04-23 07:14:58,673.673 INFO:teuthology.nuke:targets:
2023-04-23T07:14:58.675 INFO:teuthology.kill:  smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-04-23T07:14:58.675 INFO:teuthology.kill:  smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-04-23T07:14:58.675 INFO:teuthology.kill:2023-04-23 07:14:58,674.674 INFO:teuthology.nuke:Checking targets against current locks
2023-04-23T07:14:58.730 INFO:teuthology.kill:2023-04-23 07:14:58,730.730 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-23T07:14:58.753 INFO:teuthology.kill:2023-04-23 07:14:58,753.753 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-23T07:14:58.814 INFO:teuthology.kill:2023-04-23 07:14:58,813.813 INFO:teuthology.orchestra.console:Power off smithi029
2023-04-23T07:14:58.831 INFO:teuthology.kill:2023-04-23 07:14:58,831.831 INFO:teuthology.orchestra.console:Power off smithi096
2023-04-23T07:15:07.189 INFO:teuthology.kill:2023-04-23 07:15:07,189.189 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2023-04-23T07:15:07.346 INFO:teuthology.kill:2023-04-23 07:15:07,345.345 INFO:teuthology.lock.ops:unlocked: smithi029.front.sepia.ceph.com
2023-04-23T07:15:11.311 INFO:teuthology.kill:2023-04-23 07:15:11,311.311 INFO:teuthology.orchestra.console:Power off for smithi096 completed
2023-04-23T07:15:11.470 INFO:teuthology.kill:2023-04-23 07:15:11,470.470 INFO:teuthology.lock.ops:unlocked: smithi096.front.sepia.ceph.com
2023-04-23T07:17:11.830 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15