2023-05-23T07:53:25.430 INFO:root:teuthology version: 0.0.1.dev105+g6fb4f0c
2023-05-23T07:53:25.580 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging
2023-05-23T07:53:25.581 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:53:25.581 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2023-05-23T07:53:25.607 INFO:teuthology.lock.ops:Node 'smithi170.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:53:25.608 INFO:teuthology.lock.ops:Start node 'smithi194.front.sepia.ceph.com' reimaging
2023-05-23T07:53:25.608 INFO:teuthology.lock.ops:Updating [smithi194.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:53:25.608 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server
2023-05-23T07:53:25.631 INFO:teuthology.lock.ops:Node 'smithi194.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:53:25.773 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of rhel 8.6
2023-05-23T07:53:25.794 INFO:teuthology.provision.fog.smithi194:Scheduling deploy of rhel 8.6
2023-05-23T07:53:26.091 INFO:teuthology.orchestra.console:Power off smithi170
2023-05-23T07:53:26.111 INFO:teuthology.orchestra.console:Power off smithi194
2023-05-23T07:53:26.269 INFO:teuthology.orchestra.console:Power off for smithi194 completed
2023-05-23T07:53:26.370 INFO:teuthology.orchestra.console:Power on smithi194
2023-05-23T07:53:34.479 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2023-05-23T07:53:34.580 INFO:teuthology.orchestra.console:Power on smithi170
2023-05-23T07:53:34.730 INFO:teuthology.orchestra.console:Power on for smithi194 completed
2023-05-23T07:53:34.831 INFO:teuthology.provision.fog.smithi194:Waiting for deploy to finish
2023-05-23T07:53:42.950 INFO:teuthology.orchestra.console:Power on for smithi170 completed
2023-05-23T07:53:43.051 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2023-05-23T07:56:08.496 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.194
2023-05-23T07:56:17.552 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.194
2023-05-23T07:56:26.608 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.194
2023-05-23T07:56:31.824 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-05-23T07:56:35.664 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.194
2023-05-23T07:56:40.880 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-05-23T07:56:44.720 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.194
2023-05-23T07:56:49.936 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-05-23T07:56:53.776 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.194
2023-05-23T07:56:58.992 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-05-23T07:57:02.832 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.194
2023-05-23T07:57:08.048 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-05-23T07:58:08.888 WARNING:teuthology.provision.fog:timed out
2023-05-23T07:58:14.056 WARNING:teuthology.provision.fog:timed out
2023-05-23T07:58:15.230 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T07:58:15.519 INFO:teuthology.provision.fog.smithi194:Node is ready
2023-05-23T07:58:15.536 INFO:teuthology.orchestra.run.smithi194.stdout:smithi194.front.sepia.ceph.com
2023-05-23T07:58:15.593 INFO:teuthology.orchestra.run.smithi194.stdout:172.21.15.194 smithi194.front.sepia.ceph.com smithi194
2023-05-23T07:58:15.933 INFO:teuthology.provision.fog.smithi194:Deploy complete!
2023-05-23T07:58:20.401 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T07:58:20.700 INFO:teuthology.provision.fog.smithi170:Node is ready
2023-05-23T07:58:20.718 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com
2023-05-23T07:58:20.776 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170
2023-05-23T07:58:21.233 INFO:teuthology.provision.fog.smithi170:Deploy complete!
2023-05-23T07:58:21.392 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com
2023-05-23T07:58:21.392 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-23T07:58:21.423 INFO:teuthology.lock.ops:Checking smithi194.front.sepia.ceph.com
2023-05-23T07:58:21.424 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server
2023-05-23T07:58:21.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-05-23T07:58:21.608 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64
2023-05-23T07:58:21.664 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T07:58:21.664 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T07:58:21.664 INFO:teuthology.orchestra.run.smithi170.stdout:ID="rhel"
2023-05-23T07:58:21.664 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="fedora"
2023-05-23T07:58:21.665 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8.6"
2023-05-23T07:58:21.665 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8"
2023-05-23T07:58:21.665 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T07:58:21.665 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31"
2023-05-23T07:58:21.665 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T07:58:21.665 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T07:58:21.665 INFO:teuthology.orchestra.run.smithi170.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T07:58:21.666 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T07:58:21.666 INFO:teuthology.orchestra.run.smithi170.stdout:
2023-05-23T07:58:21.666 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T07:58:21.666 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T07:58:21.666 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T07:58:21.666 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T07:58:21.667 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2023-05-23T07:58:21.689 INFO:teuthology.lock.ops:Updating [smithi194.front.sepia.ceph.com]: set os type and version on server
2023-05-23T07:58:21.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com'
2023-05-23T07:58:21.882 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64
2023-05-23T07:58:21.940 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T07:58:21.940 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T07:58:21.940 INFO:teuthology.orchestra.run.smithi194.stdout:ID="rhel"
2023-05-23T07:58:21.940 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="fedora"
2023-05-23T07:58:21.941 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8.6"
2023-05-23T07:58:21.941 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8"
2023-05-23T07:58:21.941 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T07:58:21.941 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31"
2023-05-23T07:58:21.941 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T07:58:21.941 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T07:58:21.941 INFO:teuthology.orchestra.run.smithi194.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T07:58:21.942 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T07:58:21.942 INFO:teuthology.orchestra.run.smithi194.stdout:
2023-05-23T07:58:21.942 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T07:58:21.942 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T07:58:21.942 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T07:58:21.942 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T07:58:21.943 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server
2023-05-23T07:58:22.063 INFO:teuthology.dispatcher.supervisor:Running job 7283465
2023-05-23T07:58:22.063 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/bin/teuthology -v --owner mchangir --archive /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283465 --name mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi --description fs:mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} -- /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283465/orig.config.yaml
2023-05-23T07:58:22.067 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283465
2023-05-23T07:58:22.068 INFO:teuthology.dispatcher.supervisor:Job PID: 23112
2023-05-23T07:58:22.069 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-23T19:59:41.983 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-23T19:59:42.119 INFO:teuthology.kill:Killing Pids: {23112}
2023-05-23T19:59:43.105 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-05-23T19:59:43.106 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-05-23T19:59:43.106 INFO:teuthology.misc:Compressing logs...
2023-05-23T19:59:43.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-05-23T19:59:43.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com'
2023-05-23T19:59:43.761 INFO:teuthology.misc:Compressing logs...
2023-05-23T19:59:43.827 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T19:59:43.831 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T19:59:43.900 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T19:59:43.900 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now
2023-05-23T19:59:43.967 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T19:59:43.967 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now
2023-05-23T19:59:43.969 INFO:teuthology.misc:Compressing logs...
2023-05-23T19:59:53.315 INFO:teuthology.kill:No teuthology processes running
2023-05-23T19:59:54.111 INFO:teuthology.kill:Nuking machines: ['smithi170', 'smithi194']
2023-05-23T19:59:54.713 INFO:teuthology.kill:2023-05-23 19:59:54,713.713 INFO:teuthology.nuke:targets:
2023-05-23T19:59:54.714 INFO:teuthology.kill:  smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-05-23T19:59:54.714 INFO:teuthology.kill:  smithi194.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-05-23T19:59:54.715 INFO:teuthology.kill:2023-05-23 19:59:54,713.713 INFO:teuthology.nuke:Checking targets against current locks
2023-05-23T19:59:54.766 INFO:teuthology.kill:2023-05-23 19:59:54,765.765 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T19:59:54.789 INFO:teuthology.kill:2023-05-23 19:59:54,788.788 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T19:59:54.839 INFO:teuthology.kill:2023-05-23 19:59:54,839.839 INFO:teuthology.orchestra.console:Power off smithi170
2023-05-23T19:59:54.854 INFO:teuthology.kill:2023-05-23 19:59:54,854.854 INFO:teuthology.orchestra.console:Power off smithi194
2023-05-23T20:00:03.214 INFO:teuthology.kill:2023-05-23 20:00:03,213.213 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2023-05-23T20:00:03.221 INFO:teuthology.kill:2023-05-23 20:00:03,221.221 INFO:teuthology.orchestra.console:Power off for smithi194 completed
2023-05-23T20:00:03.390 INFO:teuthology.kill:2023-05-23 20:00:03,390.390 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com
2023-05-23T20:00:03.399 INFO:teuthology.kill:2023-05-23 20:00:03,399.399 INFO:teuthology.lock.ops:unlocked: smithi194.front.sepia.ceph.com
2023-05-23T20:02:03.776 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-23T20:02:03.881 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi170 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283833
2023-05-23T20:02:03.881 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi194 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283833