2023-05-23T07:56:58.181 INFO:root:teuthology version: 0.0.1.dev105+g6fb4f0c
2023-05-23T07:56:58.341 INFO:teuthology.lock.ops:Start node 'smithi017.front.sepia.ceph.com' reimaging
2023-05-23T07:56:58.342 INFO:teuthology.lock.ops:Updating [smithi017.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:56:58.343 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server
2023-05-23T07:56:58.360 INFO:teuthology.lock.ops:Node 'smithi017.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:56:58.361 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging
2023-05-23T07:56:58.361 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:56:58.361 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2023-05-23T07:56:58.390 INFO:teuthology.lock.ops:Node 'smithi037.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:56:58.528 INFO:teuthology.provision.fog.smithi017:Scheduling deploy of rhel 8.6
2023-05-23T07:56:58.548 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of rhel 8.6
2023-05-23T07:56:58.770 INFO:teuthology.orchestra.console:Power off smithi017
2023-05-23T07:56:58.785 INFO:teuthology.orchestra.console:Power off smithi037
2023-05-23T07:56:58.918 INFO:teuthology.orchestra.console:Power off for smithi017 completed
2023-05-23T07:56:58.927 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2023-05-23T07:56:59.019 INFO:teuthology.orchestra.console:Power on smithi017
2023-05-23T07:56:59.028 INFO:teuthology.orchestra.console:Power on smithi037
2023-05-23T07:57:07.390 INFO:teuthology.orchestra.console:Power on for smithi017 completed
2023-05-23T07:57:07.396 INFO:teuthology.orchestra.console:Power on for smithi037 completed
2023-05-23T07:57:07.491 INFO:teuthology.provision.fog.smithi017:Waiting for deploy to finish
2023-05-23T07:57:07.497 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish
2023-05-23T07:59:41.520 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.17
2023-05-23T07:59:41.521 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-05-23T07:59:50.576 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.17
2023-05-23T07:59:50.577 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-05-23T07:59:59.632 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.17
2023-05-23T07:59:59.633 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-05-23T08:00:08.688 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.17
2023-05-23T08:00:08.720 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-05-23T08:00:17.780 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.17
2023-05-23T08:00:26.864 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.17
2023-05-23T08:01:14.761 WARNING:teuthology.provision.fog:timed out
2023-05-23T08:01:21.122 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T08:01:21.429 INFO:teuthology.provision.fog.smithi037:Node is ready
2023-05-23T08:01:21.448 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com
2023-05-23T08:01:21.506 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037
2023-05-23T08:01:21.872 INFO:teuthology.provision.fog.smithi037:Deploy complete!
2023-05-23T08:01:32.880 WARNING:teuthology.provision.fog:timed out
2023-05-23T08:01:39.205 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T08:01:39.516 INFO:teuthology.provision.fog.smithi017:Node is ready
2023-05-23T08:01:39.534 INFO:teuthology.orchestra.run.smithi017.stdout:smithi017.front.sepia.ceph.com
2023-05-23T08:01:39.593 INFO:teuthology.orchestra.run.smithi017.stdout:172.21.15.17 smithi017.front.sepia.ceph.com smithi017
2023-05-23T08:01:39.938 INFO:teuthology.provision.fog.smithi017:Deploy complete!
2023-05-23T08:01:40.091 INFO:teuthology.lock.ops:Checking smithi017.front.sepia.ceph.com
2023-05-23T08:01:40.092 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-23T08:01:40.121 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com
2023-05-23T08:01:40.121 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-23T08:01:40.148 INFO:teuthology.lock.ops:Updating [smithi017.front.sepia.ceph.com]: set os type and version on server
2023-05-23T08:01:40.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com'
2023-05-23T08:01:40.337 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64
2023-05-23T08:01:40.392 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T08:01:40.393 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T08:01:40.393 INFO:teuthology.orchestra.run.smithi017.stdout:ID="rhel"
2023-05-23T08:01:40.393 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="fedora"
2023-05-23T08:01:40.393 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="8.6"
2023-05-23T08:01:40.393 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el8"
2023-05-23T08:01:40.393 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T08:01:40.393 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31"
2023-05-23T08:01:40.394 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T08:01:40.394 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T08:01:40.394 INFO:teuthology.orchestra.run.smithi017.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T08:01:40.394 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T08:01:40.394 INFO:teuthology.orchestra.run.smithi017.stdout:
2023-05-23T08:01:40.394 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T08:01:40.394 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T08:01:40.395 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T08:01:40.395 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T08:01:40.395 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server
2023-05-23T08:01:40.421 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server
2023-05-23T08:01:40.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-05-23T08:01:40.675 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64
2023-05-23T08:01:40.733 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T08:01:40.733 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T08:01:40.734 INFO:teuthology.orchestra.run.smithi037.stdout:ID="rhel"
2023-05-23T08:01:40.734 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="fedora"
2023-05-23T08:01:40.734 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8.6"
2023-05-23T08:01:40.734 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8"
2023-05-23T08:01:40.734 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T08:01:40.734 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2023-05-23T08:01:40.734 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T08:01:40.735 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T08:01:40.735 INFO:teuthology.orchestra.run.smithi037.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T08:01:40.735 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T08:01:40.735 INFO:teuthology.orchestra.run.smithi037.stdout:
2023-05-23T08:01:40.735 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T08:01:40.735 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T08:01:40.735 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T08:01:40.736 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T08:01:40.736 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2023-05-23T08:01:40.852 INFO:teuthology.dispatcher.supervisor:Running job 7283470
2023-05-23T08:01:40.853 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/bin/teuthology -v --owner mchangir --archive /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283470 --name mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi --description fs:mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} -- /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283470/orig.config.yaml
2023-05-23T08:01:40.856 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283470
2023-05-23T08:01:40.856 INFO:teuthology.dispatcher.supervisor:Job PID: 28445
2023-05-23T08:01:40.857 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-23T20:03:05.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-23T20:03:05.967 INFO:teuthology.kill:Killing Pids: {28445}
2023-05-23T20:03:06.783 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-05-23T20:03:06.784 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-05-23T20:03:06.784 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:03:06.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com'
2023-05-23T20:03:07.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-05-23T20:03:07.490 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:03:07.557 INFO:teuthology.orchestra.run.smithi017.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T20:03:07.560 INFO:teuthology.orchestra.run.smithi037.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T20:03:07.630 INFO:teuthology.orchestra.run.smithi017.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T20:03:07.631 INFO:teuthology.orchestra.run.smithi017.stderr:tar: Error is not recoverable: exiting now
2023-05-23T20:03:07.698 INFO:teuthology.orchestra.run.smithi037.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T20:03:07.699 INFO:teuthology.orchestra.run.smithi037.stderr:tar: Error is not recoverable: exiting now
2023-05-23T20:03:07.701 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:03:15.509 INFO:teuthology.kill:No teuthology processes running
2023-05-23T20:03:16.300 INFO:teuthology.kill:Nuking machines: ['smithi017', 'smithi037']
2023-05-23T20:03:16.912 INFO:teuthology.kill:2023-05-23 20:03:16,912.912 INFO:teuthology.nuke:targets:
2023-05-23T20:03:16.913 INFO:teuthology.kill:  smithi017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-05-23T20:03:16.914 INFO:teuthology.kill:  smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-05-23T20:03:16.914 INFO:teuthology.kill:2023-05-23 20:03:16,912.912 INFO:teuthology.nuke:Checking targets against current locks
2023-05-23T20:03:16.972 INFO:teuthology.kill:2023-05-23 20:03:16,971.971 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T20:03:16.998 INFO:teuthology.kill:2023-05-23 20:03:16,998.998 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T20:03:17.048 INFO:teuthology.kill:2023-05-23 20:03:17,047.047 INFO:teuthology.orchestra.console:Power off smithi017
2023-05-23T20:03:17.081 INFO:teuthology.kill:2023-05-23 20:03:17,081.081 INFO:teuthology.orchestra.console:Power off smithi037
2023-05-23T20:03:25.414 INFO:teuthology.kill:2023-05-23 20:03:25,414.414 INFO:teuthology.orchestra.console:Power off for smithi017 completed
2023-05-23T20:03:25.456 INFO:teuthology.kill:2023-05-23 20:03:25,455.455 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2023-05-23T20:03:25.560 INFO:teuthology.kill:2023-05-23 20:03:25,560.560 INFO:teuthology.lock.ops:unlocked: smithi017.front.sepia.ceph.com
2023-05-23T20:03:25.611 INFO:teuthology.kill:2023-05-23 20:03:25,610.610 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com
2023-05-23T20:05:25.964 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-23T20:05:26.071 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi017 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283837
2023-05-23T20:05:26.071 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi037 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283837