2023-05-23T07:54:56.137 INFO:root:teuthology version: 0.0.1.dev105+g6fb4f0c
2023-05-23T07:54:56.290 INFO:teuthology.lock.ops:Start node 'smithi121.front.sepia.ceph.com' reimaging
2023-05-23T07:54:56.291 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:54:56.291 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2023-05-23T07:54:56.319 INFO:teuthology.lock.ops:Node 'smithi121.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:54:56.319 INFO:teuthology.lock.ops:Start node 'smithi156.front.sepia.ceph.com' reimaging
2023-05-23T07:54:56.319 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:54:56.320 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server
2023-05-23T07:54:56.344 INFO:teuthology.lock.ops:Node 'smithi156.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:54:56.479 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of rhel 8.6
2023-05-23T07:54:56.499 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of rhel 8.6
2023-05-23T07:54:56.690 INFO:teuthology.orchestra.console:Power off smithi121
2023-05-23T07:54:56.732 INFO:teuthology.orchestra.console:Power off smithi156
2023-05-23T07:55:05.067 INFO:teuthology.orchestra.console:Power off for smithi121 completed
2023-05-23T07:55:05.091 INFO:teuthology.orchestra.console:Power off for smithi156 completed
2023-05-23T07:55:05.168 INFO:teuthology.orchestra.console:Power on smithi121
2023-05-23T07:55:05.199 INFO:teuthology.orchestra.console:Power on smithi156
2023-05-23T07:55:09.456 INFO:teuthology.orchestra.console:Power on for smithi156 completed
2023-05-23T07:55:09.557 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish
2023-05-23T07:55:13.577 INFO:teuthology.orchestra.console:Power on for smithi121 completed
2023-05-23T07:55:13.678 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish
2023-05-23T07:57:58.448 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-05-23T07:58:02.512 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-05-23T07:58:07.504 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-05-23T07:58:11.568 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-05-23T07:58:16.560 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-05-23T07:58:20.624 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-05-23T07:58:25.616 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-05-23T07:58:29.680 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-05-23T07:58:34.672 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-05-23T07:58:38.736 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.121
2023-05-23T07:58:43.728 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-05-23T07:59:44.790 WARNING:teuthology.provision.fog:timed out
2023-05-23T07:59:49.736 WARNING:teuthology.provision.fog:timed out
2023-05-23T07:59:51.125 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T07:59:51.416 INFO:teuthology.provision.fog.smithi121:Node is ready
2023-05-23T07:59:51.434 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121.front.sepia.ceph.com
2023-05-23T07:59:51.492 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121
2023-05-23T07:59:51.999 INFO:teuthology.provision.fog.smithi121:Deploy complete!
2023-05-23T07:59:56.069 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T07:59:56.364 INFO:teuthology.provision.fog.smithi156:Node is ready
2023-05-23T07:59:56.382 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156.front.sepia.ceph.com
2023-05-23T07:59:56.440 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.15.156 smithi156.front.sepia.ceph.com smithi156
2023-05-23T07:59:56.790 INFO:teuthology.provision.fog.smithi156:Deploy complete!
2023-05-23T07:59:56.952 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com
2023-05-23T07:59:56.953 INFO:teuthology.lock.ops:Checking smithi156.front.sepia.ceph.com
2023-05-23T07:59:56.955 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server
2023-05-23T07:59:56.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com'
2023-05-23T07:59:57.149 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64
2023-05-23T07:59:57.207 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T07:59:57.207 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T07:59:57.207 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel"
2023-05-23T07:59:57.207 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora"
2023-05-23T07:59:57.207 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.6"
2023-05-23T07:59:57.208 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8"
2023-05-23T07:59:57.208 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T07:59:57.208 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31"
2023-05-23T07:59:57.208 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T07:59:57.208 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T07:59:57.208 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T07:59:57.208 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T07:59:57.209 INFO:teuthology.orchestra.run.smithi121.stdout:
2023-05-23T07:59:57.209 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T07:59:57.209 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T07:59:57.209 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T07:59:57.209 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T07:59:57.210 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2023-05-23T07:59:57.235 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: set os type and version on server
2023-05-23T07:59:57.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com'
2023-05-23T07:59:57.419 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64
2023-05-23T07:59:57.474 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T07:59:57.475 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T07:59:57.475 INFO:teuthology.orchestra.run.smithi156.stdout:ID="rhel"
2023-05-23T07:59:57.475 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="fedora"
2023-05-23T07:59:57.475 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8.6"
2023-05-23T07:59:57.475 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8"
2023-05-23T07:59:57.475 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T07:59:57.476 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31"
2023-05-23T07:59:57.476 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T07:59:57.476 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T07:59:57.476 INFO:teuthology.orchestra.run.smithi156.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T07:59:57.476 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T07:59:57.476 INFO:teuthology.orchestra.run.smithi156.stdout:
2023-05-23T07:59:57.477 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T07:59:57.477 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T07:59:57.477 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T07:59:57.477 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T07:59:57.477 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server
2023-05-23T07:59:57.591 INFO:teuthology.dispatcher.supervisor:Running job 7283466
2023-05-23T07:59:57.591 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/bin/teuthology -v --owner mchangir --archive /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283466 --name mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi --description fs:mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} -- /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283466/orig.config.yaml
2023-05-23T07:59:57.594 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283466
2023-05-23T07:59:57.595 INFO:teuthology.dispatcher.supervisor:Job PID: 26332
2023-05-23T07:59:57.595 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-23T20:01:25.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-23T20:01:25.225 INFO:teuthology.kill:Killing Pids: {26332}
2023-05-23T20:01:26.042 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-05-23T20:01:26.042 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-05-23T20:01:26.043 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:01:26.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com'
2023-05-23T20:01:26.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com'
2023-05-23T20:01:26.687 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:01:26.757 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T20:01:26.759 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T20:01:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T20:01:26.829 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now
2023-05-23T20:01:26.896 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T20:01:26.897 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now
2023-05-23T20:01:26.899 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:01:34.433 INFO:teuthology.kill:No teuthology processes running
2023-05-23T20:01:35.254 INFO:teuthology.kill:Nuking machines: ['smithi121', 'smithi156']
2023-05-23T20:01:35.847 INFO:teuthology.kill:2023-05-23 20:01:35,844.844 INFO:teuthology.nuke:targets:
2023-05-23T20:01:35.848 INFO:teuthology.kill:  smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=
2023-05-23T20:01:35.848 INFO:teuthology.kill:  smithi156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-05-23T20:01:35.848 INFO:teuthology.kill:2023-05-23 20:01:35,844.844 INFO:teuthology.nuke:Checking targets against current locks
2023-05-23T20:01:35.904 INFO:teuthology.kill:2023-05-23 20:01:35,904.904 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T20:01:35.927 INFO:teuthology.kill:2023-05-23 20:01:35,927.927 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T20:01:35.983 INFO:teuthology.kill:2023-05-23 20:01:35,982.982 INFO:teuthology.orchestra.console:Power off smithi121
2023-05-23T20:01:36.026 INFO:teuthology.kill:2023-05-23 20:01:36,026.026 INFO:teuthology.orchestra.console:Power off smithi156
2023-05-23T20:01:44.387 INFO:teuthology.kill:2023-05-23 20:01:44,387.387 INFO:teuthology.orchestra.console:Power off for smithi156 completed
2023-05-23T20:01:44.550 INFO:teuthology.kill:2023-05-23 20:01:44,549.549 INFO:teuthology.lock.ops:unlocked: smithi156.front.sepia.ceph.com
2023-05-23T20:01:48.473 INFO:teuthology.kill:2023-05-23 20:01:48,473.473 INFO:teuthology.orchestra.console:Power off for smithi121 completed
2023-05-23T20:01:48.633 INFO:teuthology.kill:2023-05-23 20:01:48,632.632 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com
2023-05-23T20:03:49.034 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-23T20:03:49.144 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi121 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283835
2023-05-23T20:03:49.145 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi156 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283835