2023-05-23T07:58:19.629 INFO:root:teuthology version: 0.0.1.dev105+g6fb4f0c
2023-05-23T07:58:19.778 INFO:teuthology.lock.ops:Start node 'smithi035.front.sepia.ceph.com' reimaging
2023-05-23T07:58:19.780 INFO:teuthology.lock.ops:Updating [smithi035.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:58:19.780 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server
2023-05-23T07:58:19.807 INFO:teuthology.lock.ops:Node 'smithi035.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:58:19.807 INFO:teuthology.lock.ops:Start node 'smithi123.front.sepia.ceph.com' reimaging
2023-05-23T07:58:19.808 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: reset os type and version on server
2023-05-23T07:58:19.808 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2023-05-23T07:58:19.831 INFO:teuthology.lock.ops:Node 'smithi123.front.sepia.ceph.com' reimaging is complete
2023-05-23T07:58:19.964 INFO:teuthology.provision.fog.smithi035:Scheduling deploy of rhel 8.6
2023-05-23T07:58:19.987 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of rhel 8.6
2023-05-23T07:58:20.219 INFO:teuthology.orchestra.console:Power off smithi123
2023-05-23T07:58:20.236 INFO:teuthology.orchestra.console:Power off smithi035
2023-05-23T07:58:28.597 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2023-05-23T07:58:28.603 INFO:teuthology.orchestra.console:Power off for smithi035 completed
2023-05-23T07:58:28.699 INFO:teuthology.orchestra.console:Power on smithi123
2023-05-23T07:58:28.708 INFO:teuthology.orchestra.console:Power on smithi035
2023-05-23T07:58:32.958 INFO:teuthology.orchestra.console:Power on for smithi035 completed
2023-05-23T07:58:33.059 INFO:teuthology.provision.fog.smithi035:Waiting for deploy to finish
2023-05-23T07:58:37.062 INFO:teuthology.orchestra.console:Power on for smithi123 completed
2023-05-23T07:58:37.163 INFO:teuthology.provision.fog.smithi123:Waiting for deploy to finish
2023-05-23T08:01:07.024 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.35
2023-05-23T08:01:11.088 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.123
2023-05-23T08:01:16.080 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.35
2023-05-23T08:01:20.144 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.123
2023-05-23T08:01:25.136 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.35
2023-05-23T08:01:29.200 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.123
2023-05-23T08:01:34.192 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.35
2023-05-23T08:01:38.256 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.123
2023-05-23T08:01:43.248 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.35
2023-05-23T08:01:47.312 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.123
2023-05-23T08:01:56.368 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.123
2023-05-23T08:02:05.456 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.123
2023-05-23T08:02:20.908 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T08:02:21.214 INFO:teuthology.provision.fog.smithi035:Node is ready
2023-05-23T08:02:21.232 INFO:teuthology.orchestra.run.smithi035.stdout:smithi022
2023-05-23T08:02:21.290 INFO:teuthology.orchestra.run.smithi035.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-05-23T08:02:21.666 INFO:teuthology.provision.fog.smithi035:Deploy complete!
2023-05-23T08:03:11.509 WARNING:teuthology.provision.fog:timed out
2023-05-23T08:03:17.889 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-23T08:03:18.215 INFO:teuthology.provision.fog.smithi123:Node is ready
2023-05-23T08:03:18.234 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123
2023-05-23T08:03:18.296 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123 smithi123.front.sepia.ceph.com smithi123
2023-05-23T08:03:18.678 INFO:teuthology.provision.fog.smithi123:Deploy complete!
2023-05-23T08:03:18.832 INFO:teuthology.lock.ops:Checking smithi035.front.sepia.ceph.com
2023-05-23T08:03:18.833 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-23T08:03:18.857 INFO:teuthology.lock.ops:Checking smithi123.front.sepia.ceph.com
2023-05-23T08:03:18.858 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-23T08:03:18.875 INFO:teuthology.lock.ops:Updating [smithi035.front.sepia.ceph.com]: set os type and version on server
2023-05-23T08:03:18.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com'
2023-05-23T08:03:19.130 INFO:teuthology.orchestra.run.smithi035.stdout:x86_64
2023-05-23T08:03:19.187 INFO:teuthology.orchestra.run.smithi035.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T08:03:19.188 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T08:03:19.188 INFO:teuthology.orchestra.run.smithi035.stdout:ID="rhel"
2023-05-23T08:03:19.188 INFO:teuthology.orchestra.run.smithi035.stdout:ID_LIKE="fedora"
2023-05-23T08:03:19.188 INFO:teuthology.orchestra.run.smithi035.stdout:VERSION_ID="8.6"
2023-05-23T08:03:19.188 INFO:teuthology.orchestra.run.smithi035.stdout:PLATFORM_ID="platform:el8"
2023-05-23T08:03:19.188 INFO:teuthology.orchestra.run.smithi035.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T08:03:19.189 INFO:teuthology.orchestra.run.smithi035.stdout:ANSI_COLOR="0;31"
2023-05-23T08:03:19.189 INFO:teuthology.orchestra.run.smithi035.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T08:03:19.191 INFO:teuthology.orchestra.run.smithi035.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T08:03:19.191 INFO:teuthology.orchestra.run.smithi035.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T08:03:19.191 INFO:teuthology.orchestra.run.smithi035.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T08:03:19.191 INFO:teuthology.orchestra.run.smithi035.stdout:
2023-05-23T08:03:19.192 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T08:03:19.192 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T08:03:19.192 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T08:03:19.192 INFO:teuthology.orchestra.run.smithi035.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T08:03:19.193 INFO:teuthology.lock.ops:Updating smithi035.front.sepia.ceph.com on lock server
2023-05-23T08:03:19.216 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: set os type and version on server
2023-05-23T08:03:19.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com'
2023-05-23T08:03:19.413 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64
2023-05-23T08:03:19.473 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Red Hat Enterprise Linux"
2023-05-23T08:03:19.473 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8.6 (Ootpa)"
2023-05-23T08:03:19.473 INFO:teuthology.orchestra.run.smithi123.stdout:ID="rhel"
2023-05-23T08:03:19.473 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="fedora"
2023-05-23T08:03:19.473 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8.6"
2023-05-23T08:03:19.474 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8"
2023-05-23T08:03:19.474 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-05-23T08:03:19.474 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31"
2023-05-23T08:03:19.474 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-05-23T08:03:19.474 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.redhat.com/"
2023-05-23T08:03:19.474 INFO:teuthology.orchestra.run.smithi123.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-05-23T08:03:19.474 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-23T08:03:19.475 INFO:teuthology.orchestra.run.smithi123.stdout:
2023-05-23T08:03:19.475 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-23T08:03:19.475 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-05-23T08:03:19.475 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-05-23T08:03:19.475 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-05-23T08:03:19.476 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2023-05-23T08:03:19.596 INFO:teuthology.dispatcher.supervisor:Running job 7283473
2023-05-23T08:03:19.597 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6fb4f0c910de3902d27909a642e8d50d731dd28c/virtualenv/bin/teuthology -v --owner mchangir --archive /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283473 --name mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi --description fs:mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} -- /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283473/orig.config.yaml
2023-05-23T08:03:19.600 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/mchangir-2023-05-23_07:50:52-fs:mixed-clients-wip-vshankar-testing-20230509.090020-1-distro-default-smithi/7283473
2023-05-23T08:03:19.600 INFO:teuthology.dispatcher.supervisor:Job PID: 32102
2023-05-23T08:03:19.601 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-23T20:04:36.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-23T20:04:36.378 INFO:teuthology.kill:Killing Pids: {32102}
2023-05-23T20:04:37.259 INFO:teuthology.task.internal:roles: ubuntu@smithi035.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-05-23T20:04:37.261 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-05-23T20:04:37.261 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:04:37.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi035.front.sepia.ceph.com'
2023-05-23T20:04:37.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com'
2023-05-23T20:04:37.964 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:04:38.032 INFO:teuthology.orchestra.run.smithi035.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T20:04:38.033 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-05-23T20:04:38.103 INFO:teuthology.orchestra.run.smithi035.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T20:04:38.104 INFO:teuthology.orchestra.run.smithi035.stderr:tar: Error is not recoverable: exiting now
2023-05-23T20:04:38.170 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-05-23T20:04:38.170 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now
2023-05-23T20:04:38.172 INFO:teuthology.misc:Compressing logs...
2023-05-23T20:04:46.242 INFO:teuthology.kill:No teuthology processes running
2023-05-23T20:04:47.104 INFO:teuthology.kill:Nuking machines: ['smithi035', 'smithi123']
2023-05-23T20:04:47.718 INFO:teuthology.kill:2023-05-23 20:04:47,718.718 INFO:teuthology.nuke:targets:
2023-05-23T20:04:47.719 INFO:teuthology.kill:  smithi035.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=
2023-05-23T20:04:47.719 INFO:teuthology.kill:  smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=
2023-05-23T20:04:47.719 INFO:teuthology.kill:2023-05-23 20:04:47,718.718 INFO:teuthology.nuke:Checking targets against current locks
2023-05-23T20:04:47.774 INFO:teuthology.kill:2023-05-23 20:04:47,774.774 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T20:04:47.799 INFO:teuthology.kill:2023-05-23 20:04:47,799.799 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-23T20:04:47.856 INFO:teuthology.kill:2023-05-23 20:04:47,856.856 INFO:teuthology.orchestra.console:Power off smithi035
2023-05-23T20:04:47.874 INFO:teuthology.kill:2023-05-23 20:04:47,874.874 INFO:teuthology.orchestra.console:Power off smithi123
2023-05-23T20:04:56.226 INFO:teuthology.kill:2023-05-23 20:04:56,226.226 INFO:teuthology.orchestra.console:Power off for smithi035 completed
2023-05-23T20:04:56.388 INFO:teuthology.kill:2023-05-23 20:04:56,388.388 INFO:teuthology.lock.ops:unlocked: smithi035.front.sepia.ceph.com
2023-05-23T20:05:00.347 INFO:teuthology.kill:2023-05-23 20:05:00,347.347 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2023-05-23T20:05:00.516 INFO:teuthology.kill:2023-05-23 20:05:00,516.516 INFO:teuthology.lock.ops:unlocked: smithi123.front.sepia.ceph.com
2023-05-23T20:07:00.868 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-23T20:07:00.976 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi035 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283840
2023-05-23T20:07:00.976 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi123 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-23_14:47:16-rbd-wip-yuri4-testing-2023-05-22-1631-quincy-distro-default-smithi/7283843