2022-03-20T06:00:00.152 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-20T06:00:00.280 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging 2022-03-20T06:00:00.283 INFO:teuthology.lock.ops:Updating [ubuntu@smithi106.front.sepia.ceph.com]: reset os type and version on server 2022-03-20T06:00:00.283 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2022-03-20T06:00:00.309 INFO:teuthology.lock.ops:Node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging is complete 2022-03-20T06:00:00.309 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi124.front.sepia.ceph.com' reimaging 2022-03-20T06:00:00.309 INFO:teuthology.lock.ops:Updating [ubuntu@smithi124.front.sepia.ceph.com]: reset os type and version on server 2022-03-20T06:00:00.310 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2022-03-20T06:00:00.328 INFO:teuthology.lock.ops:Node 'ubuntu@smithi124.front.sepia.ceph.com' reimaging is complete 2022-03-20T06:00:00.525 INFO:teuthology.provision.fog.smithi106:Scheduling deploy of centos 8.stream 2022-03-20T06:00:00.554 INFO:teuthology.provision.fog.smithi124:Scheduling deploy of centos 8.stream 2022-03-20T06:00:00.911 INFO:teuthology.orchestra.console:Power off smithi106 2022-03-20T06:00:00.927 INFO:teuthology.orchestra.console:Power off smithi124 2022-03-20T06:00:13.393 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2022-03-20T06:00:13.406 INFO:teuthology.orchestra.console:Power off for smithi124 completed 2022-03-20T06:00:13.494 INFO:teuthology.orchestra.console:Power on smithi106 2022-03-20T06:00:13.508 INFO:teuthology.orchestra.console:Power on smithi124 2022-03-20T06:00:17.767 INFO:teuthology.orchestra.console:Power on for smithi106 completed 2022-03-20T06:00:17.772 INFO:teuthology.orchestra.console:Power on for smithi124 completed 2022-03-20T06:00:17.869 INFO:teuthology.provision.fog.smithi106:Waiting for deploy to finish 2022-03-20T06:00:17.874 INFO:teuthology.provision.fog.smithi124:Waiting for deploy to finish 2022-03-20T06:04:23.178 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-20T06:04:33.552 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-20T06:04:49.025 INFO:teuthology.provision.fog.smithi106:Node is ready 2022-03-20T06:04:49.047 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106.front.sepia.ceph.com 2022-03-20T06:04:49.108 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106 smithi106.front.sepia.ceph.com smithi106 2022-03-20T06:04:49.503 INFO:teuthology.provision.fog.smithi106:Deploy complete! 2022-03-20T06:04:53.809 INFO:teuthology.provision.fog.smithi124:Node is ready 2022-03-20T06:04:53.830 INFO:teuthology.orchestra.run.smithi124.stdout:smithi124.front.sepia.ceph.com 2022-03-20T06:04:53.889 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.15.124 smithi124.front.sepia.ceph.com smithi124 2022-03-20T06:04:54.243 INFO:teuthology.provision.fog.smithi124:Deploy complete! 2022-03-20T06:04:54.411 INFO:teuthology.lock.ops:Checking smithi106.front.sepia.ceph.com 2022-03-20T06:04:54.412 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-20T06:04:54.446 INFO:teuthology.lock.ops:Checking smithi124.front.sepia.ceph.com 2022-03-20T06:04:54.447 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-20T06:04:54.466 INFO:teuthology.lock.ops:Updating [smithi106.front.sepia.ceph.com]: set os type and version on server 2022-03-20T06:04:54.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2022-03-20T06:04:54.662 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2022-03-20T06:04:54.722 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2022-03-20T06:04:54.722 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="8" 2022-03-20T06:04:54.722 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2022-03-20T06:04:54.723 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2022-03-20T06:04:54.723 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="8" 2022-03-20T06:04:54.723 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el8" 2022-03-20T06:04:54.723 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-20T06:04:54.723 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2022-03-20T06:04:54.724 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-20T06:04:54.724 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2022-03-20T06:04:54.724 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-20T06:04:54.724 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-20T06:04:54.725 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-20T06:04:54.726 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2022-03-20T06:04:54.746 INFO:teuthology.lock.ops:Updating [smithi124.front.sepia.ceph.com]: set os type and version on server 2022-03-20T06:04:54.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2022-03-20T06:04:54.939 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2022-03-20T06:04:54.995 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="CentOS Stream" 2022-03-20T06:04:54.996 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="8" 2022-03-20T06:04:54.996 INFO:teuthology.orchestra.run.smithi124.stdout:ID="centos" 2022-03-20T06:04:54.996 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE="rhel fedora" 2022-03-20T06:04:54.996 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="8" 2022-03-20T06:04:54.996 INFO:teuthology.orchestra.run.smithi124.stdout:PLATFORM_ID="platform:el8" 2022-03-20T06:04:54.997 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-20T06:04:54.997 INFO:teuthology.orchestra.run.smithi124.stdout:ANSI_COLOR="0;31" 2022-03-20T06:04:54.997 INFO:teuthology.orchestra.run.smithi124.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-20T06:04:54.997 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://centos.org/" 2022-03-20T06:04:54.998 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-20T06:04:54.998 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-20T06:04:54.998 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-20T06:04:54.999 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2022-03-20T06:04:55.101 INFO:teuthology.dispatcher.supervisor:Running job 6747586 2022-03-20T06:04:55.102 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747586 --name vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} -- /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747586/orig.config.yaml 2022-03-20T06:04:55.105 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747586 2022-03-20T06:04:55.107 INFO:teuthology.dispatcher.supervisor:Job PID: 22056 2022-03-20T06:04:55.107 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-20T12:35:23.087 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-20T12:35:23.308 INFO:teuthology.kill:Killing Pids: {22056} 2022-03-20T12:35:24.490 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-03-20T12:35:24.491 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-03-20T12:35:24.492 INFO:teuthology.misc:Compressing logs... 2022-03-20T12:35:24.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2022-03-20T12:35:29.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2022-03-20T12:35:29.946 INFO:teuthology.misc:Compressing logs... 2022-03-20T12:35:30.218 INFO:teuthology.misc:Compressing logs... 2022-03-20T12:44:29.996 INFO:teuthology.kill:No teuthology processes running 2022-03-20T12:44:31.199 INFO:teuthology.kill:Nuking machines: ['smithi106', 'smithi124'] 2022-03-20T12:44:32.150 INFO:teuthology.kill:2022-03-20 12:44:32,149.149 INFO:teuthology.nuke:targets: 2022-03-20T12:44:32.150 INFO:teuthology.kill: smithi106.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgqSaLDmEGJrgIbocvMZK03mxwYZ9uhl4VLRB2D/DgESc3FOV7ekwaCAF6iY3/u56ZjnoSVBR6g7UeMzbNf71wnPk6oETiZDTRZexbWBHl06wha1k7ZHOmHxH1c6RcR9qyaLVBDnSTQ39pEJx7dqW8rqaoiYmUntc4N3pcGFOMOo6SPIr7bNLbZKmqojDLS4MPosXvmWjR7/EIDcNd2m8bN1uDq/SCKBqG4e8lvEjsZzZPoGRIgPnYu1zrgdRVlaWUstpDZnzmLQheuR02+IdpP7NdJLlz7I7Vy3PkECCjf2tf43E5jI4KMko6rLpqB0OU+Qm4OfDHGQdsRQo3sREmDTCDjyzQrzAxchlm62hM8OFPXMIQvbg0djqHIpHxiWC0EWcORq0MKcom1xNVZrm/9z1kL66NbXW64m6S0youo9iJ0Yc3ZM5vEeOGDM+9tyyrU2J4k7a7AhVp2tdrDSa5AkrZ8MSrs+bf2Ou8oM02P3TahnzRCVBu+4eidJv3bLs= 2022-03-20T12:44:32.151 INFO:teuthology.kill: smithi124.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ/gzBQMD6gxRpjwjtNAqd9QBY2tdqKhM3SrdF3MQaNfcDM4++0dHM352VFuC2k2SA/NjkTunpFmnvv3vBs0DwxTR0Mu/W6YSDm0VraEnQINdvGwzn2/WhXpy6B1xhODy5eDKFNQfwtEfUDpV8fQmOrdNxYb3XV05igcj2elRIJOQIiWnq4Ihmrvf12TLF1ibVGpUzeD7BUWnwCO8IrBiuFtSt+460VNNavpzslqz9VipcnuFLGCvq1mJkayIXRUv6ouekV1r3oeg6BvJ4etL5t3x1xMk3zMTs9vSRmPyr6ZJlXUDuH/KeIjtLueLEs5KX1C13XXhgTLGscnZcAyNPiB0mhZW2T3+kQ0bVd6XgRi60H6EhUHeFn+Xedsg1xJ6N0gU8dofgzbVzvyIvrA8i9Eucqzof7WLbkEupp1RC6/epgvlBgRvxVBhcD0EWUiDArGlrrJTOwzTIg4Q8mSX+M/kipltMOetPIVYn3dyKHG6JpfDNJStnW2PjvTf+yec= 2022-03-20T12:44:32.175 INFO:teuthology.kill:2022-03-20 12:44:32,174.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-20T12:44:32.177 INFO:teuthology.kill:2022-03-20 12:44:32,177.177 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-20T12:44:32.259 INFO:teuthology.kill:2022-03-20 12:44:32,259.259 INFO:teuthology.orchestra.console:Power off smithi124 2022-03-20T12:44:32.273 INFO:teuthology.kill:2022-03-20 12:44:32,273.273 INFO:teuthology.orchestra.console:Power off smithi106 2022-03-20T12:44:44.750 INFO:teuthology.kill:2022-03-20 12:44:44,750.750 INFO:teuthology.orchestra.console:Power off for smithi124 completed 2022-03-20T12:44:44.755 INFO:teuthology.kill:2022-03-20 12:44:44,755.755 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2022-03-20T12:44:44.895 INFO:teuthology.kill:2022-03-20 12:44:44,894.894 INFO:teuthology.lock.ops:unlocked: smithi124.front.sepia.ceph.com 2022-03-20T12:44:44.898 INFO:teuthology.kill:2022-03-20 12:44:44,898.898 INFO:teuthology.lock.ops:unlocked: smithi106.front.sepia.ceph.com 2022-03-20T12:46:45.185 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-20T12:46:45.295 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi106 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747780 2022-03-20T12:46:45.295 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi124 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-03-20_02:16:37-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6747780