2022-03-23T13:54:13.213 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-23T13:54:13.330 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2022-03-23T13:54:13.331 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2022-03-23T13:54:13.331 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2022-03-23T13:54:13.354 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2022-03-23T13:54:13.354 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi197.front.sepia.ceph.com' reimaging 2022-03-23T13:54:13.355 INFO:teuthology.lock.ops:Updating [ubuntu@smithi197.front.sepia.ceph.com]: reset os type and version on server 2022-03-23T13:54:13.355 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2022-03-23T13:54:13.374 INFO:teuthology.lock.ops:Node 'ubuntu@smithi197.front.sepia.ceph.com' reimaging is complete 2022-03-23T13:54:13.576 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of rhel 8.4 2022-03-23T13:54:13.586 INFO:teuthology.provision.fog.smithi197:Scheduling deploy of rhel 8.4 2022-03-23T13:54:13.872 INFO:teuthology.orchestra.console:Power off smithi197 2022-03-23T13:54:13.886 INFO:teuthology.orchestra.console:Power off smithi084 2022-03-23T13:54:14.016 INFO:teuthology.orchestra.console:Power off for smithi197 completed 2022-03-23T13:54:14.025 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2022-03-23T13:54:14.118 INFO:teuthology.orchestra.console:Power on smithi197 2022-03-23T13:54:14.126 INFO:teuthology.orchestra.console:Power on smithi084 2022-03-23T13:54:18.378 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2022-03-23T13:54:18.479 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2022-03-23T13:54:22.486 INFO:teuthology.orchestra.console:Power on for smithi197 completed 2022-03-23T13:54:22.587 INFO:teuthology.provision.fog.smithi197:Waiting for deploy to finish 2022-03-23T13:58:24.523 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-23T13:58:44.328 INFO:teuthology.provision.fog.smithi084:Node is ready 2022-03-23T13:58:44.347 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2022-03-23T13:58:44.404 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2022-03-23T13:58:44.748 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2022-03-23T13:59:00.250 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-23T13:59:00.499 INFO:teuthology.provision.fog.smithi197:Node is ready 2022-03-23T13:59:00.514 INFO:teuthology.orchestra.run.smithi197.stdout:smithi197.front.sepia.ceph.com 2022-03-23T13:59:00.570 INFO:teuthology.orchestra.run.smithi197.stdout:172.21.15.197 smithi197.front.sepia.ceph.com smithi197 2022-03-23T13:59:00.884 INFO:teuthology.provision.fog.smithi197:Deploy complete! 2022-03-23T13:59:01.035 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2022-03-23T13:59:01.036 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-23T13:59:01.054 INFO:teuthology.lock.ops:Checking smithi197.front.sepia.ceph.com 2022-03-23T13:59:01.055 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-23T13:59:01.076 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2022-03-23T13:59:01.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-03-23T13:59:01.308 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2022-03-23T13:59:01.362 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Red Hat Enterprise Linux" 2022-03-23T13:59:01.363 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8.4 (Ootpa)" 2022-03-23T13:59:01.363 INFO:teuthology.orchestra.run.smithi084.stdout:ID="rhel" 2022-03-23T13:59:01.363 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="fedora" 2022-03-23T13:59:01.363 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8.4" 2022-03-23T13:59:01.363 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2022-03-23T13:59:01.364 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-23T13:59:01.364 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2022-03-23T13:59:01.364 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-23T13:59:01.364 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.redhat.com/" 2022-03-23T13:59:01.364 INFO:teuthology.orchestra.run.smithi084.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-23T13:59:01.365 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-23T13:59:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: 2022-03-23T13:59:01.365 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-23T13:59:01.365 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-23T13:59:01.365 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-23T13:59:01.366 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-23T13:59:01.367 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2022-03-23T13:59:01.386 INFO:teuthology.lock.ops:Updating [smithi197.front.sepia.ceph.com]: set os type and version on server 2022-03-23T13:59:01.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-03-23T13:59:01.610 INFO:teuthology.orchestra.run.smithi197.stdout:x86_64 2022-03-23T13:59:01.664 INFO:teuthology.orchestra.run.smithi197.stdout:NAME="Red Hat Enterprise Linux" 2022-03-23T13:59:01.664 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION="8.4 (Ootpa)" 2022-03-23T13:59:01.664 INFO:teuthology.orchestra.run.smithi197.stdout:ID="rhel" 2022-03-23T13:59:01.664 INFO:teuthology.orchestra.run.smithi197.stdout:ID_LIKE="fedora" 2022-03-23T13:59:01.664 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION_ID="8.4" 2022-03-23T13:59:01.665 INFO:teuthology.orchestra.run.smithi197.stdout:PLATFORM_ID="platform:el8" 2022-03-23T13:59:01.665 INFO:teuthology.orchestra.run.smithi197.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-03-23T13:59:01.665 INFO:teuthology.orchestra.run.smithi197.stdout:ANSI_COLOR="0;31" 2022-03-23T13:59:01.665 INFO:teuthology.orchestra.run.smithi197.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-03-23T13:59:01.666 INFO:teuthology.orchestra.run.smithi197.stdout:HOME_URL="https://www.redhat.com/" 2022-03-23T13:59:01.666 INFO:teuthology.orchestra.run.smithi197.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-03-23T13:59:01.666 INFO:teuthology.orchestra.run.smithi197.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-23T13:59:01.666 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-03-23T13:59:01.666 INFO:teuthology.orchestra.run.smithi197.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-23T13:59:01.667 INFO:teuthology.orchestra.run.smithi197.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-03-23T13:59:01.667 INFO:teuthology.orchestra.run.smithi197.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-03-23T13:59:01.667 INFO:teuthology.orchestra.run.smithi197.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-03-23T13:59:01.668 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2022-03-23T13:59:01.757 INFO:teuthology.dispatcher.supervisor:Running job 6755000 2022-03-23T13:59:01.757 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-23_03:55:41-fs-wip-vshankar-fscrypt-20220322-143626-testing-default-smithi/6755000 --name vshankar-2022-03-23_03:55:41-fs-wip-vshankar-fscrypt-20220322-143626-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} -- /home/teuthworker/archive/vshankar-2022-03-23_03:55:41-fs-wip-vshankar-fscrypt-20220322-143626-testing-default-smithi/6755000/orig.config.yaml 2022-03-23T13:59:01.760 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-23_03:55:41-fs-wip-vshankar-fscrypt-20220322-143626-testing-default-smithi/6755000 2022-03-23T13:59:01.761 INFO:teuthology.dispatcher.supervisor:Job PID: 30278 2022-03-23T13:59:01.761 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-23T20:29:29.875 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-23T20:29:30.093 INFO:teuthology.kill:Killing Pids: {30278} 2022-03-23T20:29:31.368 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-03-23T20:29:31.369 INFO:teuthology.task.internal:roles: ubuntu@smithi197.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-03-23T20:29:31.370 INFO:teuthology.misc:Compressing logs... 2022-03-23T20:29:31.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-03-23T20:29:31.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-03-23T20:29:32.303 INFO:teuthology.misc:Compressing logs... 2022-03-23T20:29:32.776 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-03-23T20:29:33.317 INFO:teuthology.misc:Compressing logs... 2022-03-23T20:53:21.208 INFO:teuthology.kill:No teuthology processes running 2022-03-23T20:53:22.649 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi197'] 2022-03-23T20:53:23.732 INFO:teuthology.kill:2022-03-23 20:53:23,732.732 INFO:teuthology.nuke:targets: 2022-03-23T20:53:23.733 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaodRL4mTKfSUIZPlU3pqj9w7cBAasdnNyYO73v2s8G//xx8GgtbhqT5vc4dx+pGMygQdtPUJg2772cJ7CWv+cc+WZEtaTRszsVC+Ps4Btifd5qNjhxSonR86f8lYmMj2ojb9alnp21jYY0GA7YsKZuJ6bOETwM2NO73k3osdQfC+awueI42STNBQssMLu27nCAEHvUvlmCdgZu2nekuUHCq1WqiK+z4WmGLFiBT0p4mFtFqCL32K+K+8++CcKP862fqrThO0qI5p9BFz68OKna8FR/oQDMDkrY9iC6FJtQ1uSdvmGazSxchqX6hV0hDJEhrEsAo400B9/vLjxOSQpOLzR7f/BMc1ttuppv6vhvIlvxZp0DFahtYplJeC5x4btx7+U0JbiXAQPBRkGwsucKG5c/nX4SekfCxgnLwA+FWyj4GSB/AcVJLLNWefzJjQIYsnO2H4c2QJPOI+HDewR4DuMi8XWHN06g+/sf4DyHBgdc/33VUslXKu17WZ5SP8= 2022-03-23T20:53:23.734 INFO:teuthology.kill: smithi197.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3nxxxyQmB+qzbF+7kx+5EbowQaZunGyD1N9AwNAur1t515cPVMOPTq5HuX0Es6KT40Xemr67qX5IgqVb31WAtr+jd/q3awSnStEjYqftXvRUCUOe2GRDeq2qfSqrjp1hiyvqelojzxh/y8DI/5Y2WdEKD50Wu/LxPrG97vRziQJMxe2jbkMoGtxE20/OknwqgaKFmr42U95qaQ2P5nmNQd/DFy71rcytBrGK/cUU76o6lc+NPx3JH2J04QUyuaHRqrmFX/65urqNe7oZk8VSilyjAvrlzrPVKwcKyBNpQc1BeE9Ru21Yygzcl0SzOkK1HgLV2LqesEhDo1sqQzndN8rjmOJwF/j097x/mu7GUjC382rNSHKw8wNKGwbK6WMcJwAsl1Mtex8p9BFrqFMXZqxMBNRfbevJMWHUqnJRdjdjkJPdwNfw6cptcUuVif9kuOyrDlGQO3qITFozB836kpD0KRsJOsmFVxLbu1p+9CLxFYhfyVAE+5tJez7D9NcU= 2022-03-23T20:53:23.760 INFO:teuthology.kill:2022-03-23 20:53:23,760.760 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-23T20:53:23.764 INFO:teuthology.kill:2022-03-23 20:53:23,764.764 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-23T20:53:23.836 INFO:teuthology.kill:2022-03-23 20:53:23,836.836 INFO:teuthology.orchestra.console:Power off smithi197 2022-03-23T20:53:23.850 INFO:teuthology.kill:2022-03-23 20:53:23,850.850 INFO:teuthology.orchestra.console:Power off smithi084 2022-03-23T20:53:32.212 INFO:teuthology.kill:2022-03-23 20:53:32,211.211 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2022-03-23T20:53:32.360 INFO:teuthology.kill:2022-03-23 20:53:32,359.359 INFO:teuthology.lock.ops:unlocked: smithi084.front.sepia.ceph.com 2022-03-23T20:53:36.316 INFO:teuthology.kill:2022-03-23 20:53:36,316.316 INFO:teuthology.orchestra.console:Power off for smithi197 completed 2022-03-23T20:53:36.468 INFO:teuthology.kill:2022-03-23 20:53:36,468.468 INFO:teuthology.lock.ops:unlocked: smithi197.front.sepia.ceph.com 2022-03-23T20:55:36.786 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-23T20:55:36.899 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi084 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-23_20:43:24-orch-wip-yuri6-testing-2022-03-23-1009-pacific-distro-default-smithi/6756056 2022-03-23T20:55:36.899 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi197 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-23_20:43:24-orch-wip-yuri6-testing-2022-03-23-1009-pacific-distro-default-smithi/6756056