2022-01-10T08:59:51.535 INFO:root:teuthology version: 1.1.0-00e09f05
2022-01-10T08:59:51.708 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging
2022-01-10T08:59:51.709 INFO:teuthology.lock.ops:Updating [ubuntu@smithi057.front.sepia.ceph.com]: reset os type and version on server
2022-01-10T08:59:51.709 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-01-10T08:59:51.730 INFO:teuthology.lock.ops:Node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging is complete
2022-01-10T08:59:51.730 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging
2022-01-10T08:59:51.731 INFO:teuthology.lock.ops:Updating [ubuntu@smithi116.front.sepia.ceph.com]: reset os type and version on server
2022-01-10T08:59:51.731 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server
2022-01-10T08:59:51.754 INFO:teuthology.lock.ops:Node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging is complete
2022-01-10T08:59:51.754 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging
2022-01-10T08:59:51.755 INFO:teuthology.lock.ops:Updating [ubuntu@smithi198.front.sepia.ceph.com]: reset os type and version on server
2022-01-10T08:59:51.755 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2022-01-10T08:59:51.776 INFO:teuthology.lock.ops:Node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging is complete
2022-01-10T08:59:51.987 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of centos 8.3
2022-01-10T08:59:52.007 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of centos 8.3
2022-01-10T08:59:52.036 INFO:teuthology.provision.fog.smithi198:Scheduling deploy of centos 8.3
2022-01-10T08:59:52.413 INFO:teuthology.orchestra.console:Power off smithi057
2022-01-10T08:59:52.426 INFO:teuthology.orchestra.console:Power off smithi116
2022-01-10T08:59:52.440 INFO:teuthology.orchestra.console:Power off smithi198
2022-01-10T09:00:00.796 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2022-01-10T09:00:00.803 INFO:teuthology.orchestra.console:Power off for smithi116 completed
2022-01-10T09:00:00.810 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2022-01-10T09:00:00.897 INFO:teuthology.orchestra.console:Power on smithi057
2022-01-10T09:00:00.905 INFO:teuthology.orchestra.console:Power on smithi116
2022-01-10T09:00:00.912 INFO:teuthology.orchestra.console:Power on smithi198
2022-01-10T09:00:09.274 INFO:teuthology.orchestra.console:Power on for smithi057 completed
2022-01-10T09:00:09.282 INFO:teuthology.orchestra.console:Power on for smithi116 completed
2022-01-10T09:00:09.288 INFO:teuthology.orchestra.console:Power on for smithi198 completed
2022-01-10T09:00:09.376 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish
2022-01-10T09:00:09.383 INFO:teuthology.provision.fog.smithi116:Waiting for deploy to finish
2022-01-10T09:00:09.389 INFO:teuthology.provision.fog.smithi198:Waiting for deploy to finish
2022-01-10T09:04:59.148 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-10T09:04:59.195 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-10T09:04:59.197 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-10T09:04:59.438 INFO:teuthology.provision.fog.smithi198:Node is ready
2022-01-10T09:04:59.453 INFO:teuthology.orchestra.run.smithi198.stdout:smithi198.front.sepia.ceph.com
2022-01-10T09:04:59.479 INFO:teuthology.provision.fog.smithi116:Node is ready
2022-01-10T09:04:59.509 INFO:teuthology.orchestra.run.smithi198.stdout:172.21.15.198 smithi198.front.sepia.ceph.com smithi198
2022-01-10T09:04:59.519 INFO:teuthology.provision.fog.smithi057:Node is ready
2022-01-10T09:04:59.535 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com
2022-01-10T09:04:59.537 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116.front.sepia.ceph.com
2022-01-10T09:04:59.591 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057
2022-01-10T09:04:59.593 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116 smithi116.front.sepia.ceph.com smithi116
2022-01-10T09:04:59.848 INFO:teuthology.provision.fog.smithi198:Deploy complete!
2022-01-10T09:04:59.912 INFO:teuthology.provision.fog.smithi116:Deploy complete!
2022-01-10T09:04:59.917 INFO:teuthology.provision.fog.smithi057:Deploy complete!
2022-01-10T09:05:00.091 INFO:teuthology.lock.ops:Checking smithi057.front.sepia.ceph.com
2022-01-10T09:05:00.092 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-10T09:05:00.111 INFO:teuthology.lock.ops:Checking smithi116.front.sepia.ceph.com
2022-01-10T09:05:00.112 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-10T09:05:00.130 INFO:teuthology.lock.ops:Checking smithi198.front.sepia.ceph.com
2022-01-10T09:05:00.130 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-10T09:05:00.150 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: set os type and version on server
2022-01-10T09:05:00.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-01-10T09:05:00.477 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64
2022-01-10T09:05:00.545 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Linux"
2022-01-10T09:05:00.546 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8"
2022-01-10T09:05:00.546 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos"
2022-01-10T09:05:00.546 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora"
2022-01-10T09:05:00.547 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8"
2022-01-10T09:05:00.547 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8"
2022-01-10T09:05:00.547 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Linux 8"
2022-01-10T09:05:00.547 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31"
2022-01-10T09:05:00.548 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-10T09:05:00.548 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/"
2022-01-10T09:05:00.548 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2022-01-10T09:05:00.548 INFO:teuthology.orchestra.run.smithi057.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2022-01-10T09:05:00.548 INFO:teuthology.orchestra.run.smithi057.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2022-01-10T09:05:00.550 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-01-10T09:05:00.572 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: set os type and version on server
2022-01-10T09:05:00.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com'
2022-01-10T09:05:00.888 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64
2022-01-10T09:05:00.943 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Linux"
2022-01-10T09:05:00.943 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="8"
2022-01-10T09:05:00.943 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos"
2022-01-10T09:05:00.943 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora"
2022-01-10T09:05:00.944 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="8"
2022-01-10T09:05:00.944 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el8"
2022-01-10T09:05:00.944 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Linux 8"
2022-01-10T09:05:00.944 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31"
2022-01-10T09:05:00.945 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-10T09:05:00.945 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/"
2022-01-10T09:05:00.945 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2022-01-10T09:05:00.945 INFO:teuthology.orchestra.run.smithi116.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2022-01-10T09:05:00.945 INFO:teuthology.orchestra.run.smithi116.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2022-01-10T09:05:00.946 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server
2022-01-10T09:05:00.965 INFO:teuthology.lock.ops:Updating [smithi198.front.sepia.ceph.com]: set os type and version on server
2022-01-10T09:05:00.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi198.front.sepia.ceph.com'
2022-01-10T09:05:01.231 INFO:teuthology.orchestra.run.smithi198.stdout:x86_64
2022-01-10T09:05:01.284 INFO:teuthology.orchestra.run.smithi198.stdout:NAME="CentOS Linux"
2022-01-10T09:05:01.285 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION="8"
2022-01-10T09:05:01.285 INFO:teuthology.orchestra.run.smithi198.stdout:ID="centos"
2022-01-10T09:05:01.285 INFO:teuthology.orchestra.run.smithi198.stdout:ID_LIKE="rhel fedora"
2022-01-10T09:05:01.286 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION_ID="8"
2022-01-10T09:05:01.286 INFO:teuthology.orchestra.run.smithi198.stdout:PLATFORM_ID="platform:el8"
2022-01-10T09:05:01.286 INFO:teuthology.orchestra.run.smithi198.stdout:PRETTY_NAME="CentOS Linux 8"
2022-01-10T09:05:01.286 INFO:teuthology.orchestra.run.smithi198.stdout:ANSI_COLOR="0;31"
2022-01-10T09:05:01.287 INFO:teuthology.orchestra.run.smithi198.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-10T09:05:01.287 INFO:teuthology.orchestra.run.smithi198.stdout:HOME_URL="https://centos.org/"
2022-01-10T09:05:01.287 INFO:teuthology.orchestra.run.smithi198.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2022-01-10T09:05:01.287 INFO:teuthology.orchestra.run.smithi198.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2022-01-10T09:05:01.288 INFO:teuthology.orchestra.run.smithi198.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2022-01-10T09:05:01.288 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2022-01-10T09:05:01.403 INFO:teuthology.dispatcher.supervisor:Running job 6605530
2022-01-10T09:05:01.403 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-01-10_08:04:33-fs-wip-vshankar-testing-20220110-105818-testing-default-smithi/6605530 --name vshankar-2022-01-10_08:04:33-fs-wip-vshankar-testing-20220110-105818-testing-default-smithi --description fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} -- /home/teuthworker/archive/vshankar-2022-01-10_08:04:33-fs-wip-vshankar-testing-20220110-105818-testing-default-smithi/6605530/orig.config.yaml
2022-01-10T09:05:01.407 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-01-10_08:04:33-fs-wip-vshankar-testing-20220110-105818-testing-default-smithi/6605530
2022-01-10T09:05:01.408 INFO:teuthology.dispatcher.supervisor:Job PID: 31448
2022-01-10T09:05:01.409 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-01-10T21:05:56.411 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-01-10T21:05:56.626 INFO:teuthology.kill:Killing Pids: {31448}
2022-01-10T21:05:57.866 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2022-01-10T21:05:57.867 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2022-01-10T21:05:57.867 INFO:teuthology.task.internal:roles: ubuntu@smithi198.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2022-01-10T21:05:57.867 INFO:teuthology.misc:Compressing logs...
2022-01-10T21:05:57.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-01-10T21:05:58.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com'
2022-01-10T21:05:58.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi198.front.sepia.ceph.com'
2022-01-10T21:05:58.861 INFO:teuthology.misc:Compressing logs...
2022-01-10T21:05:59.124 INFO:teuthology.misc:Compressing logs...
2022-01-10T21:13:40.746 INFO:teuthology.kill:No teuthology processes running
2022-01-10T21:13:42.099 INFO:teuthology.kill:Nuking machines: ['smithi057', 'smithi116', 'smithi198']
2022-01-10T21:13:43.135 INFO:teuthology.kill:2022-01-10 21:13:43,135.135 INFO:teuthology.nuke:targets:
2022-01-10T21:13:43.136 INFO:teuthology.kill:  smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsSxWML93SafpuUVfIm0PFVTzZURZX41sbN1XZAaAphYSl2nHq2FPvPA0kp412780UqxkKy8NK9z9xsb/w2m716HVdBS2OV/Nv6m+l8objRgx8JiXIRQ3sHRCnzzc5F736yEeZFvAEK1Pu3bPN0linbl9ZxKnR/krsSSpAZMMzYGK0NsxbPFjptCkrm/7fMjaZJxjw7RznUjNAOXFJAxK7ClyLbVlKNUPIDduLsH3vR3tOt4EQA+fYbvFsjXgNzw3NI8GmDAdQi9fVxWYkX5yFcB5AKteKOpov4A3uUjPVn/zzqW2t5Awp6Sg8Hq0jPJz8CfXSTDVxsiALK6KnDihGPLnkqCrMPkgSQy59HeHdj2wQeCdQVk9M33KEXLsmnDGS8NSEcXN2diQj1zZpZk90Lz9x4sQ1KdlDce85p9bHUCPXy4V4QwWAKs7HnP/ilXRNAfcVz043amzUBoJ8vmdjPTAVNNpxMmSqDBsQ0qyoG2yKYMWLHNYlVWeI5Of7KDM=
2022-01-10T21:13:43.136 INFO:teuthology.kill:  smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqj+Nveb6XQIpHOBuyOBErxHZE3JAxcdBaaa987cpvcoJr1nkKLNVW199GKC3hvMjHQQOPefoo2Hg/s2oRWJu8yknFePtXO9Esvyt2iT2JJRXrf50trqqz8+ULQZgEsNSJGxKkA74HPpL+HD8nny3q1sXAhAdvVP+OSsRl+gWRpfjwlFpjNyZ2db+ALUd9XoGD4BFcjUbk4w9nWY6FGe2RvsLxpsrFlaFQ2q2YeXzPsKLQnQsmfmnEeHlrTXs2moFBsJLFph7fNpoIbItC50dkUbqd5pXSMEStMSglWPTLuvgK2/5Q5o3ceLT67ebAeLOV7vtFRCCX9Gpczo5x8wLcmCoDBSkCUAlmbCzBfFKrNyoTStIKxNVinPXSVezxIgFS4Gpwl7mskc/hKjKyGIjMDuA3lPMFUGZpw0pL2Z9sm9/Qa9O2QtLGIXev8VH0mNObXC5oDJmKfQBCqcD2q5XxddSpLMMTWw2XB0hGId3DqmMS9ueNdxETzBd8e2+EVN0=
2022-01-10T21:13:43.137 INFO:teuthology.kill:  smithi198.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC741xoTLfuW5ijcloxHdUJd6tGP90iMMUiArOeHjN6ADeQ+thGzsqECTiLJ6FLCn2qOPd9TlwHhlXnKajUbj2W8koASpaJyc6iGS4hOeL56j21ZPSxtEoQRCb3xv8HucqzgHalHy7B20pAaC008ylVClW1W2XvVlq70zwEb+e8mM1nZ76/S01v/dNK8WdM00DqbWJNLJvUoEX0+znXMqe6Brxo/orrafvD41wDU9amYvohQBguID04gKhlsJF5Bxo6IcC5zJ8U+h36BEF+mWI6vkSexYdsX1SXqJR1eGmfuNk8l9gzTIq9WvQL4y+uzg5dAqGKeS8n4h8Zhk7fnCdkx/WuEGZO03nI20sYMUnafSPcfp8im5w6tL4C7GzSvCIkfA+E+x4GC3n3gjrBtMEEqooSFIz5woG1xfglFMmUHyXC7CNgLI9UtNemdA7O8Q4Fzw2MYhLOaHRUuqDTZOWyaj2t1cch+zEcQMxCyGEESaPSdLvo3f19GjGUt9lWP1s=
2022-01-10T21:13:43.168 INFO:teuthology.kill:2022-01-10 21:13:43,168.168 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-10T21:13:43.174 INFO:teuthology.kill:2022-01-10 21:13:43,174.174 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-10T21:13:43.178 INFO:teuthology.kill:2022-01-10 21:13:43,178.178 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-10T21:13:43.252 INFO:teuthology.kill:2022-01-10 21:13:43,251.251 INFO:teuthology.orchestra.console:Power off smithi116
2022-01-10T21:13:43.270 INFO:teuthology.kill:2022-01-10 21:13:43,269.269 INFO:teuthology.orchestra.console:Power off smithi198
2022-01-10T21:13:43.286 INFO:teuthology.kill:2022-01-10 21:13:43,286.286 INFO:teuthology.orchestra.console:Power off smithi057
2022-01-10T21:13:51.652 INFO:teuthology.kill:2022-01-10 21:13:51,652.652 INFO:teuthology.orchestra.console:Power off for smithi116 completed
2022-01-10T21:13:51.665 INFO:teuthology.kill:2022-01-10 21:13:51,664.664 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2022-01-10T21:13:51.819 INFO:teuthology.kill:2022-01-10 21:13:51,818.818 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com
2022-01-10T21:13:51.821 INFO:teuthology.kill:2022-01-10 21:13:51,821.821 INFO:teuthology.lock.ops:unlocked: smithi116.front.sepia.ceph.com
2022-01-10T21:13:55.770 INFO:teuthology.kill:2022-01-10 21:13:55,769.769 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2022-01-10T21:13:55.918 INFO:teuthology.kill:2022-01-10 21:13:55,918.918 INFO:teuthology.lock.ops:unlocked: smithi198.front.sepia.ceph.com
2022-01-10T21:15:56.280 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-01-10T21:15:56.423 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi057 but it was locked by another job: /home/teuthworker/archive/mchangir-2022-01-09_09:38:56-fs-wip-mchangir-mds-retry-backtrace-validation-distro-default-smithi/6604692
2022-01-10T21:15:56.424 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi116 but it was locked by another job: /home/teuthworker/archive/mchangir-2022-01-09_09:38:56-fs-wip-mchangir-mds-retry-backtrace-validation-distro-default-smithi/6604692
2022-01-10T21:15:56.424 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi198 but it was locked by another job: /home/teuthworker/archive/mchangir-2022-01-09_09:38:56-fs-wip-mchangir-mds-retry-backtrace-validation-distro-default-smithi/6604691