2022-03-21T07:38:26.056 INFO:root:teuthology version: 1.1.0-a4e18662 2022-03-21T07:38:26.179 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi092.front.sepia.ceph.com' reimaging 2022-03-21T07:38:26.181 INFO:teuthology.lock.ops:Updating [ubuntu@smithi092.front.sepia.ceph.com]: reset os type and version on server 2022-03-21T07:38:26.181 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2022-03-21T07:38:26.199 INFO:teuthology.lock.ops:Node 'ubuntu@smithi092.front.sepia.ceph.com' reimaging is complete 2022-03-21T07:38:26.199 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging 2022-03-21T07:38:26.199 INFO:teuthology.lock.ops:Updating [ubuntu@smithi122.front.sepia.ceph.com]: reset os type and version on server 2022-03-21T07:38:26.200 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2022-03-21T07:38:26.236 INFO:teuthology.lock.ops:Node 'ubuntu@smithi122.front.sepia.ceph.com' reimaging is complete 2022-03-21T07:38:26.399 INFO:teuthology.provision.fog.smithi092:Scheduling deploy of ubuntu 20.04 2022-03-21T07:38:26.443 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of ubuntu 20.04 2022-03-21T07:38:26.677 INFO:teuthology.orchestra.console:Power off smithi092 2022-03-21T07:38:26.717 INFO:teuthology.orchestra.console:Power off smithi122 2022-03-21T07:38:26.825 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2022-03-21T07:38:26.854 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2022-03-21T07:38:26.927 INFO:teuthology.orchestra.console:Power on smithi092 2022-03-21T07:38:26.955 INFO:teuthology.orchestra.console:Power on smithi122 2022-03-21T07:38:35.288 INFO:teuthology.orchestra.console:Power on for smithi092 completed 2022-03-21T07:38:35.372 INFO:teuthology.orchestra.console:Power on for smithi122 completed 2022-03-21T07:38:35.390 INFO:teuthology.provision.fog.smithi092:Waiting for deploy to finish 2022-03-21T07:38:35.473 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish 2022-03-21T07:43:29.706 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-21T07:43:30.167 INFO:teuthology.provision.fog.smithi092:Node is ready 2022-03-21T07:43:30.174 INFO:teuthology.orchestra.run.smithi092.stdout:smithi092.front.sepia.ceph.com 2022-03-21T07:43:30.225 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.92 smithi092.front.sepia.ceph.com smithi092 2022-03-21T07:43:30.436 INFO:teuthology.provision.fog.smithi092:Deploy complete! 2022-03-21T07:43:38.686 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-21T07:43:39.173 INFO:teuthology.provision.fog.smithi122:Node is ready 2022-03-21T07:43:39.179 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122.front.sepia.ceph.com 2022-03-21T07:43:39.187 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122 2022-03-21T07:43:39.365 INFO:teuthology.provision.fog.smithi122:Deploy complete! 2022-03-21T07:43:39.571 INFO:teuthology.lock.ops:Checking smithi092.front.sepia.ceph.com 2022-03-21T07:43:39.572 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-21T07:43:39.589 INFO:teuthology.lock.ops:Checking smithi122.front.sepia.ceph.com 2022-03-21T07:43:39.590 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-21T07:43:39.607 INFO:teuthology.lock.ops:Updating [smithi092.front.sepia.ceph.com]: set os type and version on server 2022-03-21T07:43:39.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-03-21T07:43:39.782 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2022-03-21T07:43:39.789 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu" 2022-03-21T07:43:39.789 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-21T07:43:39.789 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu 2022-03-21T07:43:39.790 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian 2022-03-21T07:43:39.790 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-21T07:43:39.790 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="20.04" 2022-03-21T07:43:39.790 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-21T07:43:39.791 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-21T07:43:39.791 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-21T07:43:39.791 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-21T07:43:39.791 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=focal 2022-03-21T07:43:39.791 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=focal 2022-03-21T07:43:39.792 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2022-03-21T07:43:39.810 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: set os type and version on server 2022-03-21T07:43:39.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2022-03-21T07:43:39.977 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2022-03-21T07:43:39.983 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Ubuntu" 2022-03-21T07:43:39.984 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-21T07:43:39.984 INFO:teuthology.orchestra.run.smithi122.stdout:ID=ubuntu 2022-03-21T07:43:39.984 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE=debian 2022-03-21T07:43:39.984 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-21T07:43:39.985 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="20.04" 2022-03-21T07:43:39.985 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-21T07:43:39.985 INFO:teuthology.orchestra.run.smithi122.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-21T07:43:39.985 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-21T07:43:39.986 INFO:teuthology.orchestra.run.smithi122.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-21T07:43:39.986 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_CODENAME=focal 2022-03-21T07:43:39.986 INFO:teuthology.orchestra.run.smithi122.stdout:UBUNTU_CODENAME=focal 2022-03-21T07:43:39.986 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2022-03-21T07:43:40.082 INFO:teuthology.dispatcher.supervisor:Running job 6749795 2022-03-21T07:43:40.083 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_a4e18662e5de936eb0ac089e531b933ecbcadb0a/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749795 --name vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} -- /home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749795/orig.config.yaml 2022-03-21T07:43:40.085 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-21_06:07:09-fs-wip-vshankar-testing-20220319-163539-testing-default-smithi/6749795 2022-03-21T07:43:40.086 INFO:teuthology.dispatcher.supervisor:Job PID: 15272 2022-03-21T07:43:40.086 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-21T14:14:08.268 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-21T14:14:08.439 INFO:teuthology.kill:Killing Pids: {15272} 2022-03-21T14:14:09.597 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-03-21T14:14:09.598 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-03-21T14:14:09.598 INFO:teuthology.misc:Compressing logs... 2022-03-21T14:14:09.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-03-21T14:14:09.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2022-03-21T14:14:10.365 INFO:teuthology.misc:Compressing logs... 2022-03-21T14:14:10.528 INFO:teuthology.misc:Compressing logs... 2022-03-21T14:21:36.948 INFO:teuthology.kill:No teuthology processes running 2022-03-21T14:21:38.068 INFO:teuthology.kill:Nuking machines: ['smithi092', 'smithi122'] 2022-03-21T14:21:39.010 INFO:teuthology.kill:2022-03-21 14:21:39,009.009 INFO:teuthology.nuke:targets: 2022-03-21T14:21:39.011 INFO:teuthology.kill: smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT8k0+E3S55J4ShuPC9OdkVF5lO4WjPRXNbjtah8NC/MMrTzLndSHoxrh9CVoeMIxvGTnUb8Lx08EG4TS10TNb8yIdXCZ43VyWhC37m6eBBnGtBhT2FUd5DMEV4K9qpNr9R/aaZh6cTaCDRbutYzidT9Xw5FOceo67DOHpELj0Ie1X4MwYmDctU/CIrcDDkDyeX6cw9mhtZ2kkY9LQOqFl8Kg3vWx3N7FOEO21enn4P+NeBqqr5R7n48ZQQOfgSZu4Y7UKcYI4ptY2s0049NyKFORVBKrhE8z5zdC28FJ23C9cXT1S18pA5UFRT10rZY3HKyd7+Uv9j3pdX76XqLy9PDZDag+XYTYOI3ruwy79DSrPSOQTOau743lZvKBH3FiBFCq4LiqRtJmGb0z1vDIKDpaBSE0nxKmISrxjDI2z1+pgtJ+AUXQSaBGyjuJy3ZxuWkhzqTxp8b3ZXp3JuGE1euUS0jz1gMnguWe9rIJ9NPbmZyYjoNtv4ynWsKnEWJk= 2022-03-21T14:21:39.011 INFO:teuthology.kill: smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWmm3gpFn6GXNfasSj6xxhJ0CpCXtIizJBbucf7CmU1nhqXZuESSQYYa0UAb+qODIm/64RT7zzMuxjnexcZwt7zV/7IgyQSWDcbraRPUrhzSRTwGPAxPeAZOtjN/EdRb5kXb5FcoWyW2cS8TMkWI0xVG8VyowaS+9Xp99MIG7O0eqRhnB3LYK48r6IaWzVqceU6RXtv7StEsY6LK2LIlcG2KtSziHDkCwpIZOc5BfACAOgruO+swh5VtpY6vqOtpid3+BisXeUjYLSX4UsBi7A2tyen84Nx3QL/+aEeuSYRDzgo38sNT9zmKk9bs5wqbz57DSyqsAh6QW2Ghc1m6x9EyRqh+Xj7zjNs/WWhnOa7bCUczvaWr57x7AqQLGfqPTSfV6JpZx0NxDHAmONSMvGHDMggPCYFhAvuS8MsWKBLr5/x3ep+hqQ0NRoEPquUMpPDiB2p4Bi6xOcFhRAKPYRagCpH5VKSYQqvQPIlEFTaLoh2fLA4AHwLxYWvLABkQ0= 2022-03-21T14:21:39.036 INFO:teuthology.kill:2022-03-21 14:21:39,036.036 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-21T14:21:39.039 INFO:teuthology.kill:2022-03-21 14:21:39,039.039 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-21T14:21:39.117 INFO:teuthology.kill:2022-03-21 14:21:39,117.117 INFO:teuthology.orchestra.console:Power off smithi122 2022-03-21T14:21:39.129 INFO:teuthology.kill:2022-03-21 14:21:39,129.129 INFO:teuthology.orchestra.console:Power off smithi092 2022-03-21T14:21:47.485 INFO:teuthology.kill:2022-03-21 14:21:47,485.485 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2022-03-21T14:21:47.493 INFO:teuthology.kill:2022-03-21 14:21:47,493.493 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2022-03-21T14:21:47.640 INFO:teuthology.kill:2022-03-21 14:21:47,639.639 INFO:teuthology.lock.ops:unlocked: smithi092.front.sepia.ceph.com 2022-03-21T14:21:47.643 INFO:teuthology.kill:2022-03-21 14:21:47,643.643 INFO:teuthology.lock.ops:unlocked: smithi122.front.sepia.ceph.com 2022-03-21T14:23:47.950 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-21T14:23:48.062 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi092 but it was locked by another job: /home/teuthworker/archive/gsalomon-2022-03-21_06:34:03-rgw:verify-wip-test-arrow-submodule-distro-basic-smithi/6750060 2022-03-21T14:23:48.062 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi122 but it was locked by another job: /home/teuthworker/archive/gsalomon-2022-03-21_06:34:03-rgw:verify-wip-test-arrow-submodule-distro-basic-smithi/6750060