2022-01-13T09:14:12.146 INFO:root:teuthology version: 1.1.0-6fc20113 2022-01-13T09:14:12.268 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging 2022-01-13T09:14:12.269 INFO:teuthology.lock.ops:Updating [ubuntu@smithi039.front.sepia.ceph.com]: reset os type and version on server 2022-01-13T09:14:12.269 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2022-01-13T09:14:12.288 INFO:teuthology.lock.ops:Node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging is complete 2022-01-13T09:14:12.288 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging 2022-01-13T09:14:12.288 INFO:teuthology.lock.ops:Updating [ubuntu@smithi159.front.sepia.ceph.com]: reset os type and version on server 2022-01-13T09:14:12.289 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2022-01-13T09:14:12.307 INFO:teuthology.lock.ops:Node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging is complete 2022-01-13T09:14:12.502 INFO:teuthology.provision.fog.smithi039:Scheduling deploy of centos 8.3 2022-01-13T09:14:12.530 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of centos 8.3 2022-01-13T09:14:12.876 INFO:teuthology.orchestra.console:Power off smithi039 2022-01-13T09:14:12.918 INFO:teuthology.orchestra.console:Power off smithi159 2022-01-13T09:14:21.278 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2022-01-13T09:14:21.380 INFO:teuthology.orchestra.console:Power on smithi159 2022-01-13T09:14:25.369 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2022-01-13T09:14:25.471 INFO:teuthology.orchestra.console:Power on smithi039 2022-01-13T09:14:29.745 INFO:teuthology.orchestra.console:Power on for smithi159 completed 2022-01-13T09:14:29.846 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish 2022-01-13T09:14:33.841 INFO:teuthology.orchestra.console:Power on for smithi039 completed 2022-01-13T09:14:33.942 INFO:teuthology.provision.fog.smithi039:Waiting for deploy to finish 2022-01-13T09:19:34.933 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-13T09:19:35.265 INFO:teuthology.provision.fog.smithi159:Node is ready 2022-01-13T09:19:35.280 INFO:teuthology.orchestra.run.smithi159.stdout:smithi159.front.sepia.ceph.com 2022-01-13T09:19:35.336 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.15.159 smithi159.front.sepia.ceph.com smithi159 2022-01-13T09:19:35.665 INFO:teuthology.provision.fog.smithi159:Deploy complete! 2022-01-13T09:19:53.894 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-13T09:19:54.315 INFO:teuthology.provision.fog.smithi039:Node is ready 2022-01-13T09:19:54.331 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039.front.sepia.ceph.com 2022-01-13T09:19:54.387 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39 smithi039.front.sepia.ceph.com smithi039 2022-01-13T09:19:54.721 INFO:teuthology.provision.fog.smithi039:Deploy complete! 2022-01-13T09:19:54.872 INFO:teuthology.lock.ops:Checking smithi039.front.sepia.ceph.com 2022-01-13T09:19:54.872 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-13T09:19:54.901 INFO:teuthology.lock.ops:Checking smithi159.front.sepia.ceph.com 2022-01-13T09:19:54.901 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-13T09:19:54.920 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: set os type and version on server 2022-01-13T09:19:54.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2022-01-13T09:19:55.121 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2022-01-13T09:19:55.176 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Linux" 2022-01-13T09:19:55.176 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8" 2022-01-13T09:19:55.177 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2022-01-13T09:19:55.177 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2022-01-13T09:19:55.177 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8" 2022-01-13T09:19:55.177 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2022-01-13T09:19:55.178 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-13T09:19:55.178 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2022-01-13T09:19:55.178 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-13T09:19:55.178 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2022-01-13T09:19:55.179 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-13T09:19:55.179 INFO:teuthology.orchestra.run.smithi039.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-13T09:19:55.179 INFO:teuthology.orchestra.run.smithi039.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-13T09:19:55.181 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2022-01-13T09:19:55.199 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: set os type and version on server 2022-01-13T09:19:55.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2022-01-13T09:19:55.473 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2022-01-13T09:19:55.528 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Linux" 2022-01-13T09:19:55.529 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8" 2022-01-13T09:19:55.529 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2022-01-13T09:19:55.529 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2022-01-13T09:19:55.529 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8" 2022-01-13T09:19:55.530 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8" 2022-01-13T09:19:55.530 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Linux 8" 2022-01-13T09:19:55.530 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2022-01-13T09:19:55.530 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-13T09:19:55.531 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2022-01-13T09:19:55.531 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2022-01-13T09:19:55.531 INFO:teuthology.orchestra.run.smithi159.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2022-01-13T09:19:55.531 INFO:teuthology.orchestra.run.smithi159.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2022-01-13T09:19:55.532 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2022-01-13T09:19:55.631 INFO:teuthology.dispatcher.supervisor:Running job 6610984 2022-01-13T09:19:55.632 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6fc2011361437a9dfe4e45b50de224392eed8abc/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-01-12_19:45:04-rados-WIP_GBH_NCB_expand_device_fix_1-distro-basic-smithi/6610984 --name benhanokh-2022-01-12_19:45:04-rados-WIP_GBH_NCB_expand_device_fix_1-distro-basic-smithi --description rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} -- /home/teuthworker/archive/benhanokh-2022-01-12_19:45:04-rados-WIP_GBH_NCB_expand_device_fix_1-distro-basic-smithi/6610984/orig.config.yaml 2022-01-13T09:19:55.636 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-01-12_19:45:04-rados-WIP_GBH_NCB_expand_device_fix_1-distro-basic-smithi/6610984 2022-01-13T09:19:55.637 INFO:teuthology.dispatcher.supervisor:Job PID: 22021 2022-01-13T09:19:55.637 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-13T21:20:50.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-01-13T21:20:50.947 INFO:teuthology.kill:Killing Pids: {22021} 2022-01-13T21:20:52.229 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['host.a', 'client.0'] 2022-01-13T21:20:52.230 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.b', 'client.1'] 2022-01-13T21:20:52.230 INFO:teuthology.misc:Compressing logs... 2022-01-13T21:20:52.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2022-01-13T21:20:52.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2022-01-13T21:20:52.491 INFO:teuthology.orchestra.run.smithi039.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-01-13T21:20:52.752 INFO:teuthology.orchestra.run.smithi159.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-01-13T21:20:52.795 INFO:teuthology.orchestra.run.smithi039.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-01-13T21:20:52.795 INFO:teuthology.orchestra.run.smithi039.stderr:tar: Error is not recoverable: exiting now 2022-01-13T21:20:52.834 INFO:teuthology.orchestra.run.smithi159.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-01-13T21:20:52.835 INFO:teuthology.orchestra.run.smithi159.stderr:tar: Error is not recoverable: exiting now 2022-01-13T21:20:52.836 INFO:teuthology.misc:Compressing logs... 2022-01-13T21:20:53.060 INFO:teuthology.misc:Compressing logs... 2022-01-13T21:20:59.568 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-mon.smithi159.log: file size changed while zipping 2022-01-13T21:21:09.131 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-01-13T21:21:13.268 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-01-13T21:21:23.647 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-01-13T21:21:30.137 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-01-13T21:21:42.081 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-mon.smithi039.log: file size changed while zipping 2022-01-13T21:21:43.069 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-01-13T21:21:59.703 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-01-13T21:22:01.535 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-01-13T21:22:04.968 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-mgr.smithi039.wnlxop.log: file size changed while zipping 2022-01-13T21:22:19.633 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/5a0b2c04-7453-11ec-8c32-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-01-13T21:22:51.852 INFO:teuthology.kill:No teuthology processes running 2022-01-13T21:22:53.030 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi159'] 2022-01-13T21:22:53.990 INFO:teuthology.kill:2022-01-13 21:22:53,990.990 INFO:teuthology.nuke:targets: 2022-01-13T21:22:53.991 INFO:teuthology.kill: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvH8m5UlCckfsK1y3KVqqzjJS2QG/3k99xI/ivZEhy3xMpM3X+1yOhBfLUFWOUbSlsXcDsz1XbY3SmXQFvzRvEAni3yCo2UZh5gT2hLZvCpGpnIDPk8m/N2dSHiQwrPx60HM+CRT6msBsXsMamA0G+rW6KfSugyT0AJjD4Brp9HH+81bsTfDlijLYKdk+3WVES+gJRx/ZRYZAMWgL0qk6eA0rcDsRKGVnZXHcxsIXNquS+KlpJ5iRNW9X5EO+Nn1KPnmWbnAp2RqXfBbOjDzoyfyeJkPWLY/VLhm6F+vxOrttydMfFu7Zx715kxI4Dytx9HkN3wA8xOMmN1dzo2eW0/WzHILlr9FXbTbcH3heFnOpMj3VIkrE0YkXIovMPUsy2ir6f9KVg3GZsjtKpFBCg98uFGYoU2bVMvIadwV99k1PxYzDJ7lfFDXYDySCDK2WwxlWlr9ty+MKLGI3bDj+/8P/dUkxPFhAFPzSThsBRznXC9JhmlyoOg67MDS7n0sc= 2022-01-13T21:22:53.991 INFO:teuthology.kill: smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjywzr/pEcsk8chN7pies1yax6R1KJSxtmWpGETrhZTWeHyni593QODeqsCxo4TfX+np4qUV2mToYh0SZ79uEZO9J6b7skoqKiltMJ6YbpfuVQ2LRElpoYyTioMRuTqqE886rinXEnvNO1tqtW59hDAFAEateeCwf/DTWtc8S6ZAJhC7yT9P0qLwnsWxi08Bh+OjtZLD5yF50ngpO9RIUsAjDb03vaVmtm6CGAxNkvxFcZ76yZf4pQf8lW+E0aBG5hC461+4uUahSAzUjBcK2r5CoScH5hMyVvdl1pUGbcLwJTCOrtPbEPkvKqrU0XZ2W3SRoQm+P9VHTNHStzQ16FQzUIaxGKJxPtKw8OcoO0YW0A24XpfPt8N49WQgLwqQpPN3FwnpAjF3XCNSXnaqt8eSxFybzUgAMBslNfahE1++v/F8e0LkQPn2si7S5LMkfYLbDz6phUwumw4GmmMgK6U8Myx7If3tx4oz0HFYgsF6T1SaGUu9DzaeMewjMuyz8= 2022-01-13T21:22:54.017 INFO:teuthology.kill:2022-01-13 21:22:54,017.017 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-13T21:22:54.021 INFO:teuthology.kill:2022-01-13 21:22:54,020.020 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-13T21:22:54.098 INFO:teuthology.kill:2022-01-13 21:22:54,097.097 INFO:teuthology.orchestra.console:Power off smithi039 2022-01-13T21:22:54.112 INFO:teuthology.kill:2022-01-13 21:22:54,111.111 INFO:teuthology.orchestra.console:Power off smithi159 2022-01-13T21:23:02.467 INFO:teuthology.kill:2022-01-13 21:23:02,467.467 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2022-01-13T21:23:02.636 INFO:teuthology.kill:2022-01-13 21:23:02,635.635 INFO:teuthology.lock.ops:unlocked: smithi039.front.sepia.ceph.com 2022-01-13T21:23:06.595 INFO:teuthology.kill:2022-01-13 21:23:06,594.594 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2022-01-13T21:23:06.740 INFO:teuthology.kill:2022-01-13 21:23:06,740.740 INFO:teuthology.lock.ops:unlocked: smithi159.front.sepia.ceph.com 2022-01-13T21:25:07.045 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-01-13T21:25:07.148 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi039 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-13_18:06:52-rados-wip-yuri3-testing-2022-01-13-0809-distro-default-smithi/6614484 2022-01-13T21:25:07.148 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi159 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-13_18:06:52-rados-wip-yuri3-testing-2022-01-13-0809-distro-default-smithi/6614484