2021-11-10T16:08:18.978 INFO:root:teuthology version: 1.1.0-c56135d1
2021-11-10T16:08:19.108 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi012.front.sepia.ceph.com' reimaging
2021-11-10T16:08:19.110 INFO:teuthology.lock.ops:Updating [ubuntu@smithi012.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T16:08:19.111 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server
2021-11-10T16:08:19.133 INFO:teuthology.lock.ops:Node 'ubuntu@smithi012.front.sepia.ceph.com' reimaging is complete
2021-11-10T16:08:19.134 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging
2021-11-10T16:08:19.134 INFO:teuthology.lock.ops:Updating [ubuntu@smithi195.front.sepia.ceph.com]: reset os type and version on server
2021-11-10T16:08:19.134 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2021-11-10T16:08:19.153 INFO:teuthology.lock.ops:Node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging is complete
2021-11-10T16:08:19.346 INFO:teuthology.provision.fog.smithi012:Scheduling deploy of centos 8.3
2021-11-10T16:08:19.364 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of centos 8.3
2021-11-10T16:08:19.657 INFO:teuthology.orchestra.console:Power off smithi012
2021-11-10T16:08:19.675 INFO:teuthology.orchestra.console:Power off smithi195
2021-11-10T16:08:32.146 INFO:teuthology.orchestra.console:Power off for smithi012 completed
2021-11-10T16:08:32.150 INFO:teuthology.orchestra.console:Power off for smithi195 completed
2021-11-10T16:08:32.248 INFO:teuthology.orchestra.console:Power on smithi012
2021-11-10T16:08:32.256 INFO:teuthology.orchestra.console:Power on smithi195
2021-11-10T16:08:36.515 INFO:teuthology.orchestra.console:Power on for smithi195 completed
2021-11-10T16:08:36.616 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish
2021-11-10T16:08:40.636 INFO:teuthology.orchestra.console:Power on for smithi012 completed
2021-11-10T16:08:40.737 INFO:teuthology.provision.fog.smithi012:Waiting for deploy to finish
2021-11-10T16:14:30.517 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T16:14:30.802 INFO:teuthology.provision.fog.smithi012:Node is ready
2021-11-10T16:14:30.818 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012.front.sepia.ceph.com
2021-11-10T16:14:30.875 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12 smithi012.front.sepia.ceph.com smithi012
2021-11-10T16:14:31.195 INFO:teuthology.provision.fog.smithi012:Deploy complete!
2021-11-10T16:14:35.019 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-11-10T16:14:35.351 INFO:teuthology.provision.fog.smithi195:Node is ready
2021-11-10T16:14:35.367 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com
2021-11-10T16:14:35.424 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195
2021-11-10T16:14:35.761 INFO:teuthology.provision.fog.smithi195:Deploy complete!
2021-11-10T16:14:35.913 INFO:teuthology.lock.ops:Checking smithi012.front.sepia.ceph.com
2021-11-10T16:14:35.914 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T16:14:35.935 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com
2021-11-10T16:14:35.935 INFO:teuthology.lock.ops:New key found. Updating...
2021-11-10T16:14:35.952 INFO:teuthology.lock.ops:Updating [smithi012.front.sepia.ceph.com]: set os type and version on server
2021-11-10T16:14:35.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com'
2021-11-10T16:14:36.225 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64
2021-11-10T16:14:36.285 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Linux"
2021-11-10T16:14:36.285 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="8"
2021-11-10T16:14:36.286 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos"
2021-11-10T16:14:36.286 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora"
2021-11-10T16:14:36.286 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="8"
2021-11-10T16:14:36.287 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el8"
2021-11-10T16:14:36.289 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-10T16:14:36.290 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31"
2021-11-10T16:14:36.290 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-10T16:14:36.291 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/"
2021-11-10T16:14:36.291 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-10T16:14:36.291 INFO:teuthology.orchestra.run.smithi012.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-10T16:14:36.292 INFO:teuthology.orchestra.run.smithi012.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-10T16:14:36.295 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server
2021-11-10T16:14:36.321 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server
2021-11-10T16:14:36.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com'
2021-11-10T16:14:36.578 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64
2021-11-10T16:14:36.633 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Linux"
2021-11-10T16:14:36.633 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8"
2021-11-10T16:14:36.633 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos"
2021-11-10T16:14:36.634 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora"
2021-11-10T16:14:36.634 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8"
2021-11-10T16:14:36.634 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8"
2021-11-10T16:14:36.634 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Linux 8"
2021-11-10T16:14:36.634 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31"
2021-11-10T16:14:36.635 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-11-10T16:14:36.635 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/"
2021-11-10T16:14:36.635 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-11-10T16:14:36.635 INFO:teuthology.orchestra.run.smithi195.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-11-10T16:14:36.636 INFO:teuthology.orchestra.run.smithi195.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-11-10T16:14:36.636 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server
2021-11-10T16:14:36.736 INFO:teuthology.dispatcher.supervisor:Running job 6484793
2021-11-10T16:14:36.737 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_akupczyk@teuthology --archive /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484793 --name akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.3-octopus 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls mon_election/connectivity} -- /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484793/orig.config.yaml
2021-11-10T16:14:36.741 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/akupczyk-2021-11-04_09:59:58-rados-wip-bluefs-fine-grain-locking-4-distro-basic-smithi/6484793
2021-11-10T16:14:36.742 INFO:teuthology.dispatcher.supervisor:Job PID: 13726
2021-11-10T16:14:36.743 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-11-11T04:15:30.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-11-11T04:15:31.067 INFO:teuthology.kill:Killing Pids: {13726}
2021-11-11T04:15:32.164 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-11-11T04:15:32.165 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a']
2021-11-11T04:15:32.165 INFO:teuthology.misc:Compressing logs...
2021-11-11T04:15:32.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com'
2021-11-11T04:15:32.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com'
2021-11-11T04:15:32.403 INFO:teuthology.orchestra.run.smithi012.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-11-11T04:15:32.607 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2021-11-11T04:15:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-11T04:15:32.652 INFO:teuthology.orchestra.run.smithi012.stderr:tar: Error is not recoverable: exiting now
2021-11-11T04:15:32.688 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2021-11-11T04:15:32.688 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now
2021-11-11T04:15:32.690 INFO:teuthology.misc:Compressing logs...
2021-11-11T04:15:33.367 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T04:15:34.341 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2021-11-11T04:15:34.869 INFO:teuthology.misc:Compressing logs...
2021-11-11T04:15:36.661 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2021-11-11T04:15:37.336 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2021-11-11T04:15:38.047 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2021-11-11T04:15:39.609 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2021-11-11T04:15:40.360 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2021-11-11T04:15:41.607 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2021-11-11T04:15:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2021-11-11T04:15:44.969 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2021-11-11T04:15:46.783 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2021-11-11T04:15:48.437 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2021-11-11T04:15:50.671 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/8277d4d2-4242-11ec-8c2c-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2021-11-11T04:17:01.855 INFO:teuthology.kill:No teuthology processes running
2021-11-11T04:17:03.058 INFO:teuthology.kill:Nuking machines: ['smithi012', 'smithi195']
2021-11-11T04:17:04.006 INFO:teuthology.kill:2021-11-11 04:17:04,005.005 INFO:teuthology.nuke:targets:
2021-11-11T04:17:04.007 INFO:teuthology.kill:  smithi012.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLox81nefap1hZY2BeXlKk6XUdIr7IocSZ4UsMAGlVqYhV2sEjK4r7P97zmT9QJk6YRqJUUA/HBiFZJWzuxtm7lMjdkdRUMFMN+oteEs9cQmQ8moin14/H4uhTCYamf3K+jhEXVgK+fHND5vbm3bFhBQmaS+8dyKmE6NOU3hqWIhdZGcqS/NIYW8fh6Mo8li7me7TgDvJZx/2Bikmgi+F1xE6CdFwo2eabTgQClBB2xoXYLoByz8stBJ+7b1aVCJ9t3ZVPQJbR4O/fRKz5HLFfyFHpycS7Dg1YvGE/f6ZK++2m+zyNWUlH8kOLD0gMZPg0xxV7o0XqXfV3iY7Qtt9WEwAObJ26NwoyCHy3pmrfAzDgjR+BcNvzBo7mFfMjQ4lG9CUp8RMcYALqwcFDYcCTOIre9TybxRSroLj6Nh9dRWOrSPyOjCQXqtrbkDnWDDShUxErSfZlJjQxpieiL+rodOl0aNqUXkwZ2S0JeB9UVYN5HFGN/uP1fwYbKycEQkk=
2021-11-11T04:17:04.007 INFO:teuthology.kill:  smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO9qvnOODDB7VO0mgexKO3aLOCafsPmv+d0z4QYAaogd56D2WtODk+amyDvL7MB/yOam78Ps7lD0Oukl5iVP8XBthr9aDVnmctpBKcCm4WVRxO9zrxjp0nmeWtxcHuzeHvXrzWa8HqPYPQMXp/cWwMytZs43IwxeXdSDh+N5HdCn+6iHcCNlcR+0v6JWfNIRak4wqZzZsMerT4pwMYMsaY3+1LvfVIVmIPV4Mpdli+4bK6WNN29th2t60zriiLuN6/FM2XVjM4FB/03WpbazkcQ+GYDgpE/4C46dMTFZ06hao4ek5T3/nmHmDmHVLYRBU/TpW5KFwqs+ADKYbCU9bXcfxcsvKkUvq/6ZRcaAIcODHdKzxYbMs47ZfPdHve4YrAGDV9o/tlnaggAKJ9VJxlJ3gMyF1lQpyxjfvwj9atx9HczfglNgtQY9kcQAmS0SsM+L+44zz4TKpalERvGOYNjjIho0nJkoSrNTLGAOwPompzrsnIxFwR0K8OA+NRObM=
2021-11-11T04:17:04.035 INFO:teuthology.kill:2021-11-11 04:17:04,034.034 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T04:17:04.038 INFO:teuthology.kill:2021-11-11 04:17:04,038.038 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-11-11T04:17:04.120 INFO:teuthology.kill:2021-11-11 04:17:04,119.119 INFO:teuthology.orchestra.console:Power off smithi012
2021-11-11T04:17:04.133 INFO:teuthology.kill:2021-11-11 04:17:04,133.133 INFO:teuthology.orchestra.console:Power off smithi195
2021-11-11T04:17:12.497 INFO:teuthology.kill:2021-11-11 04:17:12,496.496 INFO:teuthology.orchestra.console:Power off for smithi195 completed
2021-11-11T04:17:12.643 INFO:teuthology.kill:2021-11-11 04:17:12,642.642 INFO:teuthology.lock.ops:unlocked: smithi195.front.sepia.ceph.com
2021-11-11T04:17:16.602 INFO:teuthology.kill:2021-11-11 04:17:16,602.602 INFO:teuthology.orchestra.console:Power off for smithi012 completed
2021-11-11T04:17:16.747 INFO:teuthology.kill:2021-11-11 04:17:16,746.746 INFO:teuthology.lock.ops:unlocked: smithi012.front.sepia.ceph.com
2021-11-11T04:19:17.060 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-11-11T04:19:17.147 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi012 but it was locked by another job: /home/teuthworker/archive/matan-2021-11-04_14:01:51-smoke-wip-matanb-clear-unstable-distro-basic-smithi/6485048
2021-11-11T04:19:17.147 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi195 but it was locked by another job: /home/teuthworker/archive/matan-2021-11-04_14:01:51-smoke-wip-matanb-clear-unstable-distro-basic-smithi/6485048