2022-09-10T10:53:47.132 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-10T10:53:47.310 INFO:teuthology.lock.ops:Start node 'smithi136.front.sepia.ceph.com' reimaging
2022-09-10T10:53:47.311 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: reset os type and version on server
2022-09-10T10:53:47.312 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2022-09-10T10:53:47.340 INFO:teuthology.lock.ops:Node 'smithi136.front.sepia.ceph.com' reimaging is complete
2022-09-10T10:53:47.341 INFO:teuthology.lock.ops:Start node 'smithi149.front.sepia.ceph.com' reimaging
2022-09-10T10:53:47.341 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: reset os type and version on server
2022-09-10T10:53:47.342 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server
2022-09-10T10:53:47.363 INFO:teuthology.lock.ops:Node 'smithi149.front.sepia.ceph.com' reimaging is complete
2022-09-10T10:53:47.566 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of centos 8.stream
2022-09-10T10:53:47.574 INFO:teuthology.provision.fog.smithi149:Scheduling deploy of centos 8.stream
2022-09-10T10:53:47.889 INFO:teuthology.orchestra.console:Power off smithi136
2022-09-10T10:53:47.909 INFO:teuthology.orchestra.console:Power off smithi149
2022-09-10T10:53:48.043 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2022-09-10T10:53:48.052 INFO:teuthology.orchestra.console:Power off for smithi149 completed
2022-09-10T10:53:48.145 INFO:teuthology.orchestra.console:Power on smithi136
2022-09-10T10:53:48.155 INFO:teuthology.orchestra.console:Power on smithi149
2022-09-10T10:53:52.418 INFO:teuthology.orchestra.console:Power on for smithi149 completed
2022-09-10T10:53:52.519 INFO:teuthology.provision.fog.smithi149:Waiting for deploy to finish
2022-09-10T10:53:56.520 INFO:teuthology.orchestra.console:Power on for smithi136 completed
2022-09-10T10:53:56.622 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish
2022-09-10T10:58:02.963 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-09-10T10:58:28.323 INFO:teuthology.provision.fog.smithi136:Node is ready
2022-09-10T10:58:28.345 INFO:teuthology.orchestra.run.smithi136.stdout:smithi130
2022-09-10T10:58:28.402 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.130 smithi130.front.sepia.ceph.com smithi130
2022-09-10T10:58:28.778 INFO:teuthology.provision.fog.smithi136:Deploy complete!
2022-09-10T10:58:48.553 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-09-10T10:58:48.862 INFO:teuthology.provision.fog.smithi149:Node is ready
2022-09-10T10:58:48.882 INFO:teuthology.orchestra.run.smithi149.stdout:smithi149.front.sepia.ceph.com
2022-09-10T10:58:48.944 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.15.149 smithi149.front.sepia.ceph.com smithi149
2022-09-10T10:58:49.296 INFO:teuthology.provision.fog.smithi149:Deploy complete!
2022-09-10T10:58:49.541 INFO:teuthology.lock.ops:Checking smithi136.front.sepia.ceph.com
2022-09-10T10:58:49.542 INFO:teuthology.lock.ops:New key found. Updating...
2022-09-10T10:58:49.569 INFO:teuthology.lock.ops:Checking smithi149.front.sepia.ceph.com
2022-09-10T10:58:49.570 INFO:teuthology.lock.ops:New key found. Updating...
2022-09-10T10:58:49.595 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: set os type and version on server
2022-09-10T10:58:49.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com'
2022-09-10T10:58:49.878 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64
2022-09-10T10:58:49.938 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream"
2022-09-10T10:58:49.938 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8"
2022-09-10T10:58:49.939 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos"
2022-09-10T10:58:49.939 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora"
2022-09-10T10:58:49.940 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8"
2022-09-10T10:58:49.940 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8"
2022-09-10T10:58:49.941 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 8"
2022-09-10T10:58:49.941 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31"
2022-09-10T10:58:49.942 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-09-10T10:58:49.942 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/"
2022-09-10T10:58:49.943 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-09-10T10:58:49.943 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-09-10T10:58:49.943 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-09-10T10:58:49.945 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2022-09-10T10:58:49.985 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: set os type and version on server
2022-09-10T10:58:49.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com'
2022-09-10T10:58:50.244 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64
2022-09-10T10:58:50.307 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream"
2022-09-10T10:58:50.308 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8"
2022-09-10T10:58:50.309 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos"
2022-09-10T10:58:50.310 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora"
2022-09-10T10:58:50.310 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8"
2022-09-10T10:58:50.311 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8"
2022-09-10T10:58:50.311 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 8"
2022-09-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31"
2022-09-10T10:58:50.312 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-09-10T10:58:50.313 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/"
2022-09-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-09-10T10:58:50.314 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-09-10T10:58:50.315 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-09-10T10:58:50.316 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server
2022-09-10T10:58:50.446 INFO:teuthology.dispatcher.supervisor:Running job 7017749
2022-09-10T10:58:50.447 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-09-07_00:21:03-orch-wip-yuri4-testing-2022-09-06-1408-distro-default-smithi/7017749 --name yuriw-2022-09-07_00:21:03-orch-wip-yuri4-testing-2022-09-06-1408-distro-default-smithi --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} -- /home/teuthworker/archive/yuriw-2022-09-07_00:21:03-orch-wip-yuri4-testing-2022-09-06-1408-distro-default-smithi/7017749/orig.config.yaml
2022-09-10T10:58:50.459 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-09-07_00:21:03-orch-wip-yuri4-testing-2022-09-06-1408-distro-default-smithi/7017749
2022-09-10T10:58:50.460 INFO:teuthology.dispatcher.supervisor:Job PID: 29023
2022-09-10T10:58:50.460 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-09-10T22:59:48.668 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-09-10T22:59:48.811 INFO:teuthology.kill:Killing Pids: {29023}
2022-09-10T22:59:50.265 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2022-09-10T22:59:50.266 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2022-09-10T22:59:50.266 INFO:teuthology.misc:Compressing logs...
2022-09-10T22:59:50.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com'
2022-09-10T22:59:50.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com'
2022-09-10T22:59:50.514 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-09-10T22:59:50.800 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2022-09-10T22:59:50.853 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-09-10T22:59:50.853 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now
2022-09-10T22:59:50.924 INFO:teuthology.orchestra.run.smithi149.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2022-09-10T22:59:50.924 INFO:teuthology.orchestra.run.smithi149.stderr:tar: Error is not recoverable: exiting now
2022-09-10T22:59:50.933 INFO:teuthology.misc:Compressing logs...
2022-09-10T22:59:51.314 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-09-10T22:59:51.353 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-09-10T22:59:51.594 INFO:teuthology.misc:Compressing logs...
2022-09-10T23:00:18.824 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-09-10T23:00:26.065 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2022-09-10T23:00:39.571 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-09-10T23:00:51.048 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-09-10T23:00:55.628 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2022-09-10T23:01:03.968 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-09-10T23:01:17.992 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-09-10T23:01:28.283 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-09-10T23:01:31.977 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-09-10T23:01:53.751 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-09-10T23:01:54.539 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/1f6f2a3e-30f8-11ed-8431-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2022-09-10T23:02:48.224 INFO:teuthology.kill:No teuthology processes running
2022-09-10T23:02:49.623 INFO:teuthology.kill:Nuking machines: ['smithi136', 'smithi149']
2022-09-10T23:02:50.778 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-09-10T23:02:50.779 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-09-10T23:02:50.779 INFO:teuthology.kill:2022-09-10 23:02:50,777.777 INFO:teuthology.nuke:targets:
2022-09-10T23:02:50.779 INFO:teuthology.kill:  smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDydMcqGeSqfb/pZh55E2XPGBV8T2iM/c8YZs111rFtT0kaVs9jgDwDlRH/F+Dmo5C7QHdzTKXgagUO51SfVo1sW3eQmXH0qGbIH/Try4gpcNz21y0udXp3TQfFuveTwG1fdvI8wE5gkmNhBk5jCb9OMd0NZ4afX8g6YZE7viicD2D0GEgyjBsuRahfyehY0aCVW3EemnlCVC+a+jEWTCDaLLIOBwEjK6J3op845THiGememS2re4zKYcrlHizD+0XztSOgvKQTR6ML/v+6u/pX0QId+bMIU8Wp2zDQKEAd0h3m0NGD4V4E41wl2YRc5ePK8ZDsS208OgZ5jBf4N/9nGoH7K5HaK/JA1ux+Youp1RHTugDyrJZTVOcjuvz285waFziSTAq0w5kNteWkZn3Z0YJLrz/FuocUnODxSg2sJJpuLt1XXCvhZs1W80/H9afZG25ok2N2XhoDu8qjGpvn7iUON0x77XNpozyQUjDaJowXY1t8j/I4hSA0MCdwH7s=
2022-09-10T23:02:50.779 INFO:teuthology.kill:  smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDX4un3rdD28Zwqmvc5Yi1VlWVpgUW0KaTe54a5c1sYeK15VzVfKqKmCYJyalc689xBIc0vLeiQK5oPBljfDDUEom8/j3vIAI+yO8JWkn3BdC+K73Sl2ncgSBFarAqlDUafWejGk74nIVm4ARphyujTX8VmMVAZigs22gXfkgnk5J1ZYFNtMe+Bt1EUNBljNafsZucFVeSwnW//u8Cr7kJ1vvhvylE7hnthVKnWQ250gikpjqKKSZNcSbhd9spYPmm2626YW8+bonpt2W85OPWaWKiIrriFaIqXZRnTrXGCJHcV1mzsNCd0lMyEbBuNLQlhJ2mmjmhoO7+DAg54RF0JD0rJ2n+D/h5ZPZxLlNZg9lkX75fy5cuNnCl/fgy08E+CJk+ytEJvoRqeH5AE3afnQeD1P6NjA7E0S/mcQuzbxKVHoHYKBkH6gixHrFeoZoYVzaO0n7jB+EZ0kFBPFagHf5M4I6zWxX9XE5MOrmx4eCXIREDnwwekV5TjeSk5Ygc=
2022-09-10T23:02:50.780 INFO:teuthology.kill:2022-09-10 23:02:50,777.777 INFO:teuthology.nuke:Checking targets against current locks
2022-09-10T23:02:50.835 INFO:teuthology.kill:2022-09-10 23:02:50,835.835 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-09-10T23:02:50.858 INFO:teuthology.kill:2022-09-10 23:02:50,857.857 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-09-10T23:02:50.917 INFO:teuthology.kill:2022-09-10 23:02:50,916.916 INFO:teuthology.orchestra.console:Power off smithi136
2022-09-10T23:02:50.954 INFO:teuthology.kill:2022-09-10 23:02:50,953.953 INFO:teuthology.orchestra.console:Power off smithi149
2022-09-10T23:02:59.315 INFO:teuthology.kill:2022-09-10 23:02:59,315.315 INFO:teuthology.orchestra.console:Power off for smithi149 completed
2022-09-10T23:02:59.466 INFO:teuthology.kill:2022-09-10 23:02:59,465.465 INFO:teuthology.lock.ops:unlocked: smithi149.front.sepia.ceph.com
2022-09-10T23:03:03.404 INFO:teuthology.kill:2022-09-10 23:03:03,404.404 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2022-09-10T23:03:03.557 INFO:teuthology.kill:2022-09-10 23:03:03,556.556 INFO:teuthology.lock.ops:unlocked: smithi136.front.sepia.ceph.com
2022-09-10T23:05:03.927 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-09-10T23:05:04.036 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi136 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-09-10_17:34:50-rgw-quincy-release-distro-default-smithi/7025057
2022-09-10T23:05:04.037 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi149 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-09-10_17:34:50-rgw-quincy-release-distro-default-smithi/7025056