2022-02-11T17:35:24.614 INFO:root:teuthology version: 1.1.0-7cebb3f2 2022-02-11T17:35:24.747 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging 2022-02-11T17:35:24.748 INFO:teuthology.lock.ops:Updating [ubuntu@smithi060.front.sepia.ceph.com]: reset os type and version on server 2022-02-11T17:35:24.748 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2022-02-11T17:35:24.769 INFO:teuthology.lock.ops:Node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging is complete 2022-02-11T17:35:24.770 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi181.front.sepia.ceph.com' reimaging 2022-02-11T17:35:24.770 INFO:teuthology.lock.ops:Updating [ubuntu@smithi181.front.sepia.ceph.com]: reset os type and version on server 2022-02-11T17:35:24.770 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2022-02-11T17:35:24.790 INFO:teuthology.lock.ops:Node 'ubuntu@smithi181.front.sepia.ceph.com' reimaging is complete 2022-02-11T17:35:24.971 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of rhel 8.4 2022-02-11T17:35:24.990 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of rhel 8.4 2022-02-11T17:35:25.308 INFO:teuthology.orchestra.console:Power off smithi181 2022-02-11T17:35:25.321 INFO:teuthology.orchestra.console:Power off smithi060 2022-02-11T17:35:33.683 INFO:teuthology.orchestra.console:Power off for smithi181 completed 2022-02-11T17:35:33.690 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2022-02-11T17:35:33.785 INFO:teuthology.orchestra.console:Power on smithi181 2022-02-11T17:35:33.794 INFO:teuthology.orchestra.console:Power on smithi060 2022-02-11T17:35:38.042 INFO:teuthology.orchestra.console:Power on for smithi181 completed 2022-02-11T17:35:38.143 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish 2022-02-11T17:35:42.227 INFO:teuthology.orchestra.console:Power on for smithi060 completed 2022-02-11T17:35:42.329 INFO:teuthology.provision.fog.smithi060:Waiting for deploy to finish 2022-02-11T17:39:41.387 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-11T17:39:55.889 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-11T17:39:56.106 INFO:teuthology.provision.fog.smithi060:Node is ready 2022-02-11T17:39:56.122 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060.front.sepia.ceph.com 2022-02-11T17:39:56.178 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60 smithi060.front.sepia.ceph.com smithi060 2022-02-11T17:39:56.496 INFO:teuthology.provision.fog.smithi060:Deploy complete! 2022-02-11T17:40:01.909 INFO:teuthology.provision.fog.smithi181:Node is ready 2022-02-11T17:40:01.928 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com 2022-02-11T17:40:01.985 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181 2022-02-11T17:40:02.324 INFO:teuthology.provision.fog.smithi181:Deploy complete! 2022-02-11T17:40:02.502 INFO:teuthology.lock.ops:Checking smithi060.front.sepia.ceph.com 2022-02-11T17:40:02.503 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-11T17:40:02.556 INFO:teuthology.lock.ops:Checking smithi181.front.sepia.ceph.com 2022-02-11T17:40:02.556 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-11T17:40:02.574 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: set os type and version on server 2022-02-11T17:40:02.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2022-02-11T17:40:02.813 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2022-02-11T17:40:02.868 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Red Hat Enterprise Linux" 2022-02-11T17:40:02.869 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="8.4 (Ootpa)" 2022-02-11T17:40:02.869 INFO:teuthology.orchestra.run.smithi060.stdout:ID="rhel" 2022-02-11T17:40:02.869 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="fedora" 2022-02-11T17:40:02.870 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="8.4" 2022-02-11T17:40:02.870 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el8" 2022-02-11T17:40:02.870 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-02-11T17:40:02.870 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2022-02-11T17:40:02.871 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-02-11T17:40:02.871 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.redhat.com/" 2022-02-11T17:40:02.871 INFO:teuthology.orchestra.run.smithi060.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-02-11T17:40:02.871 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-11T17:40:02.871 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-02-11T17:40:02.872 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-11T17:40:02.872 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-02-11T17:40:02.872 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-02-11T17:40:02.872 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-02-11T17:40:02.874 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2022-02-11T17:40:02.891 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: set os type and version on server 2022-02-11T17:40:02.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2022-02-11T17:40:03.117 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2022-02-11T17:40:03.171 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Red Hat Enterprise Linux" 2022-02-11T17:40:03.171 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8.4 (Ootpa)" 2022-02-11T17:40:03.172 INFO:teuthology.orchestra.run.smithi181.stdout:ID="rhel" 2022-02-11T17:40:03.172 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="fedora" 2022-02-11T17:40:03.173 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8.4" 2022-02-11T17:40:03.173 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8" 2022-02-11T17:40:03.173 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-02-11T17:40:03.173 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2022-02-11T17:40:03.173 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-02-11T17:40:03.174 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.redhat.com/" 2022-02-11T17:40:03.174 INFO:teuthology.orchestra.run.smithi181.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-02-11T17:40:03.174 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-11T17:40:03.174 INFO:teuthology.orchestra.run.smithi181.stdout: 2022-02-11T17:40:03.175 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-11T17:40:03.175 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-02-11T17:40:03.175 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-02-11T17:40:03.175 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-02-11T17:40:03.176 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2022-02-11T17:40:03.261 INFO:teuthology.dispatcher.supervisor:Running job 6676883 2022-02-11T17:40:03.261 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_7cebb3f2319fd6c8340c0f7cd15a137e747fd32e/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2022-02-11_16:35:44-orch:cephadm-wip-adk-testing-2022-02-10-1637-distro-basic-smithi/6676883 --name adking-2022-02-11_16:35:44-orch:cephadm-wip-adk-testing-2022-02-10-1637-distro-basic-smithi --description orch:cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} -- /home/teuthworker/archive/adking-2022-02-11_16:35:44-orch:cephadm-wip-adk-testing-2022-02-10-1637-distro-basic-smithi/6676883/orig.config.yaml 2022-02-11T17:40:03.265 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2022-02-11_16:35:44-orch:cephadm-wip-adk-testing-2022-02-10-1637-distro-basic-smithi/6676883 2022-02-11T17:40:03.266 INFO:teuthology.dispatcher.supervisor:Job PID: 31839 2022-02-11T17:40:03.267 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-12T00:10:31.883 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-12T00:10:32.097 INFO:teuthology.kill:Killing Pids: {31839} 2022-02-12T00:10:33.303 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['host.a', 'client.0'] 2022-02-12T00:10:33.304 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.b', 'client.1'] 2022-02-12T00:10:33.304 INFO:teuthology.misc:Compressing logs... 2022-02-12T00:10:33.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2022-02-12T00:10:33.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2022-02-12T00:10:33.515 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-02-12T00:10:33.751 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2022-02-12T00:10:33.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-02-12T00:10:33.782 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2022-02-12T00:10:33.818 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-02-12T00:10:33.818 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2022-02-12T00:10:33.820 INFO:teuthology.misc:Compressing logs... 2022-02-12T00:10:34.042 INFO:teuthology.misc:Compressing logs... 2022-02-12T00:10:49.563 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-02-12T00:10:52.778 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-mon.smithi181.log: file size changed while zipping 2022-02-12T00:10:57.315 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-02-12T00:11:03.831 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-02-12T00:11:15.496 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-02-12T00:11:21.088 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-02-12T00:11:31.280 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-02-12T00:11:36.534 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-02-12T00:11:54.205 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-02-12T00:12:03.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-mon.smithi060.log: file size changed while zipping 2022-02-12T00:12:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/1e3ffaac-8b63-11ec-8c35-001a4aab830c/ceph-mgr.smithi060.wfgtum.log: file size changed while zipping 2022-02-12T00:12:38.883 INFO:teuthology.kill:No teuthology processes running 2022-02-12T00:12:40.087 INFO:teuthology.kill:Nuking machines: ['smithi060', 'smithi181'] 2022-02-12T00:12:41.061 INFO:teuthology.kill:2022-02-12 00:12:41,061.061 INFO:teuthology.nuke:targets: 2022-02-12T00:12:41.062 INFO:teuthology.kill: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7ldLtb0UkhX0DYLzWNH/gnhMB6n2D3qfkpTk5muPtp2wcgO79yWo4gxo9G92B59ML5B5DYBSW7e61yOOGuqXI1h30BOys0O3P0RrAQkO91K/i78VeyLNIOhtilqDsQHT1PoXjpBkWFfUsufkcVFTCW/t65mamZBVVkuqTwk41+YjH3a8A8XX0yuuh+JFjZqmtK0f0k27FXhdwn8+H29y65pPGSzHxEKfufKJaq0m5K12OyI6H800C7QxaaXEZPlGKsOIyaGJsxnwxDaC/6jTDrK/gnvWzH0/VcZqVgqLpYeX0NrL79OZdK5IYWqLhus/2LObPnSZ14sP9EDm0EUBGkgYv/RRyRR6F/CxR+eth6MiVe2kRbg6Q2DPFQuO1XrlzE2SKrJUfAq5GDgF3y3PDajI3hh58XMvfOQ74v+lzFPT6demNpVZp1EUUHk36zaRPqgbVMav0k3drPgZG5HpXKIPlpPi86jY389aVcLMNsRhHCm4w5R+/5SMjTf0O5fU= 2022-02-12T00:12:41.063 INFO:teuthology.kill: smithi181.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCl/inyi9hwPK0FFa+b9ChZ/lqk1DBCpZYen152HIywMs9GHG5vPQod7rim0DLJwwjUo3PihGFjEs1tdwgW47T9jKeBJxIPRcokAsp0gEwJ84j/q1RlrZi5GiA8db8eCAD+i3Li7/k6DR6N4bdulO1wizmfQdq8ARP/EgVIayXN1//fszCYcVku+lMroOlbi+0AOH87TBo5fwnS1OKFJkYLe3vrsKPKtJhbf34Gn8q82+Hn+VnAwCoYhvN0McXpPfE58pkOgK0h9nWiN6UdgJwHOAntMIClTtB4l+tD6lK0d245qEW7ExAaJS35HzDZA9DyLzOgMHwo28kZTThgkGGtaBgQWtTe2lvEAU6Ug1v4WEmi4S7QTRrCBeJbLw6mvBNgDr6nLdzowtYRkB0BOunimt4OLM9PG84hj2VZDAY9jsJePgjXeVxRxXS2CZ3QzNzGyqLl5TDRAh2NZMs/MazWO5PvvZhK0ntP8b5FX0LX2ngUFolrs/54ffQntZHKnIU= 2022-02-12T00:12:41.088 INFO:teuthology.kill:2022-02-12 00:12:41,088.088 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-12T00:12:41.091 INFO:teuthology.kill:2022-02-12 00:12:41,091.091 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-12T00:12:41.170 INFO:teuthology.kill:2022-02-12 00:12:41,170.170 INFO:teuthology.orchestra.console:Power off smithi060 2022-02-12T00:12:41.184 INFO:teuthology.kill:2022-02-12 00:12:41,183.183 INFO:teuthology.orchestra.console:Power off smithi181 2022-02-12T00:12:49.541 INFO:teuthology.kill:2022-02-12 00:12:49,541.541 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2022-02-12T00:12:49.692 INFO:teuthology.kill:2022-02-12 00:12:49,691.691 INFO:teuthology.lock.ops:unlocked: smithi060.front.sepia.ceph.com 2022-02-12T00:12:53.661 INFO:teuthology.kill:2022-02-12 00:12:53,661.661 INFO:teuthology.orchestra.console:Power off for smithi181 completed 2022-02-12T00:12:53.818 INFO:teuthology.kill:2022-02-12 00:12:53,818.818 INFO:teuthology.lock.ops:unlocked: smithi181.front.sepia.ceph.com 2022-02-12T00:14:54.179 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-12T00:14:54.288 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi060 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-11_19:41:29-rados-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677429 2022-02-12T00:14:54.289 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi181 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-11_19:41:29-rados-wip-yuriw-quincy-RC-2.11.22-distro-default-smithi/6677429