2022-02-16T13:25:28.263 INFO:root:teuthology version: 1.1.0-65c38a8a 2022-02-16T13:25:28.424 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi099.front.sepia.ceph.com' reimaging 2022-02-16T13:25:28.425 INFO:teuthology.lock.ops:Updating [ubuntu@smithi099.front.sepia.ceph.com]: reset os type and version on server 2022-02-16T13:25:28.425 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2022-02-16T13:25:28.443 INFO:teuthology.lock.ops:Node 'ubuntu@smithi099.front.sepia.ceph.com' reimaging is complete 2022-02-16T13:25:28.444 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging 2022-02-16T13:25:28.444 INFO:teuthology.lock.ops:Updating [ubuntu@smithi100.front.sepia.ceph.com]: reset os type and version on server 2022-02-16T13:25:28.445 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2022-02-16T13:25:28.489 INFO:teuthology.lock.ops:Node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging is complete 2022-02-16T13:25:28.653 INFO:teuthology.provision.fog.smithi099:Scheduling deploy of centos 8.stream 2022-02-16T13:25:28.695 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of centos 8.stream 2022-02-16T13:25:29.082 INFO:teuthology.orchestra.console:Power off smithi099 2022-02-16T13:25:29.191 INFO:teuthology.orchestra.console:Power off smithi100 2022-02-16T13:25:41.568 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2022-02-16T13:25:41.664 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2022-02-16T13:25:41.670 INFO:teuthology.orchestra.console:Power on smithi099 2022-02-16T13:25:41.766 INFO:teuthology.orchestra.console:Power on smithi100 2022-02-16T13:25:50.039 INFO:teuthology.orchestra.console:Power on for smithi099 completed 2022-02-16T13:25:50.124 INFO:teuthology.orchestra.console:Power on for smithi100 completed 2022-02-16T13:25:50.141 INFO:teuthology.provision.fog.smithi099:Waiting for deploy to finish 2022-02-16T13:25:50.225 INFO:teuthology.provision.fog.smithi100:Waiting for deploy to finish 2022-02-16T13:28:21.392 ERROR:teuthology.orchestra.connection:Error authenticating with smithi100.front.sepia.ceph.com: Authentication failed. 2022-02-16T13:29:58.491 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-16T13:30:23.954 INFO:teuthology.provision.fog.smithi099:Node is ready 2022-02-16T13:30:23.976 INFO:teuthology.orchestra.run.smithi099.stdout:smithi099.front.sepia.ceph.com 2022-02-16T13:30:24.035 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99 smithi099.front.sepia.ceph.com smithi099 2022-02-16T13:30:24.409 INFO:teuthology.provision.fog.smithi099:Deploy complete! 2022-02-16T13:30:34.737 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-16T13:30:34.979 INFO:teuthology.provision.fog.smithi100:Node is ready 2022-02-16T13:30:35.039 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100.front.sepia.ceph.com 2022-02-16T13:30:35.099 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.15.100 smithi100.front.sepia.ceph.com smithi100 2022-02-16T13:30:35.457 INFO:teuthology.provision.fog.smithi100:Deploy complete! 2022-02-16T13:30:35.603 INFO:teuthology.lock.ops:Checking smithi099.front.sepia.ceph.com 2022-02-16T13:30:35.604 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-16T13:30:35.624 INFO:teuthology.lock.ops:Checking smithi100.front.sepia.ceph.com 2022-02-16T13:30:35.624 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-16T13:30:35.642 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: set os type and version on server 2022-02-16T13:30:35.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2022-02-16T13:30:35.836 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2022-02-16T13:30:35.895 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2022-02-16T13:30:35.895 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8" 2022-02-16T13:30:35.896 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2022-02-16T13:30:35.896 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2022-02-16T13:30:35.896 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2022-02-16T13:30:35.897 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2022-02-16T13:30:35.897 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-16T13:30:35.897 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2022-02-16T13:30:35.897 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-16T13:30:35.898 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2022-02-16T13:30:35.898 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-16T13:30:35.898 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-16T13:30:35.898 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-16T13:30:35.900 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2022-02-16T13:30:35.918 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: set os type and version on server 2022-02-16T13:30:35.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2022-02-16T13:30:36.118 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2022-02-16T13:30:36.178 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2022-02-16T13:30:36.178 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8" 2022-02-16T13:30:36.179 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2022-02-16T13:30:36.179 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2022-02-16T13:30:36.179 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8" 2022-02-16T13:30:36.179 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8" 2022-02-16T13:30:36.180 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-16T13:30:36.180 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2022-02-16T13:30:36.180 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-16T13:30:36.180 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2022-02-16T13:30:36.180 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-16T13:30:36.181 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-16T13:30:36.181 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-16T13:30:36.181 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2022-02-16T13:30:36.315 INFO:teuthology.dispatcher.supervisor:Running job 6688426 2022-02-16T13:30:36.316 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_65c38a8ac6d6694ef8ab9ff2325a7faf73afdc22/virtualenv/bin/teuthology -v --owner scheduled_ozeneva@teuthology --archive /home/teuthworker/archive/ozeneva-2022-02-16_13:20:03-rgw-wip-omri-tracing-compiled-distro-basic-smithi/6688426 --name ozeneva-2022-02-16_13:20:03-rgw-wip-omri-tracing-compiled-distro-basic-smithi --description rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} -- /home/teuthworker/archive/ozeneva-2022-02-16_13:20:03-rgw-wip-omri-tracing-compiled-distro-basic-smithi/6688426/orig.config.yaml 2022-02-16T13:30:36.319 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ozeneva-2022-02-16_13:20:03-rgw-wip-omri-tracing-compiled-distro-basic-smithi/6688426 2022-02-16T13:30:36.319 INFO:teuthology.dispatcher.supervisor:Job PID: 4691 2022-02-16T13:30:36.320 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-16T20:00:47.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-16T20:00:47.342 INFO:teuthology.kill:Killing Pids: {4691} 2022-02-16T20:00:48.630 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-02-16T20:00:48.631 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-02-16T20:00:48.632 INFO:teuthology.misc:Compressing logs... 2022-02-16T20:00:48.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2022-02-16T20:00:48.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2022-02-16T20:00:49.250 INFO:teuthology.misc:Compressing logs... 2022-02-16T20:00:53.473 INFO:teuthology.misc:Compressing logs... 2022-02-16T20:01:26.840 INFO:teuthology.kill:No teuthology processes running 2022-02-16T20:01:27.983 INFO:teuthology.kill:Nuking machines: ['smithi099', 'smithi100'] 2022-02-16T20:01:28.900 INFO:teuthology.kill:2022-02-16 20:01:28,900.900 INFO:teuthology.nuke:targets: 2022-02-16T20:01:28.901 INFO:teuthology.kill: smithi099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdSRarZU+Qtgt7LzbRCFHLWYntu26wMDJ2acv8tOqLMTNRKqPSBsnqOUE87lqsNgCPmibY7zWtVMPxyPyHpyLY9rMPCDGn0wtBCdrnRJ+nuWg7nTOs9W1oorZmhpYLPYrlj285f+AGu0Kwui9tdkqfxJQTiUyWH0ARRCmorZRd8h/10X4D6y1S+mTQYzpXwt8p36yQbQo+tE1w2bw2K7hkfkUOS09czQxmm5Ii+/UBtcSz33Zw22stbtxBJJ/m+l60C5KMhnt1ZplJH8uvPlxcpCHHj2kFk95S6JrncsghSudXUhMgbHPOlhqaIPX7Ukt1h3KdhReRfNCPfZsARfCYcMKwKdbiok7arar+2M8K61GjKAT7mKE2hjc4aUk8CRX0Tx2GSofm7zjU6r0GyI5ATRiFAJISH9iABtcvZMguZPQZNpHP76L8AlEdA981yrNGtLIyW3Lj3GWfgjifJHv95TVuuTuEOi8uPyQfDOwSmC1srXnK8bv8YdmE5YPo+zk= 2022-02-16T20:01:28.901 INFO:teuthology.kill: smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC78NN/jLQ86Lyd3l8YJFFYR/nrxlwHAG2kXed5UOnZOBTFseTiG+WSyt58VUxU9R1JjlQ23UnluoixmqHckiurBT34+3TM3FuGSbGRA7HTUEmtP7H6WVxHnJL25ty7Jum8VOv4yb5DuK5Rvw/mUxikzS2xF5jB8eHw8kPc//+UixfNXhnCUVqTnqPK5pBR/a62GiaH0BPR/Ig61tYDDhxKW14yvxtvmPTlisH6tkXlJi3F3PgjDCBiUcSjvrsk3/z2spRf0O8yaefHHClk2hDn2thm7IRnfk1/27FtJ7vuhQreNoDgeKfpdQEAiry/YZ6hnEsvqR9XWV1/EVVrlufnkFz3eQ6toKzUUKNZJ5aYlfI2jg+v77tyGfhh42IjYOFRN50/rz3MPP+rtX6mQHYT7kbEm30nV7LBO5GyrYsL79rieYCU5nn4QmHnv+VLQMH4iTUu9O7Bv1AWwDcWec96ICIItSL6Zf1dM7hXS+6DzaEl0fddf3s+Y4fSXk9xDz8= 2022-02-16T20:01:28.928 INFO:teuthology.kill:2022-02-16 20:01:28,928.928 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-16T20:01:28.931 INFO:teuthology.kill:2022-02-16 20:01:28,931.931 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-16T20:01:29.000 INFO:teuthology.kill:2022-02-16 20:01:29,000.000 INFO:teuthology.orchestra.console:Power off smithi100 2022-02-16T20:01:29.015 INFO:teuthology.kill:2022-02-16 20:01:29,015.015 INFO:teuthology.orchestra.console:Power off smithi099 2022-02-16T20:01:37.376 INFO:teuthology.kill:2022-02-16 20:01:37,375.375 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2022-02-16T20:01:37.521 INFO:teuthology.kill:2022-02-16 20:01:37,520.520 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com 2022-02-16T20:01:41.492 INFO:teuthology.kill:2022-02-16 20:01:41,491.491 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2022-02-16T20:01:41.642 INFO:teuthology.kill:2022-02-16 20:01:41,642.642 INFO:teuthology.lock.ops:unlocked: smithi099.front.sepia.ceph.com 2022-02-16T20:03:41.860 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-16T20:03:41.958 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi099 but it was locked by another job: /home/teuthworker/archive/nojha-2022-02-16_19:40:57-rados-wip-mgr-fixes-pacific-gil-distro-basic-smithi/6689247 2022-02-16T20:03:41.958 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi100 but it was locked by another job: /home/teuthworker/archive/nojha-2022-02-16_19:40:57-rados-wip-mgr-fixes-pacific-gil-distro-basic-smithi/6689243