2021-02-10T03:11:28.404 INFO:root:teuthology version: 1.1.0-17c09965 2021-02-10T03:11:28.527 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging 2021-02-10T03:11:28.528 INFO:teuthology.lock.ops:Updating [ubuntu@smithi090.front.sepia.ceph.com]: reset os type and version on server 2021-02-10T03:11:28.528 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2021-02-10T03:11:28.547 INFO:teuthology.lock.ops:Node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging is complete 2021-02-10T03:11:28.547 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging 2021-02-10T03:11:28.548 INFO:teuthology.lock.ops:Updating [ubuntu@smithi114.front.sepia.ceph.com]: reset os type and version on server 2021-02-10T03:11:28.548 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2021-02-10T03:11:28.570 INFO:teuthology.lock.ops:Node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging is complete 2021-02-10T03:11:28.750 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of centos 8.2 2021-02-10T03:11:28.761 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of centos 8.2 2021-02-10T03:11:29.078 INFO:teuthology.orchestra.console:Power off smithi090 2021-02-10T03:11:29.105 INFO:teuthology.orchestra.console:Power off smithi114 2021-02-10T03:11:29.287 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-02-10T03:11:29.312 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2021-02-10T03:11:29.391 INFO:teuthology.orchestra.console:Power on smithi114 2021-02-10T03:11:29.414 INFO:teuthology.orchestra.console:Power on smithi090 2021-02-10T03:11:33.692 INFO:teuthology.orchestra.console:Power on for smithi090 completed 2021-02-10T03:11:33.794 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish 2021-02-10T03:11:37.793 INFO:teuthology.orchestra.console:Power on for smithi114 completed 2021-02-10T03:11:37.894 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish 2021-02-10T03:16:09.525 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-10T03:16:09.819 INFO:teuthology.provision.fog.smithi114:Node is ready 2021-02-10T03:16:09.834 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114.front.sepia.ceph.com 2021-02-10T03:16:09.888 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114 2021-02-10T03:16:10.206 INFO:teuthology.provision.fog.smithi114:Deploy complete! 2021-02-10T03:16:53.800 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-10T03:16:54.147 INFO:teuthology.provision.fog.smithi090:Node is ready 2021-02-10T03:16:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com 2021-02-10T03:16:54.217 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090 2021-02-10T03:16:54.562 INFO:teuthology.provision.fog.smithi090:Deploy complete! 2021-02-10T03:16:54.694 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com 2021-02-10T03:16:54.695 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-10T03:16:54.712 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com 2021-02-10T03:16:54.712 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-10T03:16:54.727 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server 2021-02-10T03:16:54.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T03:16:54.989 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Linux" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8 (Core)" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-10T03:16:55.043 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.centos.org/" 2021-02-10T03:16:55.044 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-10T03:16:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-02-10T03:16:55.044 INFO:teuthology.orchestra.run.smithi090.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-10T03:16:55.044 INFO:teuthology.orchestra.run.smithi090.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-10T03:16:55.044 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-10T03:16:55.044 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-10T03:16:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: 2021-02-10T03:16:55.045 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2021-02-10T03:16:55.060 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server 2021-02-10T03:16:55.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T03:16:55.334 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2021-02-10T03:16:55.388 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Linux" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8 (Core)" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.centos.org/" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-02-10T03:16:55.389 INFO:teuthology.orchestra.run.smithi114.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-10T03:16:55.390 INFO:teuthology.orchestra.run.smithi114.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-10T03:16:55.390 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-10T03:16:55.390 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-10T03:16:55.390 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-02-10T03:16:55.390 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2021-02-10T03:16:55.470 INFO:teuthology.dispatcher.supervisor:Running job 5871729 2021-02-10T03:16:55.470 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_17c099650191e7aa47a7d9f502facca74f9b4d44/virtualenv/bin/teuthology -v --owner scheduled_sseshasa@teuthology --archive /home/teuthworker/archive/sseshasa-2021-02-09_10:39:09-rados-wip-sseshasa-testing-2021-02-02-1136-distro-basic-smithi/5871729 --name sseshasa-2021-02-09_10:39:09-rados-wip-sseshasa-testing-2021-02-02-1136-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} -- /home/teuthworker/archive/sseshasa-2021-02-09_10:39:09-rados-wip-sseshasa-testing-2021-02-02-1136-distro-basic-smithi/5871729/orig.config.yaml 2021-02-10T03:16:55.474 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sseshasa-2021-02-09_10:39:09-rados-wip-sseshasa-testing-2021-02-02-1136-distro-basic-smithi/5871729 2021-02-10T03:16:55.475 INFO:teuthology.dispatcher.supervisor:Job PID: 30257 2021-02-10T03:16:55.475 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-10T15:17:11.170 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-10T15:17:11.299 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-02-10T15:17:11.299 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-02-10T15:17:11.299 INFO:teuthology.misc:Compressing logs... 2021-02-10T15:17:11.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T15:17:11.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T15:17:12.029 INFO:teuthology.misc:Compressing logs... 2021-02-10T15:17:12.877 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-02-10T15:17:14.558 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-02-10T15:19:07.549 INFO:teuthology.misc:Compressing logs... 2021-02-10T15:19:10.498 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/ceph.log: file size changed while zipping 2021-02-10T15:20:31.688 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2021-02-10T15:20:39.575 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2021-02-10T15:20:56.114 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2021-02-10T15:21:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2021-02-10T15:22:46.137 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2021-02-10T15:22:57.962 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2021-02-10T15:23:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2021-02-10T15:24:29.493 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2021-02-10T15:24:33.073 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph.log: file size changed while zipping 2021-02-10T15:25:08.036 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2021-02-10T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/ceph.audit.log: file size changed while zipping 2021-02-10T15:25:13.248 INFO:teuthology.orchestra.run.smithi090.stderr:tar: ./ceph-osd.0.log: file changed as we read it 2021-02-10T15:25:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:tar: ./ceph-osd.3.log: file changed as we read it 2021-02-10T15:26:58.176 INFO:teuthology.orchestra.run.smithi090.stderr:tar: ./ceph-osd.1.log: file changed as we read it 2021-02-10T15:26:58.191 INFO:teuthology.orchestra.run.smithi090.stderr:tar: .: file changed as we read it 2021-02-10T15:27:00.616 INFO:teuthology.orchestra.run.smithi114.stderr:tar: ./ceph-osd.5.log: file changed as we read it 2021-02-10T15:27:53.703 INFO:teuthology.orchestra.run.smithi114.stderr:tar: ./ceph-osd.7.log: file changed as we read it 2021-02-10T15:27:55.758 INFO:teuthology.kill:Killing Pids: {30257} 2021-02-10T15:27:57.458 INFO:teuthology.kill:Nuking machines: ['smithi090', 'smithi114'] 2021-02-10T15:27:58.242 INFO:teuthology.kill:2021-02-10 15:27:58,242.242 INFO:teuthology.nuke:targets: 2021-02-10T15:27:58.243 INFO:teuthology.kill: smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSHZpmXxc6jd2UeU+Zt5SGMkCTcIboqXtv841NQixdW3+l9nEL4cwWI6bH4vmlwb/EL4ZjatVbTitKrrda70OG0jv38NU6g195pnEuqdi4Oc9u4Mppz9175Bi5ar4jxEK6yizeawwWrxKug98rHoK8vO8BZFDL/QRhglPDOIif28ilLsopA+GDqeGgfnf4z5iOht64n6saNrB0xdEARHhJZ0pMYkW4k2w82dzrVH5gHWBiFg+eDJ6rNYWV+ezouj4j/kHhQGSAoQQHGN/o9nOgEsgxhHHzZHAcX2acMhejy0zyKmLA4phqnua6R9U/rWfotCZsXr0opIHNXCiEvaIY/0WpTw692IQEPK3BlQhfmyxv6rcJwcvtsGY3W9Ib26SsTWYRCDRPbfBZd6WxAvVaDJXfmQHW3U4N1WCj1v+c6UvyAeWgTOHeRfr5NR21YWtnQRQ6C/XiNkdASP3wnRAU/703LpC3XMd0jRhrXyEaMbyLPJYP0xjMKP6Zczfu/6s= 2021-02-10T15:27:58.243 INFO:teuthology.kill: smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVqLeZSFhmmtU+uQLEAl2IpGQf5pDhQTsbXmCikJWFEz3oslzsyH+kDbgEKegWJEK3rQyiUXBY5J4BHcHIACSWDtuOlRpaQKXYK0KUztcNeIj7hhj3CYWJukYxK1ar3SRY5Xnyyc2OJd4gFuto3ZHYM1nvd/VrN9TirUB98n1s/FUOV2T8bSWGMWIJg+triBsGzen+Vdr01pcaLHgeQJ158QccFI0peJ2nrMf3HG+PkpYhw8H7Wl1vGC65PyN/kFJ4NmphyOe9hkOOB5ZujebkECohE3cDaGDvwKOMLMW4hGNbBhhtZC0iOzbjnM+47yRkCZ5uNqPQau5dxMmxThusEUB7OxCRG7GSG8uzMZ2AKDVtJcwvj6y1pFIvaI/3WCzbi7AAikwnEQQeKTB4k2rFap5HbMpo/Gu7W8hTDh/Etr+aSQvgZAPOnsl23u5ZcSD5MumHqEN25MFYZDXszhG3r2Z7i2c+0rMCuHsqXXcG6k6AxJLgI/9nIRTuXrR8vRU= 2021-02-10T15:27:58.265 INFO:teuthology.kill:2021-02-10 15:27:58,265.265 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-10T15:27:58.267 INFO:teuthology.kill:2021-02-10 15:27:58,267.267 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-10T15:27:58.326 INFO:teuthology.kill:2021-02-10 15:27:58,326.326 INFO:teuthology.orchestra.console:Power off smithi114 2021-02-10T15:27:58.339 INFO:teuthology.kill:2021-02-10 15:27:58,339.339 INFO:teuthology.orchestra.console:Power off smithi090 2021-02-10T15:28:10.815 INFO:teuthology.kill:2021-02-10 15:28:10,814.814 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-02-10T15:28:10.821 INFO:teuthology.kill:2021-02-10 15:28:10,821.821 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2021-02-10T15:28:10.962 INFO:teuthology.kill:2021-02-10 15:28:10,961.961 INFO:teuthology.lock.ops:unlocked smithi114.front.sepia.ceph.com 2021-02-10T15:28:10.965 INFO:teuthology.kill:2021-02-10 15:28:10,965.965 INFO:teuthology.lock.ops:unlocked smithi090.front.sepia.ceph.com 2021-02-10T15:30:11.179 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-10T15:30:11.341 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-10T15:30:11.341 INFO:teuthology.nuke:Checking targets against current locks 2021-02-10T15:30:11.427 INFO:teuthology.nuke:Not nuking smithi114.front.sepia.ceph.com because description doesn't match 2021-02-10T15:30:11.427 INFO:teuthology.nuke:Not nuking smithi090.front.sepia.ceph.com because description doesn't match