2021-01-06T22:17:06.056 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-06T22:17:06.058 DEBUG:teuthology.run:Teuthology command: teuthology --block --lock --verbose --description fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2021-01-06_20:20:52-fs-wip-pdonnell-testing-20210105.221014-distro-basic-smithi/5760126 --name pdonnell-2021-01-06_20:20:52-fs-wip-pdonnell-testing-20210105.221014-distro-basic-smithi -- /tmp/teuthology-worker.1gfdxr3w.tmp 2021-01-06T22:17:06.101 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-06T22:17:06.164 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2021-01-06_20:20:52-fs-wip-pdonnell-testing-20210105.221014-distro-basic-smithi/5760126 branch: wip-pdonnell-testing-20210105.221014 description: fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} email: pdonnell@redhat.com first_in_suite: false job_id: '5760126' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2021-01-06_20:20:52-fs-wip-pdonnell-testing-20210105.221014-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: '8.2' overrides: admin_socket: branch: wip-pdonnell-testing-20210105.221014 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d0ed162b51928c50f20cee111f8292828eda755e ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: disabled: true install: ceph: sha1: d0ed162b51928c50f20cee111f8292828eda755e selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20210105.221014 sha1: d0ed162b51928c50f20cee111f8292828eda755e owner: scheduled_pdonnell@teuthology priority: 69 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: d0ed162b51928c50f20cee111f8292828eda755e sleep_before_teardown: 0 suite: fs suite_branch: wip-pdonnell-testing-20210105.221014 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210105.221014/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: d0ed162b51928c50f20cee111f8292828eda755e tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_failover teuthology_branch: master timestamp: 2021-01-06_20:20:52 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.15451 2021-01-06T22:17:06.164 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210105.221014/qa; will attempt to use it 2021-01-06T22:17:06.165 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20210105.221014/qa/tasks 2021-01-06T22:17:06.165 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-01-06T22:17:06.167 INFO:teuthology.task.internal:Checking packages... 2021-01-06T22:17:06.187 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'd0ed162b51928c50f20cee111f8292828eda755e' 2021-01-06T22:17:06.188 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-01-06T22:17:06.188 INFO:teuthology.packaging:ref: None 2021-01-06T22:17:06.188 INFO:teuthology.packaging:tag: None 2021-01-06T22:17:06.189 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20210105.221014 2021-01-06T22:17:06.189 INFO:teuthology.packaging:sha1: d0ed162b51928c50f20cee111f8292828eda755e 2021-01-06T22:17:06.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20210105.221014 2021-01-06T22:17:06.359 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-8719.gd0ed162b 2021-01-06T22:17:06.359 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-01-06T22:17:06.370 INFO:teuthology.task.internal:no buildpackages task found 2021-01-06T22:17:06.371 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2021-01-06T22:17:06.384 INFO:teuthology.task.internal.lock_machines:Locking machines... 2021-01-06T22:17:06.385 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-06T22:17:06.445 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:17:16.466 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:17:26.482 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:17:36.499 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:17:46.514 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:17:56.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:18:06.550 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:18:16.566 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:18:26.584 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:18:36.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2021-01-06T22:18:46.619 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2021-01-06_20:20:52-fs-wip-pdonnell-testing-20210105.221014-distro-basic-smithi/5760126'} 2021-01-06T22:18:46.639 DEBUG:teuthology.lock.ops:locked ubuntu@smithi002.front.sepia.ceph.com, ubuntu@smithi098.front.sepia.ceph.com 2021-01-06T22:18:46.698 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f035fafdb70>, signals=[15]) 2021-01-06T22:18:46.907 INFO:teuthology.provision.fog.smithi098:Scheduling deploy of centos 8.2 2021-01-06T22:18:46.910 INFO:teuthology.provision.fog.smithi002:Scheduling deploy of centos 8.2 2021-01-06T22:18:47.217 INFO:teuthology.orchestra.console:Power off smithi002 2021-01-06T22:18:47.218 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-01-06T22:18:47.232 INFO:teuthology.orchestra.console:Power off smithi098 2021-01-06T22:18:47.232 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-01-06T22:18:47.241 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-06T22:18:47.252 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-06T22:18:47.364 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2021-01-06T22:18:47.375 INFO:teuthology.orchestra.console:Power off for smithi098 completed 2021-01-06T22:18:47.465 INFO:teuthology.orchestra.console:Power on smithi002 2021-01-06T22:18:47.466 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-01-06T22:18:47.476 INFO:teuthology.orchestra.console:Power on smithi098 2021-01-06T22:18:47.477 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-01-06T22:18:47.488 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-06T22:18:47.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-06T22:18:51.515 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-06T22:18:51.554 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-06T22:18:51.765 INFO:teuthology.orchestra.console:Power on for smithi098 completed 2021-01-06T22:18:51.866 INFO:teuthology.provision.fog.smithi098:Waiting for deploy to finish 2021-01-06T22:18:55.634 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-06T22:18:55.843 INFO:teuthology.orchestra.console:Power on for smithi002 completed 2021-01-06T22:18:55.948 INFO:teuthology.provision.fog.smithi002:Waiting for deploy to finish 2021-01-06T22:21:11.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:21.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:22.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:30.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:32.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:39.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:41.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:48.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:50.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:57.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:21:59.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:22:08.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:22:17.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:22:26.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:22:50.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:22:59.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:23:00.896 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-01-06T22:23:00.897 DEBUG:teuthology.orchestra.run.smithi098:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2021-01-06T22:23:03.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:23:03.631 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-01-06T22:23:03.631 DEBUG:teuthology.orchestra.run.smithi002:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2021-01-06T22:23:03.978 INFO:teuthology.provision.fog.smithi002:Node is ready 2021-01-06T22:23:03.978 DEBUG:teuthology.orchestra.run.smithi002:> hostname 2021-01-06T22:23:03.994 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002.front.sepia.ceph.com 2021-01-06T22:23:03.994 DEBUG:teuthology.orchestra.run.smithi002:> grep smithi002.front.sepia.ceph.com /etc/hosts 2021-01-06T22:23:04.051 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2 smithi002.front.sepia.ceph.com smithi002 2021-01-06T22:23:04.051 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname smithi002 2021-01-06T22:23:04.171 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i -e 's/smithi002.front.sepia.ceph.com/smithi002/g' /etc/hosts 2021-01-06T22:23:04.199 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i -e 's/172.21.15.2/172.21.15.2/g' /etc/hosts 2021-01-06T22:23:04.265 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i -e 's/smithi002.front.sepia.ceph.com/smithi002/g' /etc/hostname 2021-01-06T22:23:04.330 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname smithi002 2021-01-06T22:23:04.396 INFO:teuthology.provision.fog.smithi002:Deploy complete! 2021-01-06T22:23:04.397 DEBUG:teuthology.parallel:result is None 2021-01-06T22:23:21.113 INFO:teuthology.provision.fog.smithi098:Node is ready 2021-01-06T22:23:21.114 DEBUG:teuthology.orchestra.run.smithi098:> hostname 2021-01-06T22:23:21.133 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098.front.sepia.ceph.com 2021-01-06T22:23:21.133 DEBUG:teuthology.orchestra.run.smithi098:> grep smithi098.front.sepia.ceph.com /etc/hosts 2021-01-06T22:23:21.189 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98 smithi098.front.sepia.ceph.com smithi098 2021-01-06T22:23:21.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname smithi098 2021-01-06T22:23:21.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i -e 's/smithi098.front.sepia.ceph.com/smithi098/g' /etc/hosts 2021-01-06T22:23:21.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i -e 's/172.21.15.98/172.21.15.98/g' /etc/hosts 2021-01-06T22:23:21.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i -e 's/smithi098.front.sepia.ceph.com/smithi098/g' /etc/hostname 2021-01-06T22:23:21.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname smithi098 2021-01-06T22:23:21.544 INFO:teuthology.provision.fog.smithi098:Deploy complete! 2021-01-06T22:23:21.545 DEBUG:teuthology.parallel:result is None 2021-01-06T22:23:21.711 INFO:teuthology.lock.ops:Checking smithi002.front.sepia.ceph.com 2021-01-06T22:23:21.712 INFO:teuthology.lock.ops:New key found. Updating... 2021-01-06T22:23:21.728 INFO:teuthology.lock.ops:Checking smithi098.front.sepia.ceph.com 2021-01-06T22:23:21.729 INFO:teuthology.lock.ops:New key found. Updating... 2021-01-06T22:23:21.743 INFO:teuthology.lock.ops:Updating [smithi002.front.sepia.ceph.com]: set os type and version on server 2021-01-06T22:23:21.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-06T22:23:21.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:23:21.813 DEBUG:teuthology.orchestra.run.smithi002:> true 2021-01-06T22:23:21.970 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2021-01-06T22:23:22.024 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2021-01-06T22:23:22.026 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2021-01-06T22:23:22.079 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Linux" 2021-01-06T22:23:22.080 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8 (Core)" 2021-01-06T22:23:22.080 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2021-01-06T22:23:22.081 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2021-01-06T22:23:22.081 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8" 2021-01-06T22:23:22.082 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2021-01-06T22:23:22.082 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-01-06T22:23:22.082 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2021-01-06T22:23:22.083 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-01-06T22:23:22.083 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.centos.org/" 2021-01-06T22:23:22.083 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-01-06T22:23:22.084 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:22.084 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-01-06T22:23:22.085 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-01-06T22:23:22.085 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-01-06T22:23:22.085 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-01-06T22:23:22.086 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:22.088 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2021-01-06T22:23:22.105 INFO:teuthology.lock.ops:Updating [smithi098.front.sepia.ceph.com]: set os type and version on server 2021-01-06T22:23:22.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-06T22:23:22.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:23:22.172 DEBUG:teuthology.orchestra.run.smithi098:> true 2021-01-06T22:23:22.319 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2021-01-06T22:23:22.373 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2021-01-06T22:23:22.373 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2021-01-06T22:23:22.427 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Linux" 2021-01-06T22:23:22.519 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="8 (Core)" 2021-01-06T22:23:22.519 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2021-01-06T22:23:22.519 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2021-01-06T22:23:22.519 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="8" 2021-01-06T22:23:22.520 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el8" 2021-01-06T22:23:22.520 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-01-06T22:23:22.520 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2021-01-06T22:23:22.520 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-01-06T22:23:22.520 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.centos.org/" 2021-01-06T22:23:22.521 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-01-06T22:23:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:23:22.521 INFO:teuthology.orchestra.run.smithi098.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-01-06T22:23:22.521 INFO:teuthology.orchestra.run.smithi098.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-01-06T22:23:22.521 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-01-06T22:23:22.522 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-01-06T22:23:22.522 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:23:22.523 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2021-01-06T22:23:22.537 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2021-01-06T22:23:22.573 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqNPudaNf3ChSMTP5rsF09oGaC0KUCv5DskdivhYLMJpUHHTqoIFZyQMI44jC8gWnQgtTw7vCwc4P75kvsKuK0VjOUyPsQiu0/F61Dg8DUCxexIp7p+x46oKVQIzpr/N/VNoAippSM31vmplMELh3AO6vKpKhWjiyJwXSUFQeezjMNdypUt+B8zx0PvVeFN6ndMlLsU71O3P8exiWs40pr3rGgfsc350g38+TjfMAJHIdAPeaV7ol56Eo1vNWPhy9V4DuqinydQbDCQd/k1VQKYmPyqdotiWysqLdJG1223K5ubeBYcVCKWhBTgHVq3PlGdu6rBlRcCHGDdLD+YcURrLJ56Dnd2TrEpbjH9e0dvcHaflVRk1XY0+EWaRrRPjkUDrfYcgp4tbFS1HF7vzbB5XnsHSHLLdoLAVMLjSkKJOInJ6e7J/Oo7FPhfFnYA0BVdSLrzKzdgB8dBAb5qygjMjtWeKtYfNDHzsWbFYH3t7X11Jcx0VeyfiCE5ZH8LrE= smithi098.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7pnLBqevhcGmMTCFZpTrec43aVZyJ0LjwTu1bQKNkDrMzFf0e8HAujuzrU27DSEW2+v9y/g1HK58AXkKi9kPpFpFneLAXDjyT5/n9MCCBs9fxascfvxkQpYSRKeaZDJb547KlXA36JZjkCqbuglNmpefjTL5J04r9S3xEIoV0qYibBs0YABYzAJ1rVDltok/G6GvOZxNCextAfuaCBHryObkvscHGNb2bpEWN6srCsOkgMekK0A2WkkWA9D2CgEEj7bzOW7bUwhJQUe6RuHNcWrAoZMnf+XwH/G6o9JWBcCJI4P3MvvNdNQ3yOKpTQy5O9X3nEqnBY17EL3IEiERaW6iTa578pWlo9DE0B/BRq2isUAz+Mvk/+J28f41KgXldbVlY7tyLMG5jWVOG0RsztIQjS8nBeuEoUiPtXopDLg6CPsaEJoRcyb+iQRu8yHFMpuzvIy31KTMWXJ8dfgQaggho7cOOGXMe6shlJQmWtmrgXbOiceA3DnyTwyB4b0= 2021-01-06T22:23:22.574 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-06T22:23:22.637 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-01-06T22:23:22.645 INFO:teuthology.task.internal:Saving configuration 2021-01-06T22:23:22.664 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-01-06T22:23:22.674 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-01-06T22:23:22.691 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-01-06 22:18:46.626127', 'description': '/home/teuthworker/archive/pdonnell-2021-01-06_20:20:52-fs-wip-pdonnell-testing-20210105.221014-distro-basic-smithi/5760126', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNFhPDPgi/hOOov6XEpwzB9/zps65DCOucmBI6yhuH4+78AQMzrXf4mpXQmIxGW4m78ruXWmYrkyuU8gzWxzfM=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi002.front.sepia.ceph.com'} 2021-01-06T22:23:22.707 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-01-06 22:18:46.626147', 'description': '/home/teuthworker/archive/pdonnell-2021-01-06_20:20:52-fs-wip-pdonnell-testing-20210105.221014-distro-basic-smithi/5760126', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzKoO7NiiMkrIQdCrbU/82whb0Iq83/NcifnT5Cjn5JBoHIKjpd+I0iXplek/8oX2p//o9cE1+0jFRlDpBgRkQ=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi098.front.sepia.ceph.com'} 2021-01-06T22:23:22.707 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-01-06T22:23:22.715 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-01-06T22:23:22.716 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2021-01-06T22:23:22.716 INFO:teuthology.run_tasks:Running task console_log... 2021-01-06T22:23:22.800 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f035fab3400>, signals=[15]) 2021-01-06T22:23:22.800 INFO:teuthology.run_tasks:Running task internal.connect... 2021-01-06T22:23:22.810 INFO:teuthology.task.internal:Opening connections... 2021-01-06T22:23:22.811 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2021-01-06T22:23:22.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:23:22.939 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2021-01-06T22:23:22.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-06T22:23:23.039 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-01-06T22:23:23.055 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2021-01-06T22:23:23.090 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2021-01-06T22:23:23.090 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2021-01-06T22:23:23.175 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Linux" 2021-01-06T22:23:23.175 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8 (Core)" 2021-01-06T22:23:23.176 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2021-01-06T22:23:23.176 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2021-01-06T22:23:23.177 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8" 2021-01-06T22:23:23.179 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2021-01-06T22:23:23.180 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-01-06T22:23:23.181 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2021-01-06T22:23:23.183 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-01-06T22:23:23.183 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.centos.org/" 2021-01-06T22:23:23.184 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-01-06T22:23:23.184 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:23.185 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-01-06T22:23:23.185 INFO:teuthology.orchestra.run.smithi002.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-01-06T22:23:23.186 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-01-06T22:23:23.186 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-01-06T22:23:23.187 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:23.187 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2021-01-06T22:23:23.294 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2021-01-06T22:23:23.322 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2021-01-06T22:23:23.323 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2021-01-06T22:23:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Linux" 2021-01-06T22:23:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="8 (Core)" 2021-01-06T22:23:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2021-01-06T22:23:23.386 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2021-01-06T22:23:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="8" 2021-01-06T22:23:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el8" 2021-01-06T22:23:23.387 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-01-06T22:23:23.388 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2021-01-06T22:23:23.388 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-01-06T22:23:23.388 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.centos.org/" 2021-01-06T22:23:23.389 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-01-06T22:23:23.390 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:23:23.390 INFO:teuthology.orchestra.run.smithi098.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-01-06T22:23:23.391 INFO:teuthology.orchestra.run.smithi098.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-01-06T22:23:23.391 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-01-06T22:23:23.392 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-01-06T22:23:23.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:23:23.394 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2021-01-06T22:23:23.422 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-01-06T22:23:24.598 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-01-06T22:23:24.614 INFO:teuthology.task.internal:Checking for old test directory... 2021-01-06T22:23:24.615 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2021-01-06T22:23:24.625 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2021-01-06T22:23:24.644 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-01-06T22:23:29.219 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-01-06T22:23:29.219 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2021-01-06T22:23:29.223 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2021-01-06T22:23:29.252 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-01-06T22:23:29.364 INFO:teuthology.run_tasks:Running task kernel... 2021-01-06T22:23:29.410 DEBUG:teuthology.task.kernel:normalized config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}} 2021-01-06T22:23:29.412 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2021-01-06T22:23:29.412 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2021-01-06T22:23:29.469 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS 2021-01-06T22:23:29.470 DEBUG:teuthology.misc:System to be installed: CentOS 2021-01-06T22:23:29.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2021-01-06T22:23:29.693 DEBUG:teuthology.task.kernel:sha1 for client.0 is 0a8b815b92f8462139f8a404b3335d2c861824ed 2021-01-06T22:23:29.693 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "0a8b815b92f8462139f8a404b3335d2c861824ed"... 2021-01-06T22:23:29.694 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2021-01-06T22:23:29.712 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-193.28.1.el8_2.x86_64 2021-01-06T22:23:29.712 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-193.28.1.el8_2.x86_64 vs 0a8b815b92f8462139f8a404b3335d2c861824ed 2021-01-06T22:23:29.713 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2021-01-06T22:23:29.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo lsb_release -is 2021-01-06T22:23:29.770 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS 2021-01-06T22:23:29.771 DEBUG:teuthology.misc:System to be installed: CentOS 2021-01-06T22:23:29.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2021-01-06T22:23:29.981 DEBUG:teuthology.task.kernel:sha1 for client.1 is 0a8b815b92f8462139f8a404b3335d2c861824ed 2021-01-06T22:23:29.981 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "0a8b815b92f8462139f8a404b3335d2c861824ed"... 2021-01-06T22:23:29.981 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2021-01-06T22:23:29.999 INFO:teuthology.orchestra.run.smithi098.stdout:4.18.0-193.28.1.el8_2.x86_64 2021-01-06T22:23:29.999 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-193.28.1.el8_2.x86_64 vs 0a8b815b92f8462139f8a404b3335d2c861824ed 2021-01-06T22:23:30.000 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2021-01-06T22:23:30.000 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum upgrade -y linux-firmware 2021-01-06T22:23:32.821 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - AppStream 34 MB/s | 6.3 MB 00:00 2021-01-06T22:23:34.026 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - Base 24 MB/s | 2.3 MB 00:00 2021-01-06T22:23:34.526 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-8 - Extras 128 kB/s | 8.6 kB 00:00 2021-01-06T22:23:35.001 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 21 MB/s | 8.7 MB 00:00 2021-01-06T22:23:37.002 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 4.5 MB/s | 23 kB 00:00 2021-01-06T22:23:38.235 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2021-01-06T22:23:38.240 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2021-01-06T22:23:38.241 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2021-01-06T22:23:38.241 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2021-01-06T22:23:38.241 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2021-01-06T22:23:38.241 INFO:teuthology.orchestra.run.smithi002.stdout: linux-firmware noarch 20200619-99.git3890db36.el8 CentOS-Base 101 M 2021-01-06T22:23:38.242 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:38.242 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2021-01-06T22:23:38.242 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2021-01-06T22:23:38.242 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 1 Package 2021-01-06T22:23:38.243 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:38.243 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 101 M 2021-01-06T22:23:38.243 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2021-01-06T22:23:40.005 INFO:teuthology.orchestra.run.smithi002.stdout:linux-firmware-20200619-99.git3890db36.el8.noar 57 MB/s | 101 MB 00:01 2021-01-06T22:23:40.007 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2021-01-06T22:23:40.007 INFO:teuthology.orchestra.run.smithi002.stdout:Total 57 MB/s | 101 MB 00:01 2021-01-06T22:23:40.095 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2021-01-06T22:23:40.182 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2021-01-06T22:23:40.182 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2021-01-06T22:23:41.329 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2021-01-06T22:23:41.330 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2021-01-06T22:23:51.254 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2021-01-06T22:23:51.645 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : linux-firmware-20200619-99.git3890db36.el8.noarch 1/2 2021-01-06T22:23:51.864 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 2/2 2021-01-06T22:23:52.117 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 2/2 2021-01-06T22:23:52.118 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : linux-firmware-20200619-99.git3890db36.el8.noarch 1/2 2021-01-06T22:23:52.474 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 2/2 2021-01-06T22:23:52.475 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:52.475 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2021-01-06T22:23:52.476 INFO:teuthology.orchestra.run.smithi002.stdout: linux-firmware-20200619-99.git3890db36.el8.noarch 2021-01-06T22:23:52.476 INFO:teuthology.orchestra.run.smithi002.stdout: 2021-01-06T22:23:52.477 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2021-01-06T22:23:52.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum upgrade -y linux-firmware 2021-01-06T22:23:55.294 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - AppStream 35 MB/s | 6.3 MB 00:00 2021-01-06T22:23:56.379 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - Base 26 MB/s | 2.3 MB 00:00 2021-01-06T22:23:56.817 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-8 - Extras 133 kB/s | 8.6 kB 00:00 2021-01-06T22:23:57.215 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 25 MB/s | 8.7 MB 00:00 2021-01-06T22:23:59.023 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 4.7 MB/s | 23 kB 00:00 2021-01-06T22:24:00.232 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2021-01-06T22:24:00.239 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2021-01-06T22:24:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2021-01-06T22:24:00.240 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2021-01-06T22:24:00.240 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2021-01-06T22:24:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: linux-firmware noarch 20200619-99.git3890db36.el8 CentOS-Base 101 M 2021-01-06T22:24:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:24:00.241 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2021-01-06T22:24:00.241 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2021-01-06T22:24:00.241 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 1 Package 2021-01-06T22:24:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:24:00.242 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 101 M 2021-01-06T22:24:00.242 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2021-01-06T22:24:01.958 INFO:teuthology.orchestra.run.smithi098.stdout:linux-firmware-20200619-99.git3890db36.el8.noar 59 MB/s | 101 MB 00:01 2021-01-06T22:24:01.960 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2021-01-06T22:24:01.960 INFO:teuthology.orchestra.run.smithi098.stdout:Total 59 MB/s | 101 MB 00:01 2021-01-06T22:24:02.007 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2021-01-06T22:24:02.108 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2021-01-06T22:24:02.108 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2021-01-06T22:24:03.126 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2021-01-06T22:24:03.127 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2021-01-06T22:24:12.429 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2021-01-06T22:24:12.774 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : linux-firmware-20200619-99.git3890db36.el8.noarch 1/2 2021-01-06T22:24:12.984 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 2/2 2021-01-06T22:24:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 2/2 2021-01-06T22:24:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : linux-firmware-20200619-99.git3890db36.el8.noarch 1/2 2021-01-06T22:24:13.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 2/2 2021-01-06T22:24:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:24:13.627 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2021-01-06T22:24:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: linux-firmware-20200619-99.git3890db36.el8.noarch 2021-01-06T22:24:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-01-06T22:24:13.628 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2021-01-06T22:24:13.744 INFO:teuthology.task.kernel:Downloading kernel 0a8b815b92f8462139f8a404b3335d2c861824ed on client.0... 2021-01-06T22:24:13.745 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2021-01-06T22:24:13.787 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS 2021-01-06T22:24:13.788 DEBUG:teuthology.misc:System to be installed: CentOS 2021-01-06T22:24:13.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0a8b815b92f8462139f8a404b3335d2c861824ed 2021-01-06T22:24:13.970 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/0a8b815b92f8462139f8a404b3335d2c861824ed/centos/8/flavors/default/x86_64/ 2021-01-06T22:24:13.972 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.11.0_rc2_ceph_g0a8b815b92f8-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/0a8b815b92f8462139f8a404b3335d2c861824ed/centos/8/flavors/default/x86_64/ --input-file=- 2021-01-06T22:24:13.975 INFO:teuthology.task.kernel:Downloading kernel 0a8b815b92f8462139f8a404b3335d2c861824ed on client.1... 2021-01-06T22:24:13.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo lsb_release -is 2021-01-06T22:24:14.009 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS 2021-01-06T22:24:14.010 DEBUG:teuthology.misc:System to be installed: CentOS 2021-01-06T22:24:14.011 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0a8b815b92f8462139f8a404b3335d2c861824ed 2021-01-06T22:24:14.189 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/0a8b815b92f8462139f8a404b3335d2c861824ed/centos/8/flavors/default/x86_64/ 2021-01-06T22:24:14.190 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.11.0_rc2_ceph_g0a8b815b92f8-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/0a8b815b92f8462139f8a404b3335d2c861824ed/centos/8/flavors/default/x86_64/ --input-file=- 2021-01-06T22:24:14.192 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi002.front.sepia.ceph.com to complete... 2021-01-06T22:24:33.701 INFO:teuthology.orchestra.run.smithi002.stderr:2021-01-06 22:24:33 URL:https://4.chacra.ceph.com/r/kernel/testing/0a8b815b92f8462139f8a404b3335d2c861824ed/centos/8/flavors/default/x86_64/kernel-5.11.0_rc2_ceph_g0a8b815b92f8-1.x86_64.rpm [516514104/516514104] -> "/tmp/kernel.x86_64.rpm" [1] 2021-01-06T22:24:33.702 INFO:teuthology.orchestra.run.smithi002.stderr:FINISHED --2021-01-06 22:24:33-- 2021-01-06T22:24:33.702 INFO:teuthology.orchestra.run.smithi002.stderr:Total wall clock time: 20s 2021-01-06T22:24:33.702 INFO:teuthology.orchestra.run.smithi002.stderr:Downloaded: 1 files, 493M in 19s (25.3 MB/s) 2021-01-06T22:24:33.703 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi098.front.sepia.ceph.com to complete... 2021-01-06T22:24:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:2021-01-06 22:24:56 URL:https://4.chacra.ceph.com/r/kernel/testing/0a8b815b92f8462139f8a404b3335d2c861824ed/centos/8/flavors/default/x86_64/kernel-5.11.0_rc2_ceph_g0a8b815b92f8-1.x86_64.rpm [516514104/516514104] -> "/tmp/kernel.x86_64.rpm" [1] 2021-01-06T22:24:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:FINISHED --2021-01-06 22:24:56-- 2021-01-06T22:24:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:Total wall clock time: 42s 2021-01-06T22:24:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:Downloaded: 1 files, 493M in 42s (11.7 MB/s) 2021-01-06T22:24:56.545 INFO:teuthology.task.kernel:Installing kernel 0a8b815b92f8462139f8a404b3335d2c861824ed on client.0... 2021-01-06T22:24:56.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2021-01-06T22:24:58.638 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying... ######################################## 2021-01-06T22:24:58.877 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing... ######################################## 2021-01-06T22:24:58.920 INFO:teuthology.orchestra.run.smithi002.stdout:Updating / installing... 2021-01-06T22:25:06.837 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2021-01-06T22:25:06.838 DEBUG:teuthology.task.console_log:Killing console logger for smithi002 2021-01-06T22:25:06.838 DEBUG:teuthology.task.console_log:Killing console logger for smithi098 2021-01-06T22:25:06.839 DEBUG:teuthology.task.console_log:Killing console logger for smithi002 2021-01-06T22:25:06.839 DEBUG:teuthology.task.console_log:Killing console logger for smithi098 2021-01-06T22:25:06.839 DEBUG:teuthology.exit:Finished running handlers