2020-11-20T13:50:45.864 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-20T13:50:45.865 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_ffsb} --verbose --archive /home/teuthworker/archive/teuthology-2020-11-16_04:10:02-fs-octopus-distro-basic-smithi/5628549 --block --name teuthology-2020-11-16_04:10:02-fs-octopus-distro-basic-smithi --owner scheduled_teuthology@teuthology --lock -- /tmp/teuthology-worker.xwo0okzw.tmp 2020-11-20T13:50:45.904 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T13:50:45.968 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2020-11-16_04:10:02-fs-octopus-distro-basic-smithi/5628549 branch: octopus description: fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_ffsb} email: ceph-qa@ceph.io first_in_suite: false job_id: '5628549' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2020-11-16_04:10:02-fs-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: centos os_version: '8.1' overrides: admin_socket: branch: octopus ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd_max_omap_entries_per_request: 10000 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - SLOW_OPS - slow request sha1: a7b62340ad27b26642bb54ec916a97c7e1f07a34 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: a7b62340ad27b26642bb54ec916a97c7e1f07a34 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: octopus sha1: a7b62340ad27b26642bb54ec916a97c7e1f07a34 owner: scheduled_teuthology@teuthology priority: 1000 repo: git://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: a7b62340ad27b26642bb54ec916a97c7e1f07a34 sleep_before_teardown: 0 suite: fs suite_branch: octopus suite_path: /home/teuthworker/src/git.ceph.com_ceph_octopus/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: a7b62340ad27b26642bb54ec916a97c7e1f07a34 tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - exec: client.0: - sudo ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - ceph-fuse: null - check-counter: counters: mds: - mds.dir_split - workunit: clients: all: - suites/ffsb.sh teuthology_branch: master timestamp: 2020-11-16_04:10:02 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.6032 2020-11-20T13:50:45.969 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_octopus/qa; will attempt to use it 2020-11-20T13:50:45.969 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks 2020-11-20T13:50:45.969 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-20T13:50:45.971 INFO:teuthology.task.internal:Checking packages... 2020-11-20T13:50:45.990 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'a7b62340ad27b26642bb54ec916a97c7e1f07a34' 2020-11-20T13:50:45.991 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-20T13:50:45.991 INFO:teuthology.packaging:ref: None 2020-11-20T13:50:45.992 INFO:teuthology.packaging:tag: None 2020-11-20T13:50:45.992 INFO:teuthology.packaging:branch: octopus 2020-11-20T13:50:45.992 INFO:teuthology.packaging:sha1: a7b62340ad27b26642bb54ec916a97c7e1f07a34 2020-11-20T13:50:45.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2020-11-20T13:50:46.286 INFO:teuthology.task.internal:Found packages for ceph version 15.2.6-589.g08106742 2020-11-20T13:50:46.287 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-20T13:50:46.303 INFO:teuthology.task.internal:no buildpackages task found 2020-11-20T13:50:46.303 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-20T13:50:46.323 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-20T13:50:46.323 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T13:50:46.391 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T13:50:56.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T13:51:06.422 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2020-11-16_04:10:02-fs-octopus-distro-basic-smithi/5628549'} 2020-11-20T13:51:06.442 DEBUG:teuthology.lock.ops:locked ubuntu@smithi148.front.sepia.ceph.com, ubuntu@smithi162.front.sepia.ceph.com 2020-11-20T13:51:06.493 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f292533e7b8>, signals=[15]) 2020-11-20T13:51:07.063 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of centos 8.1 2020-11-20T13:51:07.152 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of centos 8.1 2020-11-20T13:51:09.546 INFO:teuthology.orchestra.console:Power off smithi148 2020-11-20T13:51:09.547 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T13:51:09.565 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:09.656 INFO:teuthology.orchestra.console:Power off smithi162 2020-11-20T13:51:09.657 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T13:51:09.675 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:09.685 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2020-11-20T13:51:09.786 INFO:teuthology.orchestra.console:Power on smithi148 2020-11-20T13:51:09.786 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T13:51:09.795 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2020-11-20T13:51:09.804 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:09.896 INFO:teuthology.orchestra.console:Power on smithi162 2020-11-20T13:51:09.897 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T13:51:09.915 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:13.825 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:13.938 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:17.935 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:18.047 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T13:51:18.144 INFO:teuthology.orchestra.console:Power on for smithi148 completed 2020-11-20T13:51:18.246 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2020-11-20T13:51:18.256 INFO:teuthology.orchestra.console:Power on for smithi162 completed 2020-11-20T13:51:18.356 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish 2020-11-20T13:54:21.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:54:37.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:54:46.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:54:55.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:55:04.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:55:13.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:55:22.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:55:27.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:55:31.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:55:40.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:56:08.314 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-20T13:56:33.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:56:34.221 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T13:56:34.222 INFO:teuthology.orchestra.run.smithi148:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T13:56:34.499 INFO:teuthology.provision.fog.smithi148:Node is ready 2020-11-20T13:56:34.500 INFO:teuthology.orchestra.run.smithi148:> hostname 2020-11-20T13:56:34.514 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com 2020-11-20T13:56:34.514 INFO:teuthology.orchestra.run.smithi148:> grep smithi148.front.sepia.ceph.com /etc/hosts 2020-11-20T13:56:34.569 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148 2020-11-20T13:56:34.570 INFO:teuthology.orchestra.run.smithi148:> sudo hostname smithi148 2020-11-20T13:56:34.678 INFO:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/smithi148.front.sepia.ceph.com/smithi148/g' /etc/hosts 2020-11-20T13:56:34.708 INFO:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/172.21.15.148/172.21.15.148/g' /etc/hosts 2020-11-20T13:56:34.775 INFO:teuthology.orchestra.run.smithi148:> sudo sed -i -e 's/smithi148.front.sepia.ceph.com/smithi148/g' /etc/hostname 2020-11-20T13:56:34.842 INFO:teuthology.orchestra.run.smithi148:> sudo hostname smithi148 2020-11-20T13:56:34.908 INFO:teuthology.provision.fog.smithi148:Deploy complete! 2020-11-20T13:56:34.909 DEBUG:teuthology.parallel:result is None 2020-11-20T13:56:46.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:56:46.821 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T13:56:46.822 INFO:teuthology.orchestra.run.smithi162:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T13:56:47.120 INFO:teuthology.provision.fog.smithi162:Node is ready 2020-11-20T13:56:47.120 INFO:teuthology.orchestra.run.smithi162:> hostname 2020-11-20T13:56:47.134 INFO:teuthology.orchestra.run.smithi162.stdout:smithi162.front.sepia.ceph.com 2020-11-20T13:56:47.135 INFO:teuthology.orchestra.run.smithi162:> grep smithi162.front.sepia.ceph.com /etc/hosts 2020-11-20T13:56:47.191 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.15.162 smithi162.front.sepia.ceph.com smithi162 2020-11-20T13:56:47.191 INFO:teuthology.orchestra.run.smithi162:> sudo hostname smithi162 2020-11-20T13:56:47.308 INFO:teuthology.orchestra.run.smithi162:> sudo sed -i -e 's/smithi162.front.sepia.ceph.com/smithi162/g' /etc/hosts 2020-11-20T13:56:47.337 INFO:teuthology.orchestra.run.smithi162:> sudo sed -i -e 's/172.21.15.162/172.21.15.162/g' /etc/hosts 2020-11-20T13:56:47.404 INFO:teuthology.orchestra.run.smithi162:> sudo sed -i -e 's/smithi162.front.sepia.ceph.com/smithi162/g' /etc/hostname 2020-11-20T13:56:47.472 INFO:teuthology.orchestra.run.smithi162:> sudo hostname smithi162 2020-11-20T13:56:47.538 INFO:teuthology.provision.fog.smithi162:Deploy complete! 2020-11-20T13:56:47.539 DEBUG:teuthology.parallel:result is None 2020-11-20T13:56:47.652 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com 2020-11-20T13:56:47.653 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T13:56:47.669 INFO:teuthology.lock.ops:Checking smithi162.front.sepia.ceph.com 2020-11-20T13:56:47.670 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T13:56:47.683 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server 2020-11-20T13:56:47.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T13:56:47.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:56:47.750 INFO:teuthology.orchestra.run.smithi148:> true 2020-11-20T13:56:47.915 INFO:teuthology.orchestra.run.smithi148:> uname -m 2020-11-20T13:56:47.969 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2020-11-20T13:56:47.970 INFO:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2020-11-20T13:56:48.023 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Linux" 2020-11-20T13:56:48.024 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8 (Core)" 2020-11-20T13:56:48.024 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2020-11-20T13:56:48.024 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2020-11-20T13:56:48.024 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2020-11-20T13:56:48.025 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2020-11-20T13:56:48.025 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T13:56:48.026 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2020-11-20T13:56:48.026 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T13:56:48.026 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T13:56:48.026 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T13:56:48.027 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-11-20T13:56:48.027 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T13:56:48.027 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T13:56:48.027 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T13:56:48.027 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T13:56:48.028 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-11-20T13:56:48.029 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2020-11-20T13:56:48.045 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: set os type and version on server 2020-11-20T13:56:48.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T13:56:48.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:56:48.114 INFO:teuthology.orchestra.run.smithi162:> true 2020-11-20T13:56:48.275 INFO:teuthology.orchestra.run.smithi162:> uname -m 2020-11-20T13:56:48.328 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2020-11-20T13:56:48.329 INFO:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2020-11-20T13:56:48.382 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Linux" 2020-11-20T13:56:48.382 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8 (Core)" 2020-11-20T13:56:48.383 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2020-11-20T13:56:48.383 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2020-11-20T13:56:48.383 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8" 2020-11-20T13:56:48.383 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2020-11-20T13:56:48.384 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T13:56:48.384 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2020-11-20T13:56:48.384 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T13:56:48.384 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T13:56:48.384 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T13:56:48.385 INFO:teuthology.orchestra.run.smithi162.stdout: 2020-11-20T13:56:48.385 INFO:teuthology.orchestra.run.smithi162.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T13:56:48.385 INFO:teuthology.orchestra.run.smithi162.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T13:56:48.385 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T13:56:48.386 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T13:56:48.386 INFO:teuthology.orchestra.run.smithi162.stdout: 2020-11-20T13:56:48.386 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2020-11-20T13:56:48.404 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-20T13:56:48.439 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGOjkRo5j3rjlxCYO81Y6+FwGUrFhxYLYUDAAZzCB7CfMOWxFapZqlXfVSXmaCOZzgNoWqAtirShq6ZyeR9f/h7ckorhzlLSf6CnYkE+QBOdSIbA1dKC4W0r4/RgUXqtZOMfhAs1uUMBked2mAC4wU1FnAbHqfyQKOEFsFu11FrhLMjC3KB0k0QZrLdReHcTf2OoDgK9po023LsVHN4rEGS2yfgb2yXmyvtGXg2dGAwlPfwGj9d+SuLF4HMUOqS/Ei5Iil2C7nIwNo3X3TLU1AemghPxFoEX+YzztnHiYoC4UBHkkqGOpP4gPDI8Ent0W2wOsDUYcgeU4Fawwmbp7pAzL8cDxrCieDMfhIAuhYWF5ObKZhSQjggJo78/vDs1Lxc3rRKiqHM/8PKcJAgWbHi/nfCZUQKFw5bWyfD2FepJbeYqSYxYPDQeoUADR3YjkpNqER70+5Drxb7A0cZLk2W3cQihYmj9+hQ6Mmt1+gDTQ08fb4+mJjwmTmFkY6VS0= smithi162.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaqYD5nS+P2RtE1zeGU879P6MY2YpBVKTMd1/DIwx/kCunRIKhIKXHpgoUM1lxlcf2CoRk+ZhVQUp4X2JtRa2BEX1DNnKtuaFQ2fQH7dzg+fZZy4ArgsBC9iAW2W48s4zTxlG4pMu99sSgR/E9MgAA4VFIaXPIwwCJPjf9xcJqWCxWNm9Id03avOEFDi7S02znIzRMokhgbpEUsvclNMgcBy5aXpCPcRqKuyRP9HfnovLteNTJl6hFxpGWU27TVBqvQ06YBQsv1Ivto1J1xh2ZFDrDOhlaemLfvMHD/t7DK5HuTAUiUysM3EICgxI79I338M0qlC+c93MoywCMKYAXp3qYyc4mp9r1rwoAAsOdRA0M1UifZ7HdoyuMTavr57I0LG9sQ61m2KdQKuo+VokJjckaN9yhkP5+PZEA76wygghL4vh/tSR+pfmtATcFbx0MGqOBAspTAMG0zBY0g3fDzPAouSNb/nJsmZFvpNkKR9JOLVGNeyWpxcwZpTfatx0= 2020-11-20T13:56:48.439 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T13:56:48.491 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-20T13:56:48.504 INFO:teuthology.task.internal:Saving configuration 2020-11-20T13:56:48.523 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-20T13:56:48.542 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-20T13:56:48.560 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 13:51:06.428809', 'description': '/home/teuthworker/archive/teuthology-2020-11-16_04:10:02-fs-octopus-distro-basic-smithi/5628549', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwcJ+jh28r71rw1fu+diOO4cZAmOS/0yXs2UCoaj/FnVcvM/EhYxWriakl22RiFZJveH9N4ZsFvKsvPZpjCXdo=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi148.front.sepia.ceph.com'} 2020-11-20T13:56:48.577 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 13:51:06.428842', 'description': '/home/teuthworker/archive/teuthology-2020-11-16_04:10:02-fs-octopus-distro-basic-smithi/5628549', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPeomEnSxRLPH3JXxKQ3H0d4q8KYkx7GYgFzdvcqcQuGR4SmeQB7wxMyfvvQI9mFYCR7/B7owtlXGc5i2iMOIew=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi162.front.sepia.ceph.com'} 2020-11-20T13:56:48.578 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-20T13:56:48.588 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-11-20T13:56:48.588 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2020-11-20T13:56:48.588 INFO:teuthology.run_tasks:Running task console_log... 2020-11-20T13:56:48.649 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2925324bf8>, signals=[15]) 2020-11-20T13:56:48.649 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-20T13:56:48.660 INFO:teuthology.task.internal:Opening connections... 2020-11-20T13:56:48.661 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2020-11-20T13:56:48.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:56:48.735 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2020-11-20T13:56:48.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T13:56:48.810 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-20T13:56:48.819 INFO:teuthology.orchestra.run.smithi148:> uname -m 2020-11-20T13:56:48.857 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2020-11-20T13:56:48.858 INFO:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2020-11-20T13:56:48.910 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Linux" 2020-11-20T13:56:48.911 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8 (Core)" 2020-11-20T13:56:48.911 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2020-11-20T13:56:48.911 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2020-11-20T13:56:48.911 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2020-11-20T13:56:48.912 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2020-11-20T13:56:48.912 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T13:56:48.912 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2020-11-20T13:56:48.912 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T13:56:48.912 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T13:56:48.913 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T13:56:48.913 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-11-20T13:56:48.913 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T13:56:48.913 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T13:56:48.913 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T13:56:48.914 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T13:56:48.914 INFO:teuthology.orchestra.run.smithi148.stdout: 2020-11-20T13:56:48.915 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2020-11-20T13:56:48.932 INFO:teuthology.orchestra.run.smithi162:> uname -m 2020-11-20T13:56:48.947 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2020-11-20T13:56:48.947 INFO:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2020-11-20T13:56:49.000 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Linux" 2020-11-20T13:56:49.000 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8 (Core)" 2020-11-20T13:56:49.000 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2020-11-20T13:56:49.001 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2020-11-20T13:56:49.001 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8" 2020-11-20T13:56:49.001 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2020-11-20T13:56:49.001 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-20T13:56:49.002 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2020-11-20T13:56:49.002 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-20T13:56:49.002 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.centos.org/" 2020-11-20T13:56:49.002 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-20T13:56:49.002 INFO:teuthology.orchestra.run.smithi162.stdout: 2020-11-20T13:56:49.003 INFO:teuthology.orchestra.run.smithi162.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-20T13:56:49.003 INFO:teuthology.orchestra.run.smithi162.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-20T13:56:49.003 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-20T13:56:49.003 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-20T13:56:49.003 INFO:teuthology.orchestra.run.smithi162.stdout: 2020-11-20T13:56:49.006 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2020-11-20T13:56:49.022 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-20T13:56:49.046 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-20T13:56:49.059 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-20T13:56:49.061 INFO:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2020-11-20T13:56:49.066 INFO:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2020-11-20T13:56:49.079 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-20T13:56:49.099 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-20T13:56:49.100 INFO:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2020-11-20T13:56:49.119 INFO:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2020-11-20T13:56:49.142 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-20T13:56:49.189 INFO:teuthology.run_tasks:Running task kernel... 2020-11-20T13:56:49.207 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}} 2020-11-20T13:56:49.208 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-20T13:56:49.208 INFO:teuthology.orchestra.run.smithi148:> uname -r 2020-11-20T13:56:49.222 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-193.28.1.el8_2.x86_64 2020-11-20T13:56:49.223 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-193.28.1.el8_2.x86_64 2020-11-20T13:56:49.223 INFO:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2020-11-20T13:56:51.731 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - AppStream 39 MB/s | 7.0 MB 00:00 2020-11-20T13:56:52.955 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - Base 25 MB/s | 2.2 MB 00:00 2020-11-20T13:56:53.380 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS-8 - Extras 87 kB/s | 5.5 kB 00:00 2020-11-20T13:56:53.766 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 25 MB/s | 8.3 MB 00:00 2020-11-20T13:56:55.523 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 4.1 MB/s | 23 kB 00:00 2020-11-20T13:56:56.476 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-11-20T13:56:56.476 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2020-11-20T13:56:56.632 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2020-11-20T13:56:56.633 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2020-11-20T13:56:56.634 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2020-11-20T13:56:56.753 INFO:teuthology.orchestra.run.smithi148:> echo no | sudo yum reinstall kernel || true 2020-11-20T13:56:57.199 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Nov 2020 01:56:55 PM UTC. 2020-11-20T13:56:57.442 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-20T13:56:57.443 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-20T13:56:57.495 INFO:teuthology.orchestra.run.smithi148.stderr:Error: No packages marked for reinstall. 2020-11-20T13:56:57.542 INFO:teuthology.orchestra.run.smithi148:> sudo yum reinstall -y kernel || true 2020-11-20T13:56:57.946 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Nov 2020 01:56:55 PM UTC. 2020-11-20T13:56:58.217 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-20T13:56:58.217 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-20T13:56:58.250 INFO:teuthology.orchestra.run.smithi148.stderr:Error: No packages marked for reinstall. 2020-11-20T13:56:58.291 INFO:teuthology.orchestra.run.smithi148:> rpm -q kernel --last | head -n 1 2020-11-20T13:56:58.357 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-193.28.1.el8_2.x86_64 Sun 15 Nov 2020 12:12:26 PM UTC 2020-11-20T13:56:58.358 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-193.28.1.el8_2.x86_64 2020-11-20T13:56:58.358 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-20T13:56:58.358 INFO:teuthology.orchestra.run.smithi162:> uname -r 2020-11-20T13:56:58.375 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-193.28.1.el8_2.x86_64 2020-11-20T13:56:58.375 INFO:teuthology.task.kernel:Running kernel on smithi162: 4.18.0-193.28.1.el8_2.x86_64 2020-11-20T13:56:58.376 INFO:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2020-11-20T13:57:00.889 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-8 - AppStream 32 MB/s | 7.0 MB 00:00 2020-11-20T13:57:02.142 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-8 - Base 24 MB/s | 2.2 MB 00:00 2020-11-20T13:57:02.573 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-8 - Extras 86 kB/s | 5.5 kB 00:00 2020-11-20T13:57:03.199 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 14 MB/s | 8.3 MB 00:00 2020-11-20T13:57:05.000 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 4.7 MB/s | 23 kB 00:00 2020-11-20T13:57:05.893 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-11-20T13:57:05.894 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2020-11-20T13:57:06.064 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2020-11-20T13:57:06.065 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2020-11-20T13:57:06.065 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2020-11-20T13:57:06.197 INFO:teuthology.orchestra.run.smithi162:> echo no | sudo yum reinstall kernel || true 2020-11-20T13:57:06.603 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Nov 2020 01:57:04 PM UTC.