2021-04-11T10:03:08.621 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-11T10:03:08.626 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-04-11_09:03:26-rados-wip-kefu-octopus-testing-2021-04-11-1228-distro-basic-smithi/6035674 --name kchai-2021-04-11_09:03:26-rados-wip-kefu-octopus-testing-2021-04-11-1228-distro-basic-smithi --description rados/singleton-nomsgr/{all/full-tiering rados supported-random-distro$/{centos_latest}} -- /home/teuthworker/archive/kchai-2021-04-11_09:03:26-rados-wip-kefu-octopus-testing-2021-04-11-1228-distro-basic-smithi/6035674/orig.config.yaml 2021-04-11T10:03:08.664 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-04-11T10:03:08.719 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-04-11_09:03:26-rados-wip-kefu-octopus-testing-2021-04-11-1228-distro-basic-smithi/6035674 branch: wip-kefu-octopus-testing-2021-04-11-1228 description: rados/singleton-nomsgr/{all/full-tiering rados supported-random-distro$/{centos_latest}} email: tchaikov@gmail.com first_in_suite: false job_id: '6035674' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-04-11_09:03:26-rados-wip-kefu-octopus-testing-2021-04-11-1228-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-kefu-octopus-testing-2021-04-11-1228 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd op queue: debug_random osd op queue cut off: debug_random log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - is full - overall HEALTH_ - \(POOL_FULL\) - \(POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5ed7075aceeb911c5eb13628a53ecca211402124 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 5ed7075aceeb911c5eb13628a53ecca211402124 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-octopus-testing-2021-04-11-1228 sha1: 5ed7075aceeb911c5eb13628a53ecca211402124 owner: scheduled_kchai@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 5ed7075aceeb911c5eb13628a53ecca211402124 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-octopus-testing-2021-04-11-1228 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5ed7075aceeb911c5eb13628a53ecca211402124/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 5ed7075aceeb911c5eb13628a53ecca211402124 targets: smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDk4OwoonKZXrW4lhOb7oGvBPfGnPL/A6T6qvgU2GpwPUMOoMkrSW34rWV/9ruIDpqxiEVEhkKD5FAudbf/x2eDCfVNdmUTzVGSumfypvYs+DAOCV4Wjf67y22RQBUaTcNILM00f5j2jN3aFkydbQLZQ1Wawr10bI3NnT3WkI68mi3lERcZBvdfCn0LN7YIHrg521RSmQJis+eq0c10R6keckP3VVEqR2hkmyBuGl9WXELwkYaLGPfh8T/6C0qMr4sLmxMKVLRsALgo9JWqdsv+M1k+3GV+kgn2AjPPUHLrzIHEwSGDOB/JbTPoLHE/xg1kgZ4QFNsdJ/yAYwdBOEUrcW+PhTE6XqTJ4R9/jP6IWRGiN2DzrRAG2H0+v3EKHtNoo/BgcmmDcutRZBjmKe+PMDjmTg1FJ4BVfXsEqMQgjLtPCKJ6fCp8VDMWnO5jK+pdyByixG+SrFUZbaDeZaokE0LiB2qax9up8b9JjR/Pe6OxpgtPGNT3lhItCuJT5f8= tasks: - install: null - ceph: conf: global: osd max object name len: 460 osd max object namespace len: 64 pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - exec: client.0: - ceph osd pool create ec-ca 1 1 - ceph osd pool create ec 1 1 erasure default - ceph osd pool application enable ec rados - ceph osd tier add ec ec-ca - ceph osd tier cache-mode ec-ca readproxy - ceph osd tier set-overlay ec ec-ca - ceph osd pool set ec-ca hit_set_type bloom - ceph osd pool set-quota ec-ca max_bytes 20480000 - ceph osd pool set-quota ec max_bytes 20480000 - ceph osd pool set ec-ca target_max_bytes 20480000 - timeout 30 rados -p ec-ca bench 30 write || true - ceph osd pool set-quota ec-ca max_bytes 0 - ceph osd pool set-quota ec max_bytes 0 teuthology_branch: master teuthology_sha1: 85d61eae4759f46ce21e9a37cd816a7a1a66c9d5 timestamp: 2021-04-11_09:03:26 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23935 2021-04-11T10:03:08.719 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5ed7075aceeb911c5eb13628a53ecca211402124/qa; will attempt to use it 2021-04-11T10:03:08.720 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5ed7075aceeb911c5eb13628a53ecca211402124/qa/tasks 2021-04-11T10:03:08.721 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-04-11T10:03:08.722 INFO:teuthology.task.internal:Checking packages... 2021-04-11T10:03:08.753 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '5ed7075aceeb911c5eb13628a53ecca211402124' 2021-04-11T10:03:08.754 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-04-11T10:03:08.755 INFO:teuthology.packaging:ref: None 2021-04-11T10:03:08.756 INFO:teuthology.packaging:tag: None 2021-04-11T10:03:08.756 INFO:teuthology.packaging:branch: wip-kefu-octopus-testing-2021-04-11-1228 2021-04-11T10:03:08.757 INFO:teuthology.packaging:sha1: 5ed7075aceeb911c5eb13628a53ecca211402124 2021-04-11T10:03:08.758 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-octopus-testing-2021-04-11-1228 2021-04-11T10:03:08.917 INFO:teuthology.task.internal:Found packages for ceph version 15.2.10-266.g5ed7075a 2021-04-11T10:03:08.918 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-04-11T10:03:08.930 INFO:teuthology.task.internal:no buildpackages task found 2021-04-11T10:03:08.930 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-04-11T10:03:09.047 INFO:teuthology.task.internal:Saving configuration 2021-04-11T10:03:09.060 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-04-11T10:03:09.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T10:03:09.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-11 09:58:00.527470', 'description': '/home/teuthworker/archive/kchai-2021-04-11_09:03:26-rados-wip-kefu-octopus-testing-2021-04-11-1228-distro-basic-smithi/6035674', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAwcfv5/o7ycobLV6UYyi2NDYnGoMv4FHTerBzLTiQift80oesZCOIuwPHiERVW9FkpexFmf6gqajAFY60gjqYY=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi164.front.sepia.ceph.com'} 2021-04-11T10:03:09.173 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-04-11T10:03:09.185 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-04-11T10:03:09.185 INFO:teuthology.run_tasks:Running task console_log... 2021-04-11T10:03:09.274 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe7bd37e9d8>, signals=[15]) 2021-04-11T10:03:09.276 INFO:teuthology.run_tasks:Running task internal.connect... 2021-04-11T10:03:09.301 INFO:teuthology.task.internal:Opening connections... 2021-04-11T10:03:09.301 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2021-04-11T10:03:09.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-11T10:03:09.380 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-04-11T10:03:09.392 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2021-04-11T10:03:09.559 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2021-04-11T10:03:09.560 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2021-04-11T10:03:09.639 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Linux" 2021-04-11T10:03:09.639 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8 (Core)" 2021-04-11T10:03:09.640 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2021-04-11T10:03:09.640 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2021-04-11T10:03:09.641 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8" 2021-04-11T10:03:09.641 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8" 2021-04-11T10:03:09.641 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-04-11T10:03:09.642 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2021-04-11T10:03:09.642 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-11T10:03:09.643 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.centos.org/" 2021-04-11T10:03:09.643 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-11T10:03:09.644 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-04-11T10:03:09.644 INFO:teuthology.orchestra.run.smithi164.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-11T10:03:09.644 INFO:teuthology.orchestra.run.smithi164.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-11T10:03:09.645 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-04-11T10:03:09.645 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-04-11T10:03:09.646 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-04-11T10:03:09.647 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2021-04-11T10:03:09.705 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-04-11T10:03:09.748 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-04-11T10:03:09.807 INFO:teuthology.task.internal:Checking for old test directory... 2021-04-11T10:03:09.810 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2021-04-11T10:03:09.835 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-04-11T10:03:09.974 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-04-11T10:03:09.974 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2021-04-11T10:03:10.003 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-04-11T10:03:10.162 INFO:teuthology.run_tasks:Running task kernel... 2021-04-11T10:03:10.225 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-04-11T10:03:10.226 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-04-11T10:03:10.227 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2021-04-11T10:03:10.245 INFO:teuthology.orchestra.run.smithi164.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-04-11T10:03:10.245 INFO:teuthology.task.kernel:Running kernel on smithi164: 4.18.0-240.10.1.el8_3.x86_64 2021-04-11T10:03:10.246 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2021-04-11T10:03:13.812 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-8 - AppStream 29 MB/s | 6.3 MB 00:00 2021-04-11T10:03:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-8 - Base 23 MB/s | 2.3 MB 00:00 2021-04-11T10:03:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-8 - Extras 140 kB/s | 9.6 kB 00:00 2021-04-11T10:03:16.404 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 9.2 MB 00:00 2021-04-11T10:03:18.326 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 1.1 MB/s | 24 kB 00:00 2021-04-11T10:03:19.346 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2021-04-11T10:03:19.347 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-04-11T10:03:19.515 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2021-04-11T10:03:19.538 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-04-11T10:03:19.538 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2021-04-11T10:03:19.539 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-04-11T10:03:19.539 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2021-04-11T10:03:19.540 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 4.4 M 2021-04-11T10:03:19.540 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2021-04-11T10:03:19.541 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 30 M 2021-04-11T10:03:19.541 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 4.18.0-240.22.1.el8_3 CentOS-Base 26 M 2021-04-11T10:03:19.541 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-04-11T10:03:19.542 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2021-04-11T10:03:19.542 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2021-04-11T10:03:19.543 INFO:teuthology.orchestra.run.smithi164.stdout:Install 3 Packages 2021-04-11T10:03:19.543 INFO:teuthology.orchestra.run.smithi164.stdout: 2021-04-11T10:03:19.544 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 60 M 2021-04-11T10:03:19.544 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 83 M 2021-04-11T10:03:19.545 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2021-04-11T10:03:19.843 INFO:teuthology.orchestra.run.smithi164.stdout:(1/3): kernel-4.18.0-240.22.1.el8_3.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2021-04-11T10:03:20.293 INFO:teuthology.orchestra.run.smithi164.stdout:(2/3): kernel-modules-4.18.0-240.22.1.el8_3.x86 34 MB/s | 26 MB 00:00 2021-04-11T10:03:20.612 INFO:teuthology.orchestra.run.smithi164.stdout:(3/3): kernel-core-4.18.0-240.22.1.el8_3.x86_64 28 MB/s | 30 MB 00:01 2021-04-11T10:03:20.615 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2021-04-11T10:03:20.615 INFO:teuthology.orchestra.run.smithi164.stdout:Total 56 MB/s | 60 MB 00:01 2021-04-11T10:03:20.701 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2021-04-11T10:03:21.013 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2021-04-11T10:03:21.014 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2021-04-11T10:03:22.080 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2021-04-11T10:03:22.084 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2021-04-11T10:03:24.336 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2021-04-11T10:03:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-04-11T10:03:27.699 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-4.18.0-240.22.1.el8_3.x86_64 1/3 2021-04-11T10:03:28.042 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-4.18.0-240.22.1.el8_3.x86_64 2/3 2021-04-11T10:03:29.933 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-04-11T10:03:29.934 DEBUG:teuthology.task.console_log:Killing console logger for smithi164 2021-04-11T10:03:29.934 DEBUG:teuthology.exit:Finished running handlers