2021-10-13T04:11:02.798 INFO:root:teuthology version: 1.1.0-6f40c4ca 2021-10-13T04:11:02.800 DEBUG:teuthology.run:Teuthology command: teuthology --lock --archive /home/teuthworker/archive/yuriw-2021-10-12_12:50:50-rados-wip-yuri7-testing-2021-10-08-0924-distro-basic-smithi/6433627 --description rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream}} --name yuriw-2021-10-12_12:50:50-rados-wip-yuri7-testing-2021-10-08-0924-distro-basic-smithi --block --owner scheduled_yuriw@teuthology --verbose -- /tmp/teuthology-worker.0583d2cg.tmp 2021-10-13T04:11:02.828 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-10-13T04:11:02.890 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-10-12_12:50:50-rados-wip-yuri7-testing-2021-10-08-0924-distro-basic-smithi/6433627 branch: wip-yuri7-testing-2021-10-08-0924 description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6433627' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-10-12_12:50:50-rados-wip-yuri7-testing-2021-10-08-0924-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri7-testing-2021-10-08-0924 ceph: conf: global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0fd834746e193d64ab7658be587f243d8eb574bb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 0fd834746e193d64ab7658be587f243d8eb574bb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri7-testing-2021-10-08-0924 sha1: 0fd834746e193d64ab7658be587f243d8eb574bb owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 sha1: 0fd834746e193d64ab7658be587f243d8eb574bb sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri7-testing-2021-10-08-0924 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0fd834746e193d64ab7658be587f243d8eb574bb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0fd834746e193d64ab7658be587f243d8eb574bb tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology_branch: master teuthology_sha1: 6f40c4ca66d0ac515a149c2e25df0d91b2dd1a9b timestamp: 2021-10-12_12:50:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.12641 2021-10-13T04:11:02.891 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0fd834746e193d64ab7658be587f243d8eb574bb/qa; will attempt to use it 2021-10-13T04:11:02.891 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0fd834746e193d64ab7658be587f243d8eb574bb/qa/tasks 2021-10-13T04:11:02.892 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-10-13T04:11:02.894 INFO:teuthology.task.internal:Checking packages... 2021-10-13T04:11:02.913 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '0fd834746e193d64ab7658be587f243d8eb574bb' 2021-10-13T04:11:02.913 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-10-13T04:11:02.914 INFO:teuthology.packaging:ref: None 2021-10-13T04:11:02.914 INFO:teuthology.packaging:tag: None 2021-10-13T04:11:02.915 INFO:teuthology.packaging:branch: wip-yuri7-testing-2021-10-08-0924 2021-10-13T04:11:02.915 INFO:teuthology.packaging:sha1: 0fd834746e193d64ab7658be587f243d8eb574bb 2021-10-13T04:11:02.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2021-10-08-0924 2021-10-13T04:11:03.077 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-8116.g0fd83474 2021-10-13T04:11:03.077 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-10-13T04:11:03.089 INFO:teuthology.task.internal:no buildpackages task found 2021-10-13T04:11:03.089 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2021-10-13T04:11:03.127 INFO:teuthology.lock.ops:Locking machines... 2021-10-13T04:11:03.128 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-10-13T04:11:03.185 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuriw@teuthology', 'count': 1, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuriw-2021-10-12_12:50:50-rados-wip-yuri7-testing-2021-10-08-0924-distro-basic-smithi/6433627'} 2021-10-13T04:11:03.203 DEBUG:teuthology.lock.ops:locked ubuntu@smithi018.front.sepia.ceph.com 2021-10-13T04:11:03.229 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbb8c2e6e18>, signals=[15]) 2021-10-13T04:11:03.231 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi018.front.sepia.ceph.com' reimaging 2021-10-13T04:11:03.231 INFO:teuthology.lock.ops:Updating [ubuntu@smithi018.front.sepia.ceph.com]: reset os type and version on server 2021-10-13T04:11:03.232 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2021-10-13T04:11:03.255 INFO:teuthology.lock.ops:Node 'ubuntu@smithi018.front.sepia.ceph.com' reimaging is complete 2021-10-13T04:11:03.458 INFO:teuthology.provision.fog.smithi018:Scheduling deploy of centos 8.stream 2021-10-13T04:11:03.786 INFO:teuthology.orchestra.console:Power off smithi018 2021-10-13T04:11:03.787 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-10-13T04:11:03.808 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-10-13T04:11:07.838 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-10-13T04:11:11.950 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-10-13T04:11:12.160 INFO:teuthology.orchestra.console:Power off for smithi018 completed 2021-10-13T04:11:12.261 INFO:teuthology.orchestra.console:Power on smithi018 2021-10-13T04:11:12.262 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-10-13T04:11:12.283 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-10-13T04:11:16.306 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-10-13T04:11:20.418 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-10-13T04:11:20.630 INFO:teuthology.orchestra.console:Power on for smithi018 completed 2021-10-13T04:11:20.731 INFO:teuthology.provision.fog.smithi018:Waiting for deploy to finish 2021-10-13T04:13:51.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-13T04:14:16.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-13T04:14:25.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-13T04:14:34.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-13T04:15:40.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-13T04:15:40.681 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-10-13T04:15:40.683 DEBUG:teuthology.orchestra.run.smithi018:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2021-10-13T04:15:40.952 INFO:teuthology.provision.fog.smithi018:Node is ready 2021-10-13T04:15:40.953 DEBUG:teuthology.orchestra.run.smithi018:> hostname 2021-10-13T04:15:40.970 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018.front.sepia.ceph.com 2021-10-13T04:15:40.970 DEBUG:teuthology.orchestra.run.smithi018:> grep smithi018.front.sepia.ceph.com /etc/hosts 2021-10-13T04:15:41.027 INFO:teuthology.orchestra.run.smithi018.stdout:172.21.15.18 smithi018.front.sepia.ceph.com smithi018 2021-10-13T04:15:41.028 DEBUG:teuthology.orchestra.run.smithi018:> sudo hostname smithi018 2021-10-13T04:15:41.152 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i -e 's/smithi018.front.sepia.ceph.com/smithi018/g' /etc/hosts 2021-10-13T04:15:41.179 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i -e 's/172.21.15.18/172.21.15.18/g' /etc/hosts 2021-10-13T04:15:41.247 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i -e 's/smithi018.front.sepia.ceph.com/smithi018/g' /etc/hostname 2021-10-13T04:15:41.313 DEBUG:teuthology.orchestra.run.smithi018:> sudo hostname smithi018 2021-10-13T04:15:41.379 INFO:teuthology.provision.fog.smithi018:Deploy complete! 2021-10-13T04:15:41.380 DEBUG:teuthology.parallel:result is None 2021-10-13T04:15:41.506 INFO:teuthology.lock.ops:Checking smithi018.front.sepia.ceph.com 2021-10-13T04:15:41.507 INFO:teuthology.lock.ops:New key found. Updating... 2021-10-13T04:15:41.524 INFO:teuthology.lock.ops:Updating [smithi018.front.sepia.ceph.com]: set os type and version on server 2021-10-13T04:15:41.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2021-10-13T04:15:41.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-13T04:15:41.591 DEBUG:teuthology.orchestra.run.smithi018:> true 2021-10-13T04:15:41.701 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2021-10-13T04:15:41.756 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2021-10-13T04:15:41.757 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2021-10-13T04:15:41.811 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="CentOS Stream" 2021-10-13T04:15:41.812 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="8" 2021-10-13T04:15:41.812 INFO:teuthology.orchestra.run.smithi018.stdout:ID="centos" 2021-10-13T04:15:41.812 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="rhel fedora" 2021-10-13T04:15:41.813 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="8" 2021-10-13T04:15:41.813 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el8" 2021-10-13T04:15:41.813 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="CentOS Stream 8" 2021-10-13T04:15:41.813 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31" 2021-10-13T04:15:41.813 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-13T04:15:41.814 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://centos.org/" 2021-10-13T04:15:41.814 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-10-13T04:15:41.814 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-10-13T04:15:41.814 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-10-13T04:15:41.816 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2021-10-13T04:15:41.832 INFO:teuthology.lock.ops:1 smithi machines locked this try, 1/1 locked so far 2021-10-13T04:15:41.853 INFO:teuthology.lock.ops:Locked targets: smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKLekA9P9Vgg/p6S6aRgRv/mLnEOH0IpCPs1Vpq9l+4lRGETE/C6Qzanmd5tZOfShK4wXgxgydxqdXL39kh8miexGerNZ+XRYXGxWDAQPmllWzJGbYlew/bp2rmD8bQ7PJpQ/6I5TFz5eAUja4eYqD0YwZc0Yfe9+ahsQCAGaqzg1v8vcD48VAs4ujn9u/FVcqYojANSZWyGXW49VPw2tJVqJVsWB6ion7IFl5n61HZcR/AG8Mz/73HUavTA9pJXT2b0IdmwRnwmlze/M8cHdMMwokHmIzyQqoLUlu+a7TPFHQZ1k2tGHr9C5uAblAnBx57/GkWsTpyqjcuReGAOoUPJrnbs6bxQAhSW/CpjHKa89m4xutKnLQxlEIfceAYfCgqevfOoSNWCzw/0fYc6vQiFYN4M9NjySSKXupnuA80ubcC3bVc3n1t9UtztqrDz/HsonubUC5EpTTIkSVJT/tZdsOAPZfjoyiNkkL+f4Mt8kDAqGIFco8YghlTNywL8E= 2021-10-13T04:15:41.853 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-10-13T04:15:41.905 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-10-13T04:15:41.914 INFO:teuthology.task.internal:Saving configuration 2021-10-13T04:15:41.927 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-10-13T04:15:41.950 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-13T04:15:41.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-10-12_12:50:50-rados-wip-yuri7-testing-2021-10-08-0924-distro-basic-smithi/6433627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-10-13 04:11:03.192830', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyTlk4gYK8IB8VgHtazTuANdkI7aQOftG04qz+1cniaQv86T8IpfKkiF74SxozxyWOtsp2GusP2DM1XvnCzRJA='} 2021-10-13T04:15:41.970 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-10-13T04:15:41.978 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2021-10-13T04:15:41.978 INFO:teuthology.run_tasks:Running task console_log... 2021-10-13T04:15:42.023 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbb8f6ff9d8>, signals=[15]) 2021-10-13T04:15:42.024 INFO:teuthology.run_tasks:Running task internal.connect... 2021-10-13T04:15:42.033 INFO:teuthology.task.internal:Opening connections... 2021-10-13T04:15:42.033 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2021-10-13T04:15:42.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-10-13T04:15:42.104 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-10-13T04:15:42.113 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2021-10-13T04:15:42.210 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2021-10-13T04:15:42.210 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2021-10-13T04:15:42.266 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="CentOS Stream" 2021-10-13T04:15:42.266 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="8" 2021-10-13T04:15:42.266 INFO:teuthology.orchestra.run.smithi018.stdout:ID="centos" 2021-10-13T04:15:42.266 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="rhel fedora" 2021-10-13T04:15:42.267 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="8" 2021-10-13T04:15:42.267 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el8" 2021-10-13T04:15:42.267 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="CentOS Stream 8" 2021-10-13T04:15:42.267 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31" 2021-10-13T04:15:42.267 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-10-13T04:15:42.268 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://centos.org/" 2021-10-13T04:15:42.268 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-10-13T04:15:42.268 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-10-13T04:15:42.268 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-10-13T04:15:42.269 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2021-10-13T04:15:42.287 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-10-13T04:15:42.299 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-10-13T04:15:42.313 INFO:teuthology.task.internal:Checking for old test directory... 2021-10-13T04:15:42.313 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2021-10-13T04:15:42.328 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-10-13T04:15:42.338 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-10-13T04:15:42.338 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2021-10-13T04:15:42.395 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-10-13T04:15:42.426 INFO:teuthology.run_tasks:Running task kernel... 2021-10-13T04:15:42.445 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-10-13T04:15:42.445 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}} 2021-10-13T04:15:42.446 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-10-13T04:15:42.446 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2021-10-13T04:15:42.462 INFO:teuthology.orchestra.run.smithi018.stdout:4.18.0-301.1.el8.x86_64 2021-10-13T04:15:42.463 INFO:teuthology.task.kernel:Running kernel on smithi018: 4.18.0-301.1.el8.x86_64 2021-10-13T04:15:42.463 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum install -y kernel 2021-10-13T04:15:45.117 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-8 - AppStream 35 MB/s | 15 MB 00:00 2021-10-13T04:15:48.042 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-8 - Base 44 MB/s | 12 MB 00:00 2021-10-13T04:15:49.940 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-8 - Extras 218 kB/s | 15 kB 00:00 2021-10-13T04:15:51.264 INFO:teuthology.orchestra.run.smithi018.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 10 MB 00:01 2021-10-13T04:15:53.668 INFO:teuthology.orchestra.run.smithi018.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-10-13T04:15:55.637 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-10-13T04:15:55.638 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-10-13T04:15:55.638 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-10-13T04:15:55.936 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2021-10-13T04:15:55.937 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2021-10-13T04:15:55.938 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2021-10-13T04:15:55.938 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2021-10-13T04:15:55.938 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2021-10-13T04:15:55.938 INFO:teuthology.orchestra.run.smithi018.stdout: kernel x86_64 4.18.0-338.el8 CentOS-Base 7.0 M 2021-10-13T04:15:55.939 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core x86_64 4.18.0-338.el8 CentOS-Base 38 M 2021-10-13T04:15:55.939 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules x86_64 4.18.0-338.el8 CentOS-Base 30 M 2021-10-13T04:15:55.939 INFO:teuthology.orchestra.run.smithi018.stdout:Removing: 2021-10-13T04:15:55.939 INFO:teuthology.orchestra.run.smithi018.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-10-13T04:15:55.939 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-10-13T04:15:55.940 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-10-13T04:15:55.940 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-10-13T04:15:55.940 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2021-10-13T04:15:55.940 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2021-10-13T04:15:55.941 INFO:teuthology.orchestra.run.smithi018.stdout:Install 3 Packages 2021-10-13T04:15:55.941 INFO:teuthology.orchestra.run.smithi018.stdout:Remove 3 Packages 2021-10-13T04:15:55.941 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-10-13T04:15:55.942 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 74 M 2021-10-13T04:15:55.942 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2021-10-13T04:15:56.131 INFO:teuthology.orchestra.run.smithi018.stdout:(1/3): kernel-4.18.0-338.el8.x86_64.rpm 36 MB/s | 7.0 MB 00:00 2021-10-13T04:15:57.228 INFO:teuthology.orchestra.run.smithi018.stdout:(2/3): kernel-core-4.18.0-338.el8.x86_64.rpm 29 MB/s | 38 MB 00:01 2021-10-13T04:15:57.451 INFO:teuthology.orchestra.run.smithi018.stdout:(3/3): kernel-modules-4.18.0-338.el8.x86_64.rpm 20 MB/s | 30 MB 00:01 2021-10-13T04:15:57.455 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2021-10-13T04:15:57.456 INFO:teuthology.orchestra.run.smithi018.stdout:Total 49 MB/s | 74 MB 00:01 2021-10-13T04:15:57.593 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2021-10-13T04:15:58.280 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2021-10-13T04:15:58.280 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2021-10-13T04:15:59.358 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2021-10-13T04:15:59.372 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2021-10-13T04:16:02.131 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2021-10-13T04:16:02.607 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-core-4.18.0-338.el8.x86_64 1/6 2021-10-13T04:16:06.975 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-4.18.0-338.el8.x86_64 1/6 2021-10-13T04:16:07.315 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-modules-4.18.0-338.el8.x86_64 2/6 2021-10-13T04:16:10.607 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-4.18.0-338.el8.x86_64 2/6 2021-10-13T04:16:10.770 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-4.18.0-338.el8.x86_64 3/6 2021-10-13T04:16:10.771 INFO:teuthology.orchestra.run.smithi018.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-10-13T04:16:10.978 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-10-13T04:16:10.979 INFO:teuthology.orchestra.run.smithi018.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-10-13T04:16:13.088 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-10-13T04:16:13.309 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-10-13T04:16:13.319 INFO:teuthology.orchestra.run.smithi018.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-10-13T04:16:16.303 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-10-13T04:16:38.317 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-4.18.0-338.el8.x86_64 6/6 2021-10-13T04:16:38.727 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-10-13T04:16:38.728 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-4.18.0-338.el8.x86_64 1/6 2021-10-13T04:16:38.728 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-core-4.18.0-338.el8.x86_64 2/6 2021-10-13T04:16:38.728 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-4.18.0-338.el8.x86_64 3/6 2021-10-13T04:16:38.728 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-10-13T04:16:38.728 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-10-13T04:16:39.623 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-10-13T04:16:39.623 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-10-13T04:16:39.624 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2021-10-13T04:16:39.624 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-4.18.0-338.el8.x86_64 kernel-core-4.18.0-338.el8.x86_64 2021-10-13T04:16:39.624 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-4.18.0-338.el8.x86_64 2021-10-13T04:16:39.624 INFO:teuthology.orchestra.run.smithi018.stdout:Removed: 2021-10-13T04:16:39.624 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-10-13T04:16:39.625 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-10-13T04:16:39.625 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-10-13T04:16:39.625 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2021-10-13T04:16:39.734 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel --last | head -n 1 2021-10-13T04:16:39.941 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-4.18.0-338.el8.x86_64 Wed 13 Oct 2021 04:16:10 AM UTC 2021-10-13T04:16:39.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-338.el8.x86_64 2021-10-13T04:16:39.943 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-338.el8.x86_64 2021-10-13T04:16:39.944 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-10-13T04:16:39.944 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-10-13T04:16:39.944 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-10-13T04:16:39.945 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2021-10-13T04:16:39.945 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel --last | head -n 1 2021-10-13T04:16:40.092 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-4.18.0-338.el8.x86_64 Wed 13 Oct 2021 04:16:10 AM UTC 2021-10-13T04:16:40.094 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-338.el8.x86_64 2021-10-13T04:16:40.094 DEBUG:teuthology.orchestra.run.smithi018:> sudo rpm -qi grub2-tools 2021-10-13T04:16:40.127 INFO:teuthology.orchestra.run.smithi018.stdout:Name : grub2-tools 2021-10-13T04:16:40.128 INFO:teuthology.orchestra.run.smithi018.stdout:Epoch : 1 2021-10-13T04:16:40.128 INFO:teuthology.orchestra.run.smithi018.stdout:Version : 2.02 2021-10-13T04:16:40.128 INFO:teuthology.orchestra.run.smithi018.stdout:Release : 99.el8 2021-10-13T04:16:40.129 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: x86_64 2021-10-13T04:16:40.129 INFO:teuthology.orchestra.run.smithi018.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-10-13T04:16:40.129 INFO:teuthology.orchestra.run.smithi018.stdout:Group : System Environment/Base 2021-10-13T04:16:40.129 INFO:teuthology.orchestra.run.smithi018.stdout:Size : 9264704 2021-10-13T04:16:40.129 INFO:teuthology.orchestra.run.smithi018.stdout:License : GPLv3+ 2021-10-13T04:16:40.130 INFO:teuthology.orchestra.run.smithi018.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-10-13T04:16:40.130 INFO:teuthology.orchestra.run.smithi018.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-10-13T04:16:40.130 INFO:teuthology.orchestra.run.smithi018.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-10-13T04:16:40.130 INFO:teuthology.orchestra.run.smithi018.stdout:Build Host : kbuilder.bsys.centos.org 2021-10-13T04:16:40.131 INFO:teuthology.orchestra.run.smithi018.stdout:Relocations : (not relocatable) 2021-10-13T04:16:40.131 INFO:teuthology.orchestra.run.smithi018.stdout:Packager : CentOS BuildSystem 2021-10-13T04:16:40.131 INFO:teuthology.orchestra.run.smithi018.stdout:Vendor : CentOS 2021-10-13T04:16:40.131 INFO:teuthology.orchestra.run.smithi018.stdout:URL : http://www.gnu.org/software/grub/ 2021-10-13T04:16:40.132 INFO:teuthology.orchestra.run.smithi018.stdout:Summary : Support tools for GRUB. 2021-10-13T04:16:40.132 INFO:teuthology.orchestra.run.smithi018.stdout:Description : 2021-10-13T04:16:40.132 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-10-13T04:16:40.132 INFO:teuthology.orchestra.run.smithi018.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-10-13T04:16:40.133 INFO:teuthology.orchestra.run.smithi018.stdout:customizable bootloader with modular architecture. It supports a rich 2021-10-13T04:16:40.133 INFO:teuthology.orchestra.run.smithi018.stdout:variety of kernel formats, file systems, computer architectures and 2021-10-13T04:16:40.133 INFO:teuthology.orchestra.run.smithi018.stdout:hardware devices. 2021-10-13T04:16:40.133 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-10-13T04:16:40.134 INFO:teuthology.orchestra.run.smithi018.stdout:This subpackage provides tools for support of all platforms. 2021-10-13T04:16:40.135 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-10-13T04:16:40.135 INFO:teuthology.task.kernel:Updating grub on smithi018 to boot 4.18.0-338.el8.x86_64 2021-10-13T04:16:40.135 DEBUG:teuthology.orchestra.run.smithi018:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-10-13T04:16:41.477 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2021-10-13T04:16:42.026 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2021-10-13T04:16:42.026 DEBUG:teuthology.task.console_log:Killing console logger for smithi018 2021-10-13T04:16:42.027 DEBUG:teuthology.task.console_log:Killing console logger for smithi018 2021-10-13T04:16:42.028 DEBUG:teuthology.exit:Finished running handlers