2020-11-12T17:20:47.751 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-12T17:20:47.757 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --block --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap msgr-failures/fastclose msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect_set_object} --archive /home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616045 --name yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi --lock -- /tmp/teuthology-worker.wu247x75.tmp 2020-11-12T17:20:47.809 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-12T17:20:47.881 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616045 branch: octopus description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap msgr-failures/fastclose msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect_set_object} email: ceph-qa@ceph.io first_in_suite: false job_id: '5616045' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: '8.1' overrides: admin_socket: branch: octopus ceph: conf: global: mon client directed command retry: 5 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd backoff on degraded: true osd backoff on peering: true osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd inject bad map crc probability: 0.1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd sloppy crc: true osd snap trim sleep: 2 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: octopus sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af sleep_before_teardown: 0 suite: rados suite_branch: octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph_octopus/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph osd set-require-min-compat-client luminous - ceph balancer mode upmap - ceph balancer on - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 low_tier_pool: low_tier objects: 500 op_weights: copy_from: 100 set_redirect: 100 ops: 4000 set_redirect: true teuthology_branch: master timestamp: 2020-11-12_17:03:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.11566 2020-11-12T17:20:47.882 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_octopus/qa; will attempt to use it 2020-11-12T17:20:47.883 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_octopus/qa/tasks 2020-11-12T17:20:47.884 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-12T17:20:47.885 INFO:teuthology.task.internal:Checking packages... 2020-11-12T17:20:47.933 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '31d4e76d0d584790763b4b1146b29ea4cfc2e9af' 2020-11-12T17:20:47.933 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-12T17:20:47.934 INFO:teuthology.packaging:ref: None 2020-11-12T17:20:47.934 INFO:teuthology.packaging:tag: None 2020-11-12T17:20:47.935 INFO:teuthology.packaging:branch: octopus 2020-11-12T17:20:47.935 INFO:teuthology.packaging:sha1: 31d4e76d0d584790763b4b1146b29ea4cfc2e9af 2020-11-12T17:20:47.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2020-11-12T17:20:48.244 INFO:teuthology.task.internal:Found packages for ceph version 15.2.5-585.g31d4e76d 2020-11-12T17:20:48.245 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-12T17:20:48.253 INFO:teuthology.task.internal:no buildpackages task found 2020-11-12T17:20:48.253 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-12T17:20:48.271 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-12T17:20:48.295 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-12T17:20:48.361 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:20:58.382 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:21:08.407 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:21:18.434 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-12T17:21:28.478 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:21:38.525 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:21:48.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:21:58.606 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:08.632 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:18.667 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:28.703 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:38.728 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:48.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:22:58.820 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:08.877 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:18.918 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:28.966 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:39.010 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:49.036 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:23:59.087 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-12T17:24:09.128 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuriw@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616045'} 2020-11-12T17:24:09.172 DEBUG:teuthology.lock.ops:locked ubuntu@smithi192.front.sepia.ceph.com, ubuntu@smithi186.front.sepia.ceph.com 2020-11-12T17:24:09.256 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fec52240730>, signals=[15]) 2020-11-12T17:24:09.497 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of centos 8.1 2020-11-12T17:24:09.531 INFO:teuthology.provision.fog.smithi192:Scheduling deploy of centos 8.1 2020-11-12T17:24:09.954 INFO:teuthology.orchestra.console:Power off smithi186 2020-11-12T17:24:09.956 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-12T17:24:10.010 INFO:teuthology.orchestra.console:Power off smithi192 2020-11-12T17:24:10.042 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-12T17:24:10.059 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:10.074 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:10.189 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2020-11-12T17:24:10.198 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2020-11-12T17:24:10.290 INFO:teuthology.orchestra.console:Power on smithi186 2020-11-12T17:24:10.291 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-12T17:24:10.303 INFO:teuthology.orchestra.console:Power on smithi192 2020-11-12T17:24:10.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-12T17:24:10.317 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:10.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:14.358 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:20.848 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-12T17:24:21.049 INFO:teuthology.orchestra.console:Power on for smithi186 completed 2020-11-12T17:24:21.107 INFO:teuthology.orchestra.console:Power on for smithi192 completed 2020-11-12T17:24:21.151 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish 2020-11-12T17:24:21.208 INFO:teuthology.provision.fog.smithi192:Waiting for deploy to finish 2020-11-12T17:26:52.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:26:52.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:01.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:01.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:10.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:10.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:19.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:19.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:28.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:28.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:38.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:38.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:47.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:47.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:27:56.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:05.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:18.801 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-12T17:28:18.815 INFO:teuthology.orchestra.run.smithi192:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-12T17:28:36.968 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-12T17:28:37.002 INFO:teuthology.orchestra.run.smithi186:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-12T17:28:38.747 INFO:teuthology.provision.fog.smithi192:Node is ready 2020-11-12T17:28:38.747 INFO:teuthology.orchestra.run.smithi192:> hostname 2020-11-12T17:28:38.767 INFO:teuthology.orchestra.run.smithi192.stdout:smithi192.front.sepia.ceph.com 2020-11-12T17:28:38.768 INFO:teuthology.orchestra.run.smithi192:> grep smithi192.front.sepia.ceph.com /etc/hosts 2020-11-12T17:28:38.827 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.15.192 smithi192.front.sepia.ceph.com smithi192 2020-11-12T17:28:38.828 INFO:teuthology.orchestra.run.smithi192:> sudo hostname smithi192 2020-11-12T17:28:38.977 INFO:teuthology.orchestra.run.smithi192:> sudo sed -i -e 's/smithi192.front.sepia.ceph.com/smithi192/g' /etc/hosts 2020-11-12T17:28:39.010 INFO:teuthology.orchestra.run.smithi192:> sudo sed -i -e 's/172.21.15.192/172.21.15.192/g' /etc/hosts 2020-11-12T17:28:39.080 INFO:teuthology.orchestra.run.smithi192:> sudo sed -i -e 's/smithi192.front.sepia.ceph.com/smithi192/g' /etc/hostname 2020-11-12T17:28:39.153 INFO:teuthology.orchestra.run.smithi192:> sudo hostname smithi192 2020-11-12T17:28:39.224 INFO:teuthology.provision.fog.smithi192:Deploy complete! 2020-11-12T17:28:39.225 DEBUG:teuthology.parallel:result is None 2020-11-12T17:28:57.275 INFO:teuthology.provision.fog.smithi186:Node is ready 2020-11-12T17:28:57.294 INFO:teuthology.orchestra.run.smithi186:> hostname 2020-11-12T17:28:57.312 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186.front.sepia.ceph.com 2020-11-12T17:28:57.313 INFO:teuthology.orchestra.run.smithi186:> grep smithi186.front.sepia.ceph.com /etc/hosts 2020-11-12T17:28:57.369 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.186 smithi186.front.sepia.ceph.com smithi186 2020-11-12T17:28:57.369 INFO:teuthology.orchestra.run.smithi186:> sudo hostname smithi186 2020-11-12T17:28:57.511 INFO:teuthology.orchestra.run.smithi186:> sudo sed -i -e 's/smithi186.front.sepia.ceph.com/smithi186/g' /etc/hosts 2020-11-12T17:28:57.542 INFO:teuthology.orchestra.run.smithi186:> sudo sed -i -e 's/172.21.15.186/172.21.15.186/g' /etc/hosts 2020-11-12T17:28:57.611 INFO:teuthology.orchestra.run.smithi186:> sudo sed -i -e 's/smithi186.front.sepia.ceph.com/smithi186/g' /etc/hostname 2020-11-12T17:28:57.679 INFO:teuthology.orchestra.run.smithi186:> sudo hostname smithi186 2020-11-12T17:28:57.746 INFO:teuthology.provision.fog.smithi186:Deploy complete! 2020-11-12T17:28:57.748 DEBUG:teuthology.parallel:result is None 2020-11-12T17:28:59.604 INFO:teuthology.lock.ops:Checking smithi192.front.sepia.ceph.com 2020-11-12T17:28:59.608 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-12T17:28:59.625 INFO:teuthology.lock.ops:Checking smithi186.front.sepia.ceph.com 2020-11-12T17:28:59.626 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-12T17:28:59.640 INFO:teuthology.lock.ops:Updating [smithi192.front.sepia.ceph.com]: set os type and version on server 2020-11-12T17:28:59.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-12T17:28:59.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:28:59.741 INFO:teuthology.orchestra.run.smithi192:> true 2020-11-12T17:28:59.904 INFO:teuthology.orchestra.run.smithi192:> uname -m 2020-11-12T17:28:59.959 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2020-11-12T17:28:59.959 INFO:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2020-11-12T17:29:00.012 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Linux" 2020-11-12T17:29:00.013 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8 (Core)" 2020-11-12T17:29:00.013 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2020-11-12T17:29:00.013 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2020-11-12T17:29:00.014 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8" 2020-11-12T17:29:00.014 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2020-11-12T17:29:00.014 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-12T17:29:00.014 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2020-11-12T17:29:00.014 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-12T17:29:00.015 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.centos.org/" 2020-11-12T17:29:00.015 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-12T17:29:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: 2020-11-12T17:29:00.015 INFO:teuthology.orchestra.run.smithi192.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-12T17:29:00.015 INFO:teuthology.orchestra.run.smithi192.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-12T17:29:00.015 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-12T17:29:00.016 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-12T17:29:00.016 INFO:teuthology.orchestra.run.smithi192.stdout: 2020-11-12T17:29:00.017 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2020-11-12T17:29:00.033 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: set os type and version on server 2020-11-12T17:29:00.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-12T17:29:00.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:00.106 INFO:teuthology.orchestra.run.smithi186:> true 2020-11-12T17:29:00.272 INFO:teuthology.orchestra.run.smithi186:> uname -m 2020-11-12T17:29:00.327 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2020-11-12T17:29:00.327 INFO:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2020-11-12T17:29:00.391 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Linux" 2020-11-12T17:29:00.391 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8 (Core)" 2020-11-12T17:29:00.391 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2020-11-12T17:29:00.392 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2020-11-12T17:29:00.392 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8" 2020-11-12T17:29:00.392 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2020-11-12T17:29:00.393 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-12T17:29:00.393 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2020-11-12T17:29:00.393 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-12T17:29:00.393 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.centos.org/" 2020-11-12T17:29:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-12T17:29:00.394 INFO:teuthology.orchestra.run.smithi186.stdout: 2020-11-12T17:29:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-12T17:29:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-12T17:29:00.395 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-12T17:29:00.395 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-12T17:29:00.395 INFO:teuthology.orchestra.run.smithi186.stdout: 2020-11-12T17:29:00.396 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2020-11-12T17:29:00.418 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-12T17:29:00.808 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyYN2+SQ+B5mKR2t4DrpUs3Pqr6tdQBKh9z2Uyjt5CZHqC9Y5SFWr5YYDyNSivR0FHBbY9Eo4GAo9o0UyFm4ec5x69ntwMkuUjYIdZPj1IvEbC69KuI9FBEKd4OhdIN0mzURNLb2RhQMIwfoPhkGrYfuvM2Hs6xJ+lsiLxbYyFis/dMMOmXSl2EIW5pQzLqBQ74xz1tH8J949gY5d2VFOsJH5gqtrny45RpTcatsFzFrn8V937LegwN94fF+GGt7qyj+K6hOUfdV4a9NyxlHCsidnK070GhDenTwmxg16GRJHyMU1wulZv5qJUiN5vn4NYxLPG1BSxi9vfr0uBf/WkVaPixXLWZXfbJLD+v8T6VntKMPTuRXKR7k9hiEu59oatmtz3A/5R4UWK1cDSaMMWHeu4/NP7/1Ii8028WuKNsBod6XZevBekWlHFf+wdPfNKLDDpciJ/6I3VO3usC1lp5d4N75cAykhCfMFw5CRNYUTGP4oEqSxIIeum/9/38Kk= smithi192.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoo3Kpa9so8M2O7Hc1tHb9I4xCzTrRzzH5ZmFF8YtYj854ivlu3W9VlokGhC/9UQFdItagQ7Ubzjz9zqHSrjb5bn189WvBCYgTMrZ27/QgKOH4NnAfaWRJA0EkHYac1/00IqjQzAOxH8iSvRWKxFezE9bOTNPITZLYaW2wyuVLY9dyM6gFEEHFwWUI/m3maxj/b0k1DFOopn+0bCUOhRBuRghD2dHRgMQ/8IdSYAZ4Q71AcFVKwDteKYZV+wKEb/8C21O7XldnqZ02og+B6Gba8AyUkPIPOY3h0+rZ3jWewb4sYrUn8UttD9CM/u+yQVUTp60OU/T9fXmR7d/CeHKnVuwA3i8JnhYiyZ+nHoxTTmHwv0Zv7/aBNnjiWN4veg/2/EDdBGxlFT7g32IsVBBKIzxVw7H/O5pDuStT2SjzatSSxZdIihl/ksJ11cfP3kz4mvTC8nb/OQRbCZRN4J35i4KZ66FwAp6xsNDzxjkt0eykyVVasZ3uHqfr3jGldzc= 2020-11-12T17:29:00.809 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-12T17:29:00.864 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-12T17:29:00.874 INFO:teuthology.task.internal:Saving configuration 2020-11-12T17:29:00.911 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-12T17:29:00.939 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-12T17:29:00.962 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-12 17:24:09.159814', 'description': '/home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616045', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC6sc0MGVVfvZbo/gdxmsmELkdU4v7ZwyAHqHc88lqU4tXI+ED/K1boBY3z1Pzqto9WB9g1CAC32/Hr5mXMoAvM=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi192.front.sepia.ceph.com'} 2020-11-12T17:29:00.980 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-12 17:24:09.159835', 'description': '/home/teuthworker/archive/yuriw-2020-11-12_17:03:16-rados-octopus-distro-basic-smithi/5616045', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPh/w5sex7CbgRxj01PvBjN4IJ2wuEM/bDwrMPgcrkERJgMiSKfYFuQc6WvYs/TeSSooYhi+7aMViA8m7350FfI=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi186.front.sepia.ceph.com'} 2020-11-12T17:29:00.981 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-12T17:29:01.034 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2020-11-12T17:29:01.034 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2020-11-12T17:29:01.034 INFO:teuthology.run_tasks:Running task console_log... 2020-11-12T17:29:03.544 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fec55cad9d8>, signals=[15]) 2020-11-12T17:29:03.549 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-12T17:29:07.145 INFO:teuthology.task.internal:Opening connections... 2020-11-12T17:29:07.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2020-11-12T17:29:07.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:07.343 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2020-11-12T17:29:07.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-12T17:29:07.419 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-12T17:29:07.429 INFO:teuthology.orchestra.run.smithi192:> uname -m 2020-11-12T17:29:07.468 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2020-11-12T17:29:07.469 INFO:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2020-11-12T17:29:07.522 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Linux" 2020-11-12T17:29:07.522 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8 (Core)" 2020-11-12T17:29:07.522 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2020-11-12T17:29:07.523 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2020-11-12T17:29:07.523 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8" 2020-11-12T17:29:07.523 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2020-11-12T17:29:07.523 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-12T17:29:07.523 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2020-11-12T17:29:07.523 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-12T17:29:07.524 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.centos.org/" 2020-11-12T17:29:07.524 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-12T17:29:07.524 INFO:teuthology.orchestra.run.smithi192.stdout: 2020-11-12T17:29:07.524 INFO:teuthology.orchestra.run.smithi192.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-12T17:29:07.524 INFO:teuthology.orchestra.run.smithi192.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-12T17:29:07.524 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-12T17:29:07.524 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-12T17:29:07.525 INFO:teuthology.orchestra.run.smithi192.stdout: 2020-11-12T17:29:07.529 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2020-11-12T17:29:07.551 INFO:teuthology.orchestra.run.smithi186:> uname -m 2020-11-12T17:29:07.566 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2020-11-12T17:29:07.567 INFO:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2020-11-12T17:29:07.620 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Linux" 2020-11-12T17:29:07.620 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8 (Core)" 2020-11-12T17:29:07.621 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2020-11-12T17:29:07.621 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2020-11-12T17:29:07.621 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8" 2020-11-12T17:29:07.621 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2020-11-12T17:29:07.621 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-11-12T17:29:07.621 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2020-11-12T17:29:07.622 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-11-12T17:29:07.622 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.centos.org/" 2020-11-12T17:29:07.622 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-11-12T17:29:07.622 INFO:teuthology.orchestra.run.smithi186.stdout: 2020-11-12T17:29:07.622 INFO:teuthology.orchestra.run.smithi186.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-11-12T17:29:07.622 INFO:teuthology.orchestra.run.smithi186.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-11-12T17:29:07.623 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-11-12T17:29:07.623 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-11-12T17:29:07.623 INFO:teuthology.orchestra.run.smithi186.stdout: 2020-11-12T17:29:07.623 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2020-11-12T17:29:07.657 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-12T17:29:07.790 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-12T17:29:07.805 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-12T17:29:07.808 INFO:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2020-11-12T17:29:07.814 INFO:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2020-11-12T17:29:07.828 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-12T17:29:07.868 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-12T17:29:07.869 INFO:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2020-11-12T17:29:07.872 INFO:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2020-11-12T17:29:07.896 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-12T17:29:08.056 INFO:teuthology.run_tasks:Running task kernel... 2020-11-12T17:29:08.141 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}} 2020-11-12T17:29:08.195 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-12T17:29:08.196 INFO:teuthology.orchestra.run.smithi192:> uname -r 2020-11-12T17:29:08.211 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-193.28.1.el8_2.x86_64 2020-11-12T17:29:08.212 INFO:teuthology.task.kernel:Running kernel on smithi192: 4.18.0-193.28.1.el8_2.x86_64 2020-11-12T17:29:08.212 INFO:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2020-11-12T17:29:10.598 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-8 - AppStream 46 MB/s | 7.0 MB 00:00 2020-11-12T17:29:11.855 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-8 - Base 23 MB/s | 2.2 MB 00:00 2020-11-12T17:29:12.298 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-8 - Extras 74 kB/s | 5.5 kB 00:00 2020-11-12T17:29:13.311 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 13 MB/s | 8.3 MB 00:00 2020-11-12T17:29:14.713 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 3.9 MB/s | 23 kB 00:00 2020-11-12T17:29:15.797 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-11-12T17:29:15.798 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2020-11-12T17:29:15.953 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2020-11-12T17:29:15.954 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2020-11-12T17:29:15.954 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2020-11-12T17:29:16.101 INFO:teuthology.orchestra.run.smithi192:> echo no | sudo yum reinstall kernel || true 2020-11-12T17:29:16.506 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Nov 2020 05:29:14 PM UTC. 2020-11-12T17:29:16.754 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-12T17:29:16.754 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-12T17:29:16.805 INFO:teuthology.orchestra.run.smithi192.stderr:Error: No packages marked for reinstall. 2020-11-12T17:29:16.859 INFO:teuthology.orchestra.run.smithi192:> sudo yum reinstall -y kernel || true 2020-11-12T17:29:17.264 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Nov 2020 05:29:14 PM UTC. 2020-11-12T17:29:17.512 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-12T17:29:17.512 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-12T17:29:17.562 INFO:teuthology.orchestra.run.smithi192.stderr:Error: No packages marked for reinstall. 2020-11-12T17:29:17.608 INFO:teuthology.orchestra.run.smithi192:> rpm -q kernel --last | head -n 1 2020-11-12T17:29:17.679 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-193.28.1.el8_2.x86_64 Sun 08 Nov 2020 12:14:50 PM UTC 2020-11-12T17:29:17.680 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-193.28.1.el8_2.x86_64 2020-11-12T17:29:17.680 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-12T17:29:17.681 INFO:teuthology.orchestra.run.smithi186:> uname -r 2020-11-12T17:29:17.697 INFO:teuthology.orchestra.run.smithi186.stdout:4.18.0-193.28.1.el8_2.x86_64 2020-11-12T17:29:17.698 INFO:teuthology.task.kernel:Running kernel on smithi186: 4.18.0-193.28.1.el8_2.x86_64 2020-11-12T17:29:17.698 INFO:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2020-11-12T17:29:20.146 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-8 - AppStream 37 MB/s | 7.0 MB 00:00 2020-11-12T17:29:21.331 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-8 - Base 26 MB/s | 2.2 MB 00:00 2020-11-12T17:29:21.759 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-8 - Extras 85 kB/s | 5.5 kB 00:00 2020-11-12T17:29:22.217 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 20 MB/s | 8.3 MB 00:00 2020-11-12T17:29:23.903 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 4.0 MB/s | 23 kB 00:00 2020-11-12T17:29:24.832 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-11-12T17:29:24.832 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2020-11-12T17:29:24.983 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2020-11-12T17:29:24.984 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2020-11-12T17:29:24.984 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2020-11-12T17:29:25.093 INFO:teuthology.orchestra.run.smithi186:> echo no | sudo yum reinstall kernel || true 2020-11-12T17:29:25.536 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:02 ago on Thu 12 Nov 2020 05:29:23 PM UTC. 2020-11-12T17:29:25.785 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-12T17:29:25.785 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-12T17:29:25.842 INFO:teuthology.orchestra.run.smithi186.stderr:Error: No packages marked for reinstall. 2020-11-12T17:29:25.910 INFO:teuthology.orchestra.run.smithi186:> sudo yum reinstall -y kernel || true 2020-11-12T17:29:26.312 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:03 ago on Thu 12 Nov 2020 05:29:23 PM UTC. 2020-11-12T17:29:26.559 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-11-12T17:29:26.559 INFO:teuthology.orchestra.run.smithi186.stdout:Installed package kernel-4.18.0-193.28.1.el8_2.x86_64 (from BaseOS) not available. 2020-11-12T17:29:26.628 INFO:teuthology.orchestra.run.smithi186.stderr:Error: No packages marked for reinstall. 2020-11-12T17:29:26.677 INFO:teuthology.orchestra.run.smithi186:> rpm -q kernel --last | head -n 1 2020-11-12T17:29:26.747 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-4.18.0-193.28.1.el8_2.x86_64 Sun 08 Nov 2020 12:14:50 PM UTC 2020-11-12T17:29:26.748 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-193.28.1.el8_2.x86_64 2020-11-12T17:29:26.749 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-12T17:29:26.749 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-11-12T17:29:26.749 INFO:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-12T17:29:26.789 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2020-11-12T17:29:26.808 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2020-11-12T17:29:26.809 INFO:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-12T17:29:26.843 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2020-11-12T17:29:26.857 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-12T17:29:26.867 INFO:teuthology.task.internal:Creating test directory... 2020-11-12T17:29:26.868 INFO:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-12T17:29:26.899 INFO:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-12T17:29:26.923 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-12T17:29:27.938 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-12T17:29:27.947 INFO:teuthology.task.internal:Creating archive directory... 2020-11-12T17:29:27.947 INFO:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-12T17:29:27.950 INFO:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-12T17:29:28.145 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-12T17:29:28.155 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-12T17:29:28.155 INFO:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-12T17:29:28.158 INFO:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-12T17:29:28.190 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-12T17:29:28.194 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-12T17:29:28.205 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-12T17:29:28.209 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-12T17:29:28.210 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-12T17:29:28.220 INFO:teuthology.task.internal:Configuring sudo... 2020-11-12T17:29:28.221 INFO:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-12T17:29:28.253 INFO:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-12T17:29:28.391 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-12T17:29:28.407 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-12T17:29:28.408 INFO:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-12T17:29:28.411 INFO:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-12T17:29:28.446 INFO:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-12T17:29:28.524 INFO:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-12T17:29:28.608 INFO:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-12T17:29:28.644 INFO:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-12T17:29:28.694 INFO:teuthology.orchestra.run.smithi192:> set -ex 2020-11-12T17:29:28.696 INFO:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-12T17:29:28.758 INFO:teuthology.orchestra.run.smithi192.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-12T17:29:28.800 INFO:teuthology.orchestra.run.smithi192.stderr:0+1 records in 2020-11-12T17:29:28.800 INFO:teuthology.orchestra.run.smithi192.stderr:0+1 records out 2020-11-12T17:29:28.800 INFO:teuthology.orchestra.run.smithi192.stderr:152 bytes copied, 0.0399632 s, 3.8 kB/s 2020-11-12T17:29:28.801 INFO:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-12T17:29:28.836 INFO:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-12T17:29:28.904 INFO:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-12T17:29:28.979 INFO:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-12T17:29:29.030 INFO:teuthology.orchestra.run.smithi186:> set -ex 2020-11-12T17:29:29.031 INFO:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-12T17:29:29.084 INFO:teuthology.orchestra.run.smithi186.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-12T17:29:29.098 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records in 2020-11-12T17:29:29.098 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records out 2020-11-12T17:29:29.098 INFO:teuthology.orchestra.run.smithi186.stderr:152 bytes copied, 3.3043e-05 s, 4.6 MB/s 2020-11-12T17:29:29.099 INFO:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2020-11-12T17:29:29.142 INFO:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2020-11-12T17:29:29.215 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-12T17:29:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-12T17:29:29.666 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-12T17:29:29.676 INFO:teuthology.task.internal:Starting timer... 2020-11-12T17:29:29.676 INFO:teuthology.run_tasks:Running task pcp... 2020-11-12T17:29:33.432 INFO:teuthology.run_tasks:Running task selinux... 2020-11-12T17:29:33.451 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-11-12T17:29:33.454 INFO:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2020-11-12T17:29:33.517 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: [ OK ] 2020-11-12T17:29:33.519 INFO:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2020-11-12T17:29:33.594 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: [ OK ] 2020-11-12T17:29:33.595 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-12T17:29:33.596 INFO:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2020-11-12T17:29:33.621 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2020-11-12T17:29:33.622 INFO:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2020-11-12T17:29:33.641 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2020-11-12T17:29:33.641 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi192.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2020-11-12T17:29:33.642 INFO:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-12T17:29:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-12T17:29:33.690 INFO:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-12T17:29:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-12T17:29:33.719 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-12T17:29:33.720 INFO:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2020-11-12T17:29:33.758 INFO:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2020-11-12T17:29:33.794 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-12T17:29:33.825 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-11-12T17:29:38.251 INFO:teuthology.repo_utils:Fetching master from origin 2020-11-12T17:30:10.734 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-12T17:30:15.670 DEBUG:teuthology.task.console_log:Killing console logger for smithi192 2020-11-12T17:30:15.673 DEBUG:teuthology.task.console_log:Killing console logger for smithi186 2020-11-12T17:30:15.674 DEBUG:teuthology.task.console_log:Killing console logger for smithi192 2020-11-12T17:30:15.674 DEBUG:teuthology.task.console_log:Killing console logger for smithi186 2020-11-12T17:30:15.675 DEBUG:teuthology.exit:Finished running handlers