2021-01-23T12:32:28.354 INFO:root:teuthology version: 1.0.0-fe79ba1b 2021-01-23T12:32:28.356 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-01-23_12:09:34-rados-wip-kefu-testing-2021-01-23-1743-distro-basic-smithi/5819843 --lock --block --name kchai-2021-01-23_12:09:34-rados-wip-kefu-testing-2021-01-23-1743-distro-basic-smithi --description rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/radosbench} -- /tmp/teuthology-worker.v3681sux.tmp 2021-01-23T12:32:28.393 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-23T12:32:28.462 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-01-23_12:09:34-rados-wip-kefu-testing-2021-01-23-1743-distro-basic-smithi/5819843 branch: wip-kefu-testing-2021-01-23-1743 description: rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/radosbench} email: tchaikov@gmail.com first_in_suite: false job_id: '5819843' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-01-23_12:09:34-rados-wip-kefu-testing-2021-01-23-1743-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-kefu-testing-2021-01-23-1743 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd backoff on degraded: true osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd max markdown count: 1000 osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f8316f3bee54b62f33109ca7e4c7571a55013951 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: f8316f3bee54b62f33109ca7e4c7571a55013951 thrashosds: min_in: 4 workunit: branch: wip-kefu-testing-2021-01-23-1743 sha1: f8316f3bee54b62f33109ca7e4c7571a55013951 owner: scheduled_kchai@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 sha1: f8316f3bee54b62f33109ca7e4c7571a55013951 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-01-23-1743 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-23-1743/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: f8316f3bee54b62f33109ca7e4c7571a55013951 tasks: - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: null - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology_branch: master timestamp: 2021-01-23_12:09:34 tube: smithi user: kchai verbose: false verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.9386 2021-01-23T12:32:28.463 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-23-1743/qa; will attempt to use it 2021-01-23T12:32:28.463 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2021-01-23-1743/qa/tasks 2021-01-23T12:32:28.464 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2021-01-23T12:32:28.467 INFO:teuthology.task.internal.lock_machines:Locking machines... 2021-01-23T12:32:28.467 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-23T12:32:28.513 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2021-01-23_12:09:34-rados-wip-kefu-testing-2021-01-23-1743-distro-basic-smithi/5819843'} 2021-01-23T12:32:28.535 DEBUG:teuthology.lock.ops:locked ubuntu@smithi139.front.sepia.ceph.com, ubuntu@smithi086.front.sepia.ceph.com, ubuntu@smithi165.front.sepia.ceph.com 2021-01-23T12:32:28.613 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd09b476d90>, signals=[15]) 2021-01-23T12:32:28.819 INFO:teuthology.provision.fog.smithi086:Scheduling deploy of ubuntu 18.04 2021-01-23T12:32:28.824 INFO:teuthology.provision.fog.smithi165:Scheduling deploy of ubuntu 18.04 2021-01-23T12:32:28.826 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of ubuntu 18.04 2021-01-23T12:32:29.152 INFO:teuthology.orchestra.console:Power off smithi086 2021-01-23T12:32:29.153 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-01-23T12:32:29.172 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:29.188 INFO:teuthology.orchestra.console:Power off smithi139 2021-01-23T12:32:29.189 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-01-23T12:32:29.201 INFO:teuthology.orchestra.console:Power off smithi165 2021-01-23T12:32:29.201 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-01-23T12:32:29.209 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:29.221 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:29.309 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2021-01-23T12:32:29.332 INFO:teuthology.orchestra.console:Power off for smithi139 completed 2021-01-23T12:32:29.342 INFO:teuthology.orchestra.console:Power off for smithi165 completed 2021-01-23T12:32:29.410 INFO:teuthology.orchestra.console:Power on smithi086 2021-01-23T12:32:29.411 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-01-23T12:32:29.432 INFO:teuthology.orchestra.console:Power on smithi139 2021-01-23T12:32:29.433 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-01-23T12:32:29.441 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:29.449 INFO:teuthology.orchestra.console:Power on smithi165 2021-01-23T12:32:29.450 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-01-23T12:32:29.457 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:29.468 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:33.466 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:33.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:33.491 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:37.578 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:37.589 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:37.601 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-23T12:32:37.789 INFO:teuthology.orchestra.console:Power on for smithi086 completed 2021-01-23T12:32:37.856 INFO:teuthology.orchestra.console:Power on for smithi139 completed 2021-01-23T12:32:37.875 INFO:teuthology.orchestra.console:Power on for smithi165 completed 2021-01-23T12:32:37.890 INFO:teuthology.provision.fog.smithi086:Waiting for deploy to finish 2021-01-23T12:32:37.957 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish 2021-01-23T12:32:37.976 INFO:teuthology.provision.fog.smithi165:Waiting for deploy to finish 2021-01-23T12:33:18.944 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-01-23T12:33:18.945 DEBUG:teuthology.task.console_log:Killing console logger for smithi139 2021-01-23T12:33:18.945 DEBUG:teuthology.task.console_log:Killing console logger for smithi086 2021-01-23T12:33:18.946 DEBUG:teuthology.task.console_log:Killing console logger for smithi165 2021-01-23T12:33:18.946 DEBUG:teuthology.exit:Finished running handlers