2020-04-07T15:04:19.149 INFO:root:teuthology version: 1.0.0-b1b3d82 2020-04-07T15:04:19.152 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi/4931148 --block -- --name kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi /tmp/teuthology-worker._2RK8W.tmp --verbose --lock --description rados/upgrade/mimic-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/mimic.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-octopus.yaml 8-workload/{rbd-python.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} --owner scheduled_kchai@teuthology 2020-04-07T15:04:19.245 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-04-07T15:04:19.325 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi/4931148 branch: wip-kefu-testing-2020-04-07-1603 description: rados/upgrade/mimic-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/mimic.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-octopus.yaml 8-workload/{rbd-python.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} email: null first_in_suite: false job_id: '4931148' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'Run ceph on two nodes, with a separate client-only node. Use xfs beneath the osds. ' - desc: install ceph/mimic latest - desc: 'install upgrade ceph/-x on one node only 1st half restart : osd.0,1,2,3,4,5 ' - desc: 'randomly kill and revive osd small chance to increase the number of pgs ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'librbd C and C++ api tests ' - desc: 'install upgrade on remaining node restartin remaining osds ' - desc: 'librbd python api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' name: kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi nuke-on-error: true openstack: - machine: disk: 100 - volumes: count: 3 size: 30 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-kefu-testing-2020-04-07-1603 ceph: conf: global: bluestore warn on legacy statfs: false bluestore warn on no per pool omap: false mon pg warn min per osd: 0 ms bind msgr2: false ms dump corrupt message level: 0 mds: debug mds: 20 debug ms: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(MGR_DOWN\) - slow request - but it is still running - objects unfound and apparently lost - log bound mismatch - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_ - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request mon_bind_addrvec: false mon_bind_msgr2: false sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2020-04-07-2234 sha1: 2a27f6eef9dc660ad72ab6b9373d474c164cdf6a owner: scheduled_kchai@teuthology priority: 30 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - osd.4 - osd.5 - osd.6 - osd.7 - - mon.c - mgr.x - mgr.y - osd.8 - osd.9 - osd.10 - osd.11 - - client.0 sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 split_tasks: sequential: - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgpnum_fix: 1 disable_objectstore_tool_tests: true timeout: 1200 - print: '**** done thrashosds 3-thrash' - workunit: branch: mimic clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: mimic clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' - full_sequential: - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - radosbench: clients: - client.0 time: 150 - print: '**** done radosbench 7-workload' - workunit: branch: mimic clients: client.0: - rbd/test_librbd.sh - print: '**** done rbd/test_librbd.sh 7-workload' suite: rados suite_branch: wip-kefu-testing-2020-04-07-2234 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-kefu-testing-2020-04-07-2234/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 2a27f6eef9dc660ad72ab6b9373d474c164cdf6a tasks: - install: branch: mimic exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm extra_packages: - librados2 - print: '**** done install mimic' - ceph: null - print: '**** done ceph' - install.upgrade: mon.a: null mon.b: null - print: '**** done install.upgrade osd.0' - ceph.restart: daemons: - mon.a - mon.b mon-health-to-clog: false wait-for-healthy: false - ceph.restart: daemons: - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 wait-for-healthy: false - print: '**** done ceph.restart 1st 2/3s' - parallel: - split_tasks - install.upgrade: mon.c: null - ceph.restart: daemons: - mon.c - mgr.x - mgr.y wait-for-healthy: false wait-for-up: true - ceph.restart: daemons: - osd.8 - osd.9 - osd.10 - osd.11 wait-for-healthy: false wait-for-up: true - ceph.restart: daemons: - mds.a wait-for-healthy: false wait-for-up: true - exec: mon.a: - ceph mon enable-msgr2 - install.upgrade: client.0: null - exec: osd.0: - ceph osd require-osd-release octopus - ceph osd set-require-min-compat-client octopus - for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done - ceph.healthy: null - workunit: clients: client.0: - rbd/test_librbd_python.sh - print: '**** done rbd/test_librbd_python.sh 9-workload' - rados: clients: - client.0 objects: 500 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false teuthology_branch: wip-py3 tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2417 2020-04-07T15:04:19.326 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-kefu-testing-2020-04-07-2234/qa; will attempt to use it 2020-04-07T15:04:19.326 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-kefu-testing-2020-04-07-2234/qa/tasks 2020-04-07T15:04:19.327 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-04-07T15:04:19.329 INFO:teuthology.task.internal:Checking packages... 2020-04-07T15:04:19.347 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'f66d07eeb60a8cf6e623ab16f545b8e195f4d964' 2020-04-07T15:04:19.347 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-04-07T15:04:19.348 INFO:teuthology.packaging:ref: None 2020-04-07T15:04:19.349 INFO:teuthology.packaging:tag: None 2020-04-07T15:04:19.349 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-04-07-1603 2020-04-07T15:04:19.349 INFO:teuthology.packaging:sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 2020-04-07T15:04:19.350 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-kefu-testing-2020-04-07-1603 2020-04-07T15:04:19.594 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-486-gf66d07e-1bionic 2020-04-07T15:04:19.594 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-04-07T15:04:19.605 INFO:teuthology.task.internal:no buildpackages task found 2020-04-07T15:04:19.605 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-04-07T15:04:19.698 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-04-07T15:04:19.699 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-04-07T15:04:19.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-04-07T15:04:29.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:04:39.785 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-04-07T15:04:49.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:04:59.817 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:05:09.833 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:05:19.849 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:05:29.866 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:05:39.885 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:05:49.904 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:05:59.921 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:06:09.939 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:06:19.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:06:29.978 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:06:39.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:06:50.013 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:07:00.031 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:07:10.049 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:07:20.065 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:07:30.083 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:07:40.099 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:07:50.116 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:08:00.134 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:08:10.152 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:08:20.169 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:08:30.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:08:40.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:08:50.217 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:09:00.234 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:09:10.252 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:09:20.270 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:09:30.286 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:09:40.306 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:09:50.328 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:10:00.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:10:10.363 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:10:20.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:10:30.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:10:40.410 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:10:50.425 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:11:00.445 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:11:10.462 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:11:20.478 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:11:30.494 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:11:40.509 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:11:50.528 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:12:00.545 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:12:10.563 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:12:20.579 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:12:30.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:12:40.625 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:12:50.640 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 1)... 2020-04-07T15:13:00.658 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:13:10.674 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:13:20.693 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:13:30.708 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:13:40.724 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)... 2020-04-07T15:13:50.741 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 4, have 0)...