2020-06-15T13:44:16.975 INFO:root:teuthology version: 1.0.0-b8c39ea 2020-06-15T13:44:16.978 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150363 --name kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi --verbose --description rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/mimic-v1only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/on.yaml distro$/{centos_7.6.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/default.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} --owner scheduled_kchai@teuthology --lock --block -- /tmp/teuthology-worker.o6d37lkv.tmp 2020-06-15T13:44:17.015 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-15T13:44:17.083 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150363 branch: wip-kefu-testing-2020-06-15-1050 description: rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/mimic-v1only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/on.yaml distro$/{centos_7.6.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/default.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '5150363' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'rbd object class functional tests ' name: kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-kefu-testing-2020-06-15-1050 ceph: conf: global: mon client directed command retry: 5 ms bind msgr2: false ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd backoff on peering: true osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request mon_bind_msgr2: false sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-kefu-testing-2020-06-15-1050 sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2020-06-15-1050 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-06-15-1050/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 tasks: - install: branch: mimic exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-immutable-object-cache - ceph-base - python3-rados - python3-rgw - python3-rbd - python3-cephfs - librados-devel extra_packages: - librados2 - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: null - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.2: - ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27828 2020-06-15T13:44:17.084 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-06-15-1050/qa; will attempt to use it 2020-06-15T13:44:17.085 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-06-15-1050/qa/tasks 2020-06-15T13:44:17.085 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-06-15T13:44:17.088 INFO:teuthology.task.internal:Checking packages... 2020-06-15T13:44:17.113 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '76f35ab4e1d5fd49cb6131e102a9258075936e41' 2020-06-15T13:44:17.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-06-15T13:44:17.114 INFO:teuthology.packaging:ref: None 2020-06-15T13:44:17.115 INFO:teuthology.packaging:tag: None 2020-06-15T13:44:17.116 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-06-15-1050 2020-06-15T13:44:17.116 INFO:teuthology.packaging:sha1: 76f35ab4e1d5fd49cb6131e102a9258075936e41 2020-06-15T13:44:17.118 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2020-06-15-1050 2020-06-15T13:44:17.313 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-2495.g76f35ab 2020-06-15T13:44:17.314 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-06-15T13:44:17.336 INFO:teuthology.task.internal:no buildpackages task found 2020-06-15T13:44:17.337 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-06-15T13:44:17.389 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-06-15T13:44:17.390 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-15T13:44:17.460 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:44:27.487 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:44:37.522 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:44:47.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:44:57.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:45:07.595 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:45:17.620 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-06-15_13:24:10-rados-wip-kefu-testing-2020-06-15-1050-distro-basic-smithi/5150363'} 2020-06-15T13:45:17.639 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 3 smithi nodes. 2020-06-15T13:45:17.639 ERROR:teuthology.lock.ops:{"message": "only 2 nodes available"} 2020-06-15T13:45:17.640 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/3 locked so far 2020-06-15T13:45:17.640 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 3 more 2020-06-15T13:45:17.641 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting... 2020-06-15T13:45:27.679 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:45:37.700 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:45:47.739 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:45:57.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:46:07.791 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:46:17.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:46:27.829 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:46:37.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:46:47.871 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:46:57.893 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:47:07.915 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:47:17.931 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:47:27.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:47:37.973 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:47:47.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:47:58.020 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:48:08.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:48:18.065 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:48:28.086 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:48:38.103 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:48:48.121 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:48:58.144 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:49:08.165 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:49:18.184 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:49:28.219 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:49:38.239 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:49:48.255 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:49:58.289 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:50:08.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:50:18.324 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:50:28.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:50:38.370 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:50:48.389 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:50:58.416 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:51:08.435 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:51:18.457 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:51:28.477 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:51:38.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:51:48.516 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:51:58.536 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:52:08.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:52:18.578 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:52:28.595 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:52:38.629 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:52:48.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:52:58.671 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:53:08.688 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:53:18.708 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:53:28.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:53:38.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:53:48.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:53:58.791 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:54:08.817 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:54:18.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:54:28.851 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:54:38.874 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:54:48.895 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:54:58.912 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:55:08.938 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:55:18.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-06-15T13:55:28.980 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:55:38.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:55:49.020 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:55:59.041 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:56:09.051 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-06-15T13:56:19.071 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:56:29.111 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:56:39.140 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:56:49.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:56:59.179 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:57:09.197 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:57:19.224 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:57:29.245 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-06-15T13:57:39.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...