2021-10-25T11:42:29.096 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-25T11:42:29.098 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/trociny-2021-10-25_11:34:24-rbd-wip-mgolub-testing-distro-basic-smithi/6460347 --owner scheduled_trociny@teuthology --name trociny-2021-10-25_11:34:24-rbd-wip-mgolub-testing-distro-basic-smithi --verbose --description rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd} -- /home/teuthworker/archive/trociny-2021-10-25_11:34:24-rbd-wip-mgolub-testing-distro-basic-smithi/6460347/orig.config.yaml 2021-10-25T11:42:29.135 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-10-25T11:42:29.196 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/trociny-2021-10-25_11:34:24-rbd-wip-mgolub-testing-distro-basic-smithi/6460347 branch: wip-mgolub-testing description: rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd} email: to.my.trociny@gmail.com first_in_suite: false job_id: '6460347' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: trociny-2021-10-25_11:34:24-rbd-wip-mgolub-testing-distro-basic-smithi nuke-on-error: true openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-mgolub-testing ceph: conf: global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0efa758f5dd904ae1b1b1db0654cffb844da3f9f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: extra_packages: - rbd-nbd sha1: 0efa758f5dd904ae1b1b1db0654cffb844da3f9f selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mgolub-testing sha1: 0efa758f5dd904ae1b1b1db0654cffb844da3f9f owner: scheduled_trociny@teuthology priority: 200 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 sha1: 0efa758f5dd904ae1b1b1db0654cffb844da3f9f sleep_before_teardown: 0 suite: rbd suite_branch: wip-mgolub-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0efa758f5dd904ae1b1b1db0654cffb844da3f9f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0efa758f5dd904ae1b1b1db0654cffb844da3f9f targets: smithi026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1o5GQZs771ymtnrswFUqJd0zEivso1FosijiBgnn7+aMlkrwsQ2yk3nQNmcBlsfLzfGX9gWWVTz05wmjE2U1hEszeRpB5PdeD6/2+fczOueNGN6D2dGRoD4+42i5fSwlNpb8IhJvmbIo+r+YygH2p/ULJa2EQfnKGd3WfD7oR1n8aP4RwHSzmZtej+4SNvhHuqnC94Q5TX/rH0kXL3MZbMn224KiS/wsJZmtD69PALMBqr6+oyYYPOPASgFJiIIg9dzSqm54Gy4Z8CC1lrWY7WJQsJ0uC1NImyenE6O7N22vhSye1z2/vChScOuvNrzu62Tfma1SLPx0bK97J/qHHwHv024HxrqkzkgHYjQ/OUsvaCNCFvMdsHzMhDMKxZO2E/UMDzlaEZ7iWPujPIdz6SJ0kqrOmf2fAKQ31FAGSjch+VRNVvfRbYtdIMf+jl2CAxa928z9Laaqp1J49KC2cyXXfpCrT5hd3rVQduiCCvNAqfqdkiLbI7KbPa0Xol20= smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKmLR9Aznou4OXRiNMaqfDDelw2p1iysOEJF/JOXr3cGmL15bPn0ih69SJOwaqO4k3ePwtRfVJmMx6DSjJu6FTUHsIMZjeoLBvsmxIsBA28HJCSOju+r/Q43YgzVkD8woM5j6xvbP4pL06M6rwjNHSvFkTg/9dRm4GuhFnCCtjeqhB3QiyxGQRTG4vaR6zd3iAo59I1GdPVSxOXFTM65bDFDsJaKwkbkJDUOTsoO8nKdD15d4C2RRXbR/HCH7iX/5mUcMf5wC8U6qI3U+3wzvf/QOZ3D/gK/rHpnFwcD0Z69DvqSqAyC1iGh+4cidZMK3rv5EKiNX2a2zn3QvE0t6owkdb+JO0lAY73dzZw8oR/uJwCp5ryfUj5F+jTUbaB7ry61occva16qI4xnhcuOzZ52PdwnMhV5pU8p9mddkrdpn+ELZidQdFN5ZGpsbMox8xyzqlqcuqxYjkIzNNLgxz5iY5ElsAnOAL4GioWSlB/g+hz0AjCDrRJLFYIln0xUk= smithi181.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS/71+yzqqc7vXWRvRtC02XZeUyn0WSkz4ZzscbRNerji8LXtHSiUV5ivNDFWxz9Z+L3+bVX6xHlUhwAUCORTfKo3dpIfkLBtH4cTMrlshJJr8+w2iGCJeOrJijYXccPSYS5cAHEo43dlHmMOA+WVUSRxGCDQSX8ovKtuc4HiAmcOl7q1dJbVmPsLBPgBAzdJGcb+n5tUrIHhBEako4rbRNZKjn7TLkRKNjZZsj+DVSFinOveua66IkFfMZ8oUo73snlbTT45Rp2rEKOHPCxD3uLNXCWmr+bztcAs64qBjsx3cA2gpt2p7xYaTwn/tbZUEao2L5zuokJ/lt2yvb249cJYJmKOBnMUHvefdiKzWwP2OzT/yZvwyEjIHWYcU7ZFD2K2v6OwN0oZQbsIteYVnSEHryjSUo+QKx3jNcm/K29OX3tKBWe2Y1QcQStUnwh5HYBEj2FY1HqHibBSKzh49XR4SDDPHNyQUrpGIk66zP60cJthXLayiJ6jpZJ9EZdc= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology_branch: master teuthology_sha1: c56135d151713269e811ede3163c9743c2e269de timestamp: 2021-10-25_11:34:24 tube: smithi user: trociny verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.6888 2021-10-25T11:42:29.196 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0efa758f5dd904ae1b1b1db0654cffb844da3f9f/qa; will attempt to use it 2021-10-25T11:42:29.197 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0efa758f5dd904ae1b1b1db0654cffb844da3f9f/qa/tasks 2021-10-25T11:42:29.197 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-10-25T11:42:29.199 INFO:teuthology.task.internal:Checking packages... 2021-10-25T11:42:29.217 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '0efa758f5dd904ae1b1b1db0654cffb844da3f9f' 2021-10-25T11:42:29.218 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-10-25T11:42:29.218 INFO:teuthology.packaging:ref: None 2021-10-25T11:42:29.219 INFO:teuthology.packaging:tag: None 2021-10-25T11:42:29.219 INFO:teuthology.packaging:branch: wip-mgolub-testing 2021-10-25T11:42:29.219 INFO:teuthology.packaging:sha1: 0efa758f5dd904ae1b1b1db0654cffb844da3f9f 2021-10-25T11:42:29.220 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-mgolub-testing 2021-10-25T11:42:29.396 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-8482.g0efa758f 2021-10-25T11:42:29.397 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-10-25T11:42:29.407 INFO:teuthology.task.internal:no buildpackages task found 2021-10-25T11:42:29.408 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-10-25T11:42:29.455 INFO:teuthology.task.internal:Saving configuration 2021-10-25T11:42:29.469 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-10-25T11:42:29.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-25T11:42:29.522 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': None, 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIf3ujH7jYIZcuA2Pml4xsEcBhkD4+B/NvW4xpAbgC5aU5E63NmD4UkvGJ25i2VEKUfJy7iwdLMj0ZVRRABA/FM='} 2021-10-25T11:42:29.522 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi026.front.sepia.ceph.com is not locked 2021-10-25T11:42:29.594 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=11fca123d6a74c27b0259a9d543ec971 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi026.front.sepia.ceph.com is not locked 2021-10-25T11:42:29.603 INFO:teuthology.nuke:Checking targets against current locks 2021-10-25T11:42:29.744 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run.py", line 414, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run.py", line 271, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable