2021-10-27T12:25:06.797 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-27T12:25:06.799 DEBUG:teuthology.run:Teuthology command: teuthology --description powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_misc thrashosds-health whitelist_health} --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-10-27_12:17:02-powercycle-pacific-distro-default-smithi/6463212 --name teuthology-2021-10-27_12:17:02-powercycle-pacific-distro-default-smithi --verbose -- /home/teuthworker/archive/teuthology-2021-10-27_12:17:02-powercycle-pacific-distro-default-smithi/6463212/orig.config.yaml 2021-10-27T12:25:06.836 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-10-27T12:25:06.899 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2021-10-27_12:17:02-powercycle-pacific-distro-default-smithi/6463212 branch: pacific description: powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_misc thrashosds-health whitelist_health} email: ceph-qa@ceph.io first_in_suite: false job_id: '6463212' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2021-10-27_12:17:02-powercycle-pacific-distro-default-smithi nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - \(MDS_TRIM\) - \(MDS_SLOW_REQUEST\) - MDS_SLOW_METADATA_IO - Behind on trimming log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9c8bdbc8afe90dacd18f2ee15044f48109538abe ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 9c8bdbc8afe90dacd18f2ee15044f48109538abe selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific sha1: 265c52c13f2c6681cc615210dceef36c89412246 owner: scheduled_teuthology@teuthology priority: 500 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - mgr.y - mds.a - client.0 - - osd.0 - - osd.1 - - osd.2 sha1: 9c8bdbc8afe90dacd18f2ee15044f48109538abe sleep_before_teardown: 0 suite: powercycle suite_branch: pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph_265c52c13f2c6681cc615210dceef36c89412246/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 265c52c13f2c6681cc615210dceef36c89412246 targets: smithi017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0KVP+Rr9szb/jk+LXz9Gng2STAQYeqwmxZyCTha2eDTo8lXQkZ9ZNF82wdo1ErkeHxvAhLCZfN3RxrAkjRS3oznxpSSvAMU3Fh2W8oXmK3ceGsncFpLxxMHU8Sgo1986pTXt+4fmPdkW7ud00DKJjrkPvxPhmR0CMsIS2FtRypEzh7cprmuQNRa05SdqHPRJVI6AzO3rhLDkHjHmRgaW/vHBGD6RC90FGNeiGZPnG7YIYMM2cQijWioK29qrVnk7TxmBluhlAl9ktL0WyoXRMPECaaCVQKkLf7hIDIUhlRehfb5rO1Hq9MIZ7Rs3XwnVNUNkdoa5tD0twKiWm8PWc66YTmBi6Q59sHgKvAW2c41A/XXqDo1DC5aY2s59SvdtYib8rSqRoOKtE+5eBcd+PrWuptE19laMPnP8T1M4s4hK4JskDYkpBGz7bUaDMXfE0HPNcLRwDeydIRrCnBV8LYSYBc0ehgyLhgb4OnAHxm/YnSjlG+CLugJ9th9x/z20= smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgLLQNpq9Hhb1c3nsTUV1kb05lxHTNHVk4HVSUaTFP7Rrs0DVPgKU/AZsdEdWZMIGGFVx+lGBlUFO9x+jKikBnQQZev5fEm65x/rNH82WcKMX/vBa7rj7U12BPmPCoco+LiCllS3woghrhB3V56tdTC1RfnFsm/hVjoicR1LI4t7l0KomX3KCQYiXvwgdQ9bfgd86Hi6lRkC8IP/sAFXGdKTxISbfRWTTuUqxFoutdxqXTDDWtzwBfCSl2Sg0iBpEy7yjHYR/zoEjHJGwtj9wuBIQmxAGN0gSqTx04ruBgMm2emzTUqi3nsmzIRu3BwAX5+unXY6TfiLodumg4wcrke+Jo3LVlIChnB6K3FCe4miHa2auXcgxZS7n2FD1Y6RyRSfKxW3vzaDxy2Ag504CTiud3UG4BkyisBGpsELYHsgxrOSSjpiOTXnKsYSpHLYpnVQx4q2xCsRvWjwYaRABTfw9NYvdaXC5rM8mme5nstIESPZjH2d/b6lr9IkKZ+0k= smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2qap5sq96NodTHUrZPV4b8CEp0V2b9unGSrUztagIbTsUOT8HH1b8GE9IfQaBuIRSVHKZlT60mzdq0M7BDTgbGWEe7+J5Tuy25Rzfz+cVkjFIWpVpz33QiEptO0svKtSS7r8ow9z9DRjpAJwZaant0Uh2E/eAjXM+JE1G/jZYgEBkjB5kkwRdgl7reZ4QaCl8Z22gCMPr8Poh6ZGp5oC/nLjpNH+RzuLnK2ByQvoQbfILQF/qRUSN/V5Xys3AUG/2wflcoLUKxOfjGVd9Q9oUr1WcUfWy4QeuIO9gRd5iFEmaqQSlozXlKYMgfJ8Yp8EwZ57g8GVeRN/DX1T5TKI9UuMjT0P+Oa6Gxbtl5CKJBFHMaJSBYoE0/Nt+USoNm/w1Rqkb5ith/Whk904TSt4HqWAckKbQuk5xxRT9MRQaF1JtOjqVVP/ao6iZkWEdiINtXUdCWU+ZZuqRV4Hf0PRpL8rhzElc7jc3KQJoi76GwRP/bitw5P5AEqcHjQSB8bM= smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8Y3+84pSQN7mz7NWkxJAQkxrPw2uEtSIhp7jjyu8ZgiPyKKog8vWOzXz2u9hMFM7YduSJvsb0n6LrV+5QypcauJwrX2CEnBuyujQ1V/8UUu3TCzy05OXOHasOBS2g2zRvLQidZJhowGcgOWF/zqXMkT4ILf0nU2088p1jN8cWECBW7yrVh93Bsy3yipooNIeIg5a7+o7DrzAlpJ7uPdpaQl9P+JuLgrTcoESxln0QujYEsyRAJSrsvyUbIXgrp1rNQn2KcXduV+uYXW+OuJ5WDXz8hVsqlFuENCKvO9SopolGwkbD9TgxQv4Xbco60P7DW0RxHC7ofbjKGttYGqxwIN2gt5y4siCbcwfzRPPHcJagiKCEnTlN2PYq7oN5slpXpksuJMFJr+EEiSCqf/no8ZBhVj5q4fWYtJ6+wvVpTIBBJusH3hPN0Tk4Ht8X64h9Vz4Sk2pwY64LZWbqT+dpzDrcCj5vUnoWsp+fTx6E5AE6kf8n9n3AKxjx7gQYG4M= tasks: - install: extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - thrashosds: chance_down: 1.0 powercycle: true timeout: 600 - ceph-fuse: null - workunit: clients: all: - fs/misc timeout: 6h teuthology_branch: master teuthology_sha1: c56135d151713269e811ede3163c9743c2e269de timestamp: 2021-10-27_12:17:02 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.6888 2021-10-27T12:25:06.899 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_265c52c13f2c6681cc615210dceef36c89412246/qa; will attempt to use it 2021-10-27T12:25:06.900 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_265c52c13f2c6681cc615210dceef36c89412246/qa/tasks 2021-10-27T12:25:06.900 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-10-27T12:25:06.902 INFO:teuthology.task.internal:Checking packages... 2021-10-27T12:25:06.922 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '9c8bdbc8afe90dacd18f2ee15044f48109538abe' 2021-10-27T12:25:06.922 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-10-27T12:25:06.923 INFO:teuthology.packaging:ref: None 2021-10-27T12:25:06.923 INFO:teuthology.packaging:tag: None 2021-10-27T12:25:06.923 INFO:teuthology.packaging:branch: pacific 2021-10-27T12:25:06.924 INFO:teuthology.packaging:sha1: 9c8bdbc8afe90dacd18f2ee15044f48109538abe 2021-10-27T12:25:06.924 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2021-10-27T12:25:07.099 INFO:teuthology.task.internal:Found packages for ceph version 16.2.6-242.g9c8bdbc8 2021-10-27T12:25:07.100 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-10-27T12:25:07.108 INFO:teuthology.task.internal:no buildpackages task found 2021-10-27T12:25:07.108 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-10-27T12:25:07.139 INFO:teuthology.task.internal:Saving configuration 2021-10-27T12:25:07.153 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-10-27T12:25:07.165 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-27T12:25:07.188 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': None, 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIbDdQ1HYKff9GxcYHHX1xtuJJrvQIabxDh/Jw6rBLMn'} 2021-10-27T12:25:07.189 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi017.front.sepia.ceph.com is not locked 2021-10-27T12:25:07.266 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=85a81ac999d54e4cb377c502f3391a61 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi017.front.sepia.ceph.com is not locked 2021-10-27T12:25:07.277 INFO:teuthology.nuke:Checking targets against current locks 2021-10-27T12:25:07.418 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run.py", line 414, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run.py", line 271, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable