2021-10-28T18:25:16.647 INFO:root:teuthology version: 1.1.0-c56135d1 2021-10-28T18:25:16.649 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amaredia-2021-10-28_18:15:55-rgw-master-distro-basic-smithi/6465255 --owner scheduled_amaredia@teuthology --name amaredia-2021-10-28_18:15:55-rgw-master-distro-basic-smithi --description rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} --verbose -- /home/teuthworker/archive/amaredia-2021-10-28_18:15:55-rgw-master-distro-basic-smithi/6465255/orig.config.yaml 2021-10-28T18:25:16.690 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-10-28T18:25:16.754 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amaredia-2021-10-28_18:15:55-rgw-master-distro-basic-smithi/6465255 branch: master description: rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} email: null first_in_suite: false job_id: '6465255' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amaredia-2021-10-28_18:15:55-rgw-master-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '8.3' overrides: admin_socket: branch: master ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 lockdep: true osd: debug ms: 1 debug osd: 20 lockdep: true osd objectstore: filestore osd shutdown pgref assert: true osd sloppy crc: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 558a2ec6189261b2a2657471b4272993dbbfe5e0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: 558a2ec6189261b2a2657471b4272993dbbfe5e0 rgw: compression type: random ec-data-pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: testprofile frontend: beast storage classes: LUKEWARM, FROZEN s3tests: slow_backend: true selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-marcus-teuthvault-2 sha1: 454cc8a18c4c3851de5976d3e36e42644dbb1a70 owner: scheduled_amaredia@teuthology priority: 100 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 558a2ec6189261b2a2657471b4272993dbbfe5e0 sleep_before_teardown: 0 suite: rgw suite_branch: wip-marcus-teuthvault-2 suite_path: /home/teuthworker/src/github.com_alimaredia_ceph_454cc8a18c4c3851de5976d3e36e42644dbb1a70/qa suite_relpath: qa suite_repo: https://github.com/alimaredia/ceph.git suite_sha1: 454cc8a18c4c3851de5976d3e36e42644dbb1a70 targets: smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDh/nV7ieu6FuLzk0t7/MQq22ii0yiO53vTl67qWRcneXtWxAG7YhxAQz8UutKIRWECruqTL3WicEgrU2RwthwITUFsXreKMI2bVyPhLTVQxit5dgzRtzAPcjSTv+hX00LVBqjwEuSlZ1HREdpPwYFaWwrL1vSUjoAChRm57ZnPebdgLrHuY1ewF6N7Q83A1OFDU3SEmGRpAwdu0F0+CJ/kWY+cvC2X8lU2l9G2WHE3kPYzgPGQLz2DgXSsBoN8MOC/Dl9lDgPGASQHj16NzScWsAhULa4dARLxkmVpyN9+HAZPhhZ/M1IXCrqb3GOBaKv+RjIHNK0Mg0YAqYsnlW3EYuKoBlocivnCnsfI5qecJBN5Q9r9+4Gxob4iCWzzfIMCfqs7u+1Pk6L2iHK+hAph91j7Wp/zFT5DhbKtX8k47mTqDc7F+WF8Q6qvZT482fBxMzTvr5xqpCRBWpCuNyjX8/jAMM8QVRUzrdReE9h+5Ml+hbUqje7sRZk378kgmDU= smithi179.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdG+I43ZF3hLaTWwzFuMp4p4PNF3K1FoOP6WgZjz3ECxIkpNeJcg9Yrw+fnIry6Z4P3DhZD8rBOB8wzFsACQAE81FHQKtifYvmjhzj0IDIcaiKdUVUm939rOFbVrAy8ovkcmFPKPih5h1f15YbY3NPzhet6q61fxiLNoQKiBZ7QsHp2kRvRQD2Eezd9bp+AXrr6+2STDgzieWwX17RVid7fqtNtT1K8aO2Ov9LSGuC3DQSr19zBhnu3sSS8wJni3d/L3vU6xoG99TmYhlF0ySWCO3gCpnv7TiP5x/lmXjQPhiaBLjQP5h3QTQVyT1MmnK2fzk4+h7UZ5rivqnT2oiNMeANg+otLIJ0NihCfgg3osGvn8xUxx3u+Jlwih4kh4PbOcK7Tc4ujq9lMor3BSHS7OPQkwDsF1NVVnTDaVWXDJUWmlBKdoiV+O0XmJ0o7ho7BUBFJlvBv8Q/H+yUq7lyPPwv+o8dKzCnaS6rmUgxGsS/B/sNPjzXpbs+Vge/tXU= tasks: - install: extra_packages: deb: - s3cmd rpm: - s3cmd - ceph: null - openssl_keys: null - rgw: client.0: null - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_rgw_throttle.sh - rgw/test_librgw_file.sh - ragweed: client.0: default-branch: ceph-master rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: master force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: ceph-master rgw_server: client.0 teuthology_branch: master teuthology_sha1: c56135d151713269e811ede3163c9743c2e269de timestamp: 2021-10-28_18:15:55 tube: smithi user: amaredia verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.6888 2021-10-28T18:25:16.755 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_alimaredia_ceph_454cc8a18c4c3851de5976d3e36e42644dbb1a70/qa; will attempt to use it 2021-10-28T18:25:16.755 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_alimaredia_ceph_454cc8a18c4c3851de5976d3e36e42644dbb1a70/qa/tasks 2021-10-28T18:25:16.756 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-10-28T18:25:16.757 INFO:teuthology.task.internal:Checking packages... 2021-10-28T18:25:16.779 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '558a2ec6189261b2a2657471b4272993dbbfe5e0' 2021-10-28T18:25:16.779 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-10-28T18:25:16.779 INFO:teuthology.packaging:ref: None 2021-10-28T18:25:16.780 INFO:teuthology.packaging:tag: None 2021-10-28T18:25:16.780 INFO:teuthology.packaging:branch: master 2021-10-28T18:25:16.780 INFO:teuthology.packaging:sha1: 558a2ec6189261b2a2657471b4272993dbbfe5e0 2021-10-28T18:25:16.781 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=master 2021-10-28T18:25:17.102 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-8577.g558a2ec6 2021-10-28T18:25:17.102 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-10-28T18:25:17.113 INFO:teuthology.task.internal:no buildpackages task found 2021-10-28T18:25:17.113 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-10-28T18:25:17.142 INFO:teuthology.task.internal:Saving configuration 2021-10-28T18:25:17.157 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-10-28T18:25:17.170 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-10-28T18:25:17.192 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amaredia-2021-10-28_18:15:55-rgw-master-distro-basic-smithi/6465255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-10-28 18:17:44.687626', 'locked_by': 'scheduled_amaredia@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIlkgdwPtVdwoRRdhIWbX7jBWj19IqKphq/hx0HyTIjd2YJ9L7FEo6nvDdq+iGYlMFasiT3hSY1taEkClBHhICY='} 2021-10-28T18:25:17.214 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': None, 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': False, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMPucqIAmc+KblTScpnpwH77zRsDxJi5zDXz7sSYL9HA'} 2021-10-28T18:25:17.215 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi179.front.sepia.ceph.com is not locked 2021-10-28T18:25:17.302 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0731940120654dfd96de9deb3346df28 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine smithi179.front.sepia.ceph.com is not locked 2021-10-28T18:25:17.313 INFO:teuthology.nuke:Checking targets against current locks 2021-10-28T18:25:17.433 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run.py", line 414, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/run.py", line 271, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable