2020-04-29T16:01:28.749 INFO:root:teuthology version: 1.0.0-6cf40593 2020-04-29T16:01:28.751 INFO:teuthology.results:seed: '9970' 2020-04-29T16:05:41.888 INFO:root:teuthology version: 1.0.0-6cf40593 2020-04-29T16:05:41.898 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-04-29T16:39:42.514 INFO:teuthology.results:Tests finished! gathering results... 2020-04-29T16:39:42.549 INFO:teuthology.results:Sending results to sanupkri@redhat.com: Test Run: sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/ failed: 4 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [4997936] multimds/basic/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} ----------------------------------------------------------------- time: 00:16:52 info: http://pulpito.ceph.com/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997936/ log: http://qa-proxy.ceph.com/teuthology/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997936/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=f812fe70557b4d98b918d4516838eec1 Test failure: test_export_pin_getfattr (tasks.cephfs.test_exports.TestExports) [4997937] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} ----------------------------------------------------------------- time: 00:16:00 info: http://pulpito.ceph.com/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997937/ log: http://qa-proxy.ceph.com/teuthology/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997937/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=182d69497e1b442e93984d9f089e12ff Test failure: test_export_pin_getfattr (tasks.cephfs.test_exports.TestExports) [4997938] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} ----------------------------------------------------------------- time: 00:11:08 info: http://pulpito.ceph.com/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997938/ log: http://qa-proxy.ceph.com/teuthology/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997938/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=67d921bc8b024dcca33f8f97dc416961 Command failed on smithi202 with status 4: "sudo bash -c 'iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT'" [4997940] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} ----------------------------------------------------------------- time: 00:12:51 info: http://pulpito.ceph.com/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997940/ log: http://qa-proxy.ceph.com/teuthology/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997940/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e53ec66f7da34389b3c01332f39b2fcd Test failure: test_export_pin_getfattr (tasks.cephfs.test_exports.TestExports) Dead ================================================================= [4997939] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} ----------------------------------------------------------------- time: 00:01:45 info: http://pulpito.ceph.com/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997939/ log: http://qa-proxy.ceph.com/teuthology/sidharthanup-2020-04-29_15:59:48-multimds-master-distro-default-smithi/4997939/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=9d46461af6f84f4db8d212d62fd00bb6 File "/usr/lib64/python3.6/site- packages/subscription_manager/scripts/subscription_manager.py", line 85, in main\n return managercli.ManagerCLI().main()\n File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2982, in main\n ret = CLI.main(self)\n File "/usr/lib64/python3.6/site-packages/subscription_manager/cli.py", line 183, in main\n return cmd.main()\n File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 536, in main\n return_code = self._do_command()\n File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 1680, in _do_command\n release=self.options.release\n File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 1121, in updateConsumer\n ret = self.conn.request_put(method, params)\n File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 735, in request_put\n return self._request("PUT", method, params, headers=headers)\n File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request\n info=info, headers=headers)\n File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 627, in _request\n self.validateResponse(result, request_type, handler)\n File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 689, in validateResponse\n raise RestlibException(response[\'status\'], error_msg, response.get(\'headers\'))\nrhsm.connection.RestlibException: HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.\nPlease inspect their state, fix their errors and resume them.\n\nRequired lock: update\nConflicts with tasks:\n- https://satellite.front.sepia.ceph.com/foreman_tasks/tasks/9039a5be- acb9-4563-9f6b-82b0ac52e7c4', 'rc': 1, 'start': '2020-04-29 16:13:05.081724', 'end': '2020-04-29 16:13:06.162333', 'delta': '0:00:01.080609', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'subscription-manager release --set=8.1', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Traceback (most recent call last):', ' File "/sbin/subscription- manager", line 11, in ', " load_entry_point('subscription- manager==1.25.17', 'console_scripts', 'subscription-manager')()", ' File "/usr/lib64/python3.6/site- packages/subscription_manager/scripts/subscription_manager.py", line 85, in main', ' return managercli.ManagerCLI().main()', ' File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2982, in main', ' ret = CLI.main(self)', ' File "/usr/lib64/python3.6/site-packages/subscription_manager/cli.py", line 183, in main', ' return cmd.main()', ' File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 536, in main', ' return_code = self._do_command()', ' File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 1680, in _do_command', ' release=self.options.release', ' File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 1121, in updateConsumer', ' ret = self.conn.request_put(method, params)', ' File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 735, in request_put', ' return self._request("PUT", method, params, headers=headers)', ' File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request', ' info=info, headers=headers)', ' File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 627, in _request', ' self.validateResponse(result, request_type, handler)', ' File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 689, in validateResponse', " raise RestlibException(response['status'], error_msg, response.get('headers'))", 'rhsm.connection.RestlibException: HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.', 'Please inspect their state, fix their errors and resume them.', '', 'Required lock: update', 'Conflicts with tasks:', '- https://satellite.front.sepia.ceph.com/foreman_tasks/tasks/9039a5be- acb9-4563-9f6b-82b0ac52e7c4'], '_ansible_no_log': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.co m_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworke r/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_ git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/ src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthwor ker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/ src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthwor ker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/ src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence('tag:yaml.org,2002:seq', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_t euthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.c eph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'subscription-manager')Failure object was: {'smithi065.front.sepia.ceph.com': {'msg': 'non-zero return code', 'cmd': ['subscription-manager', 'release', '--set=8.1'], 'stdout': '', 'stderr': 'Traceback (most recent call last):\n File "/sbin/subscription-manager", line 11, in \n load_entry_point(\'subscription- manager==1.25.17\', \'console_scripts\', \'subscription-manager\')()\n File "/usr/lib64/python3.6/site- packages/subscription_manager/scripts/subscription_manager.py", line 85, in main\n return managercli.ManagerCLI().main()\n File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2982, in main\n ret = CLI.main(self)\n File "/usr/lib64/python3.6/site-packages/subscription_manager/cli.py", line 183, in main\n return cmd.main()\n File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 536, in main\n return_code = self._do_command()\n File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 1680, in _do_command\n release=self.options.release\n File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 1121, in updateConsumer\n ret = self.conn.request_put(method, params)\n File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 735, in request_put\n return self._request("PUT", method, params, headers=headers)\n File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request\n info=info, headers=headers)\n File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 627, in _request\n self.validateResponse(result, request_type, handler)\n File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 689, in validateResponse\n raise RestlibException(response[\'status\'], error_msg, response.get(\'headers\'))\nrhsm.connection.RestlibException: HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.\nPlease inspect their state, fix their errors and resume them.\n\nRequired lock: update\nConflicts with tasks:\n- https://sat ellite.front.sepia.ceph.com/foreman_tasks/tasks/88c40284-e563-44e4-9068-da7 608f95477', 'rc': 1, 'start': '2020-04-29 16:13:06.047455', 'end': '2020-04-29 16:13:07.137908', 'delta': '0:00:01.090453', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'subscription-manager release --set=8.1', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Traceback (most recent call last):', ' File "/sbin/subscription-manager", line 11, in ', " load_entry_point('subscription-manager==1.25.17', 'console_scripts', 'subscription-manager')()", ' File "/usr/lib64/python3.6/site- packages/subscription_manager/scripts/subscription_manager.py", line 85, in main', ' return managercli.ManagerCLI().main()', ' File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2982, in main', ' ret = CLI.main(self)', ' File "/usr/lib64/python3.6/site-packages/subscription_manager/cli.py", line 183, in main', ' return cmd.main()', ' File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 536, in main', ' return_code = self._do_command()', ' File "/usr/lib64/python3.6/site- packages/subscription_manager/managercli.py", line 1680, in _do_command', ' release=self.options.release', ' File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 1121, in updateConsumer', ' ret = self.conn.request_put(method, params)', ' File "/usr/lib64/python3.6/site- packages/rhsm/connection.py", line 735, in request_put', ' return self._request("PUT", method, params, headers=headers)', ' File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 752, in _request', ' info=info, headers=headers)', ' File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 627, in _request', ' self.validateResponse(result, request_type, handler)', ' File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 689, in validateResponse', " raise RestlibException(response['status'], error_msg, response.get('headers'))", 'rhsm.connection.RestlibException: HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.', 'Please inspect their state, fix their errors and resume them.', '', 'Required lock: update', 'Conflicts with tasks:', '- htt ps://satellite.front.sepia.ceph.com/foreman_tasks/tasks/88c40284-e563-44e4- 9068-da7608f95477'], '_ansible_no_log': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.co m_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworke r/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_ git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/ src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthwor ker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/ src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthwor ker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/ src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence('tag:yaml.org,2002:seq', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_t euthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.c eph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'subscription-manager') 2020-04-29T16:39:42.628 INFO:teuthology.results:starting coverage generation 2020-04-29T16:39:42.634 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results generate_coverage(archive_dir, name) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage archive_dir, File "/usr/lib/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'