2022-05-24T05:03:44.793 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-24T05:03:44.796 INFO:teuthology.results:seed: '8771' 2022-05-24T05:32:16.581 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-24T05:32:16.595 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-24T06:48:23.063 INFO:teuthology.results:Tests finished! gathering results... 2022-05-24T06:48:25.204 INFO:scrape:Found 24 jobs 2022-05-24T06:48:26.843 INFO:scrape:Found 2 distinct failure reasons 2022-05-24T06:48:26.844 INFO:scrape:Failure: "2022-05-24T05:40:36.636645+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log 2022-05-24T06:48:26.844 INFO:scrape:7 jobs: ['6847002', '6846997', '6846996', '6846990', '6846989', '6847004', '6846994'] 2022-05-24T06:48:26.845 INFO:scrape:suites intersection: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{0-install'] 2022-05-24T06:48:26.845 INFO:scrape:suites union: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{0-install', 'test/cfuse_workunit_suites_blogbench}}', 'test/cfuse_workunit_suites_fsstress}}', 'test/kclient_workunit_suites_dbench}}', 'test/kclient_workunit_suites_pjd}}', 'test/libcephfs_interface_tests}}', 'test/rados_cls_all}}', 'test/rados_python}}'] 2022-05-24T06:48:26.846 INFO:scrape: 2022-05-24T06:48:26.846 INFO:scrape:Failure: {'smithi183.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['sudo', 'apt-get', 'clean'], 'delta': '0:00:00.025937', 'end': '2022-05-24 05:43:38.446173', 'invocation': {'module_args': {'_raw_params': 'sudo apt-get clean', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 100, 'start': '2022-05-24 05:43:38.420236', 'stderr': 'E: Could not get lock /var/lib/apt/lists/lock. It is held by process 23142 (python)\nE: Unable to lock directory /var/lib/apt/lists/', 'stderr_lines': ['E: Could not get lock /var/lib/apt/lists/lock. It is held by process 23142 (python)', 'E: Unable to lock directory /var/lib/apt/lists/'], 'stdout': '', 'stdout_lines': [], 'warnings': ["Consider using 'become', 'become_method', and 'become_user' rather than running sudo"]}, 'smithi053.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': "failed to transfer file to /home/teuthworker/.ansible/tmp/ansible-local-11021zb5txer2/tmp67j0no6u /home/ubuntu/.ansible/tmp/ansible-tmp-1653371151.602766-20045-212671187511260/AnsiballZ_lvg.py:\n\ndd: failed to open '/home/ubuntu/.ansible/tmp/ansible-tmp-1653371151.602766-20045-212671187511260/AnsiballZ_lvg.py': No such file or directory\n"}} 2022-05-24T06:48:26.847 INFO:scrape:1 jobs: ['6847012'] 2022-05-24T06:48:26.848 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{rhel_8}', 'tasks/{0-install', 'test/rgw_s3tests}}'] 2022-05-24T06:48:26.848 INFO:scrape: 2022-05-24T06:48:26.850 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 16 Fail ================================================================= [6846989] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/cfuse_workunit_suites_blogbench}} ----------------------------------------------------------------- time: 00:16:58 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846989/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846989/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4aa58544a0274887a1cfc86a3e226f94 "2022-05-24T05:20:46.381859+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log [6846990] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:17:10 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846990/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d61e975402d4ae08204dc51874edebb "2022-05-24T05:21:52.884311+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log [6846994] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_suites_dbench}} ----------------------------------------------------------------- time: 00:16:55 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846994/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846994/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6871b7755bd845d691c689772d63c1c8 "2022-05-24T05:28:58.306575+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log [6846996] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:16:37 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846996/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846996/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06dbf64db38648d3b78c3d7e68bb426f "2022-05-24T05:31:02.045867+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log [6846997] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/libcephfs_interface_tests}} ----------------------------------------------------------------- time: 00:17:07 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846997/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846997/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b02fc9dbde4468b9d9b77383641b77b "2022-05-24T05:32:08.678146+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log [6847002] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_cls_all}} ----------------------------------------------------------------- time: 00:16:30 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847002/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0cfde2bffec8421c8679d7a4e82eb4ec "2022-05-24T05:40:36.636645+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log [6847004] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_python}} ----------------------------------------------------------------- time: 00:16:43 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847004/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847004/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b25a2ed85cce43d5b8949b2788cb39a1 "2022-05-24T05:42:13.138268+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log [6847012] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rgw_s3tests}} ----------------------------------------------------------------- time: 00:11:51 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847012/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847012/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8a6c7cab718d44d5bca2af820efeacae {'smithi183.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['sudo', 'apt-get', 'clean'], 'delta': '0:00:00.025937', 'end': '2022-05-24 05:43:38.446173', 'invocation': {'module_args': {'_raw_params': 'sudo apt-get clean', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 100, 'start': '2022-05-24 05:43:38.420236', 'stderr': 'E: Could not get lock /var/lib/apt/lists/lock. It is held by process 23142 (python)\nE: Unable to lock directory /var/lib/apt/lists/', 'stderr_lines': ['E: Could not get lock /var/lib/apt/lists/lock. It is held by process 23142 (python)', 'E: Unable to lock directory /var/lib/apt/lists/'], 'stdout': '', 'stdout_lines': [], 'warnings': ["Consider using 'become', 'become_method', and 'become_user' rather than running sudo"]}, 'smithi053.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': "failed to transfer file to /home/teuthworker/.ansible/tmp/ansible-local-11021zb5txer2/tmp67j0no6u /home/ubuntu/.ansible/tmp/ansible- tmp-1653371151.602766-20045-212671187511260/AnsiballZ_lvg.py:\n\ndd: failed to open '/home/ubuntu/.ansible/tmp/ansible- tmp-1653371151.602766-20045-212671187511260/AnsiballZ_lvg.py': No such file or directory\n"}} Pass ================================================================= [6846991] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_iozone}} time: 00:19:29 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846991/ [6846992] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/cfuse_workunit_suites_pjd}} time: 00:15:43 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846992/ [6846993] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/kclient_workunit_direct_io}} time: 00:14:10 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846993/ [6846995] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/kclient_workunit_suites_fsstress}} time: 00:14:51 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846995/ [6846998] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/mon_thrash}} time: 00:35:29 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846998/ [6846999] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rados_api_tests}} time: 00:30:46 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6846999/ [6847000] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_bench}} time: 01:18:06 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847000/ [6847001] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rados_cache_snaps}} time: 00:24:28 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847001/ [6847003] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rados_ec_snaps}} time: 00:25:49 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847003/ [6847005] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_workunit_loadgen_mix}} time: 00:27:49 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847005/ [6847006] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_api_tests}} time: 00:29:46 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847006/ [6847007] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_cli_import_export}} time: 00:14:30 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847007/ [6847008] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_fsx}} time: 00:15:35 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847008/ [6847009] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_python_api_tests}} time: 00:27:34 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847009/ [6847010] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_workunit_suites_iozone}} time: 00:21:53 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847010/ [6847011] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rgw_ec_s3tests}} time: 00:23:30 info: http://pulpito.ceph.com/teuthology-2022-05-24_05:00:04-smoke-master-distro-default-smithi/6847011/