2020-05-08T18:40:31.622 INFO:root:teuthology version: 1.0.0-28c1ab59 2020-05-08T18:40:31.624 INFO:teuthology.results:subset: '1/20' 2020-05-08T18:40:31.625 INFO:teuthology.results:seed: '6363' 2020-05-08T19:17:10.195 INFO:root:teuthology version: 1.0.0-28c1ab59 2020-05-08T19:17:10.224 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-05-09T07:11:24.024 INFO:teuthology.results:Tests finished! gathering results... 2020-05-09T07:11:24.122 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/ failed: 14 dead: 0 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [5035119] rgw/crypt/{0-cluster/fixed-1.yaml 1-ceph-install/install.yaml 2-kms/barbican.yaml 3-rgw/rgw.yaml 4-tests/{s3tests.yaml}} ----------------------------------------------------------------- time: 00:20:03 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035119/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035119/ "2020-05-08T19:13:55.124290+0000 mon.a (mon.0) 164 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [5035122] rgw/multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} ----------------------------------------------------------------- time: 12:05:21 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035122/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035122/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=59b88c450e94469aafcf8ebd6042a249 rgw multisite, rgw multisite pubsub, test failures [5035124] rgw/tempest/{clusters/fixed-1.yaml frontend/beast.yaml overrides.yaml tasks/rgw_tempest.yaml ubuntu_latest.yaml} ----------------------------------------------------------------- time: 00:19:44 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035124/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035124/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=6fac0290e57e4bdf88bf347e39a4ec9d Command failed on smithi095 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [5035126] rgw/tools/{centos_latest.yaml cluster.yaml tasks.yaml} ----------------------------------------------------------------- time: 00:14:18 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035126/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035126/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=8d88f79a3b3a424c948acec0e3fba144 Command failed (workunit test rgw/test_rgw_orphan_list.sh) on smithi150 with status 12: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ab02ee2df9cae9305f0178ec2012b676970256ca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rgw/test_rgw_orphan_list.sh' [5035127] rgw/website/{clusters/fixed-2.yaml frontend/beast.yaml http.yaml overrides.yaml tasks/s3tests-website.yaml ubuntu_latest.yaml} ----------------------------------------------------------------- time: 00:12:31 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035127/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035127/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=42976c90e3c041668b6e712c2df282ed Command failed on smithi173 with status 2: 'cd /home/ubuntu/cephtest/s3-tests && ./bootstrap' [5035129] rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml objectstore/filestore-xfs.yaml overrides.yaml rgw_pool_type/ec.yaml tasks/rgw_ragweed.yaml} ----------------------------------------------------------------- time: 00:08:52 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035129/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035129/ "2020-05-08T19:14:36.634560+0000 mon.a (mon.0) 179 : cluster [WRN] Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY)" in cluster log [5035130] rgw/singleton/{all/radosgw-admin.yaml frontend/civetweb.yaml objectstore/filestore-xfs.yaml overrides.yaml rgw_pool_type/ec.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:12:07 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035130/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035130/ "2020-05-08T19:23:18.640833+0000 mon.a (mon.0) 160 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY)" in cluster log [5035132] rgw/multisite/{clusters.yaml frontend/civetweb.yaml omap_limits.yaml overrides.yaml realms/two-zonegroup.yaml tasks/test_multi.yaml valgrind.yaml} ----------------------------------------------------------------- time: 00:27:07 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035132/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035132/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=9e84aa7e3e4c4f0bb5512484f6b92d91 rgw multisite, test failures [5035140] rgw/multisite/{clusters.yaml frontend/civetweb.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} ----------------------------------------------------------------- time: 11:47:08 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035140/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035140/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=6ef78d0ce5484940b03691aa8252f61a rgw multisite pubsub, test failures [5035142] rgw/tempest/{clusters/fixed-1.yaml frontend/civetweb.yaml overrides.yaml tasks/rgw_tempest.yaml ubuntu_latest.yaml} ----------------------------------------------------------------- time: 00:18:27 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035142/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035142/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e5eb058d6a894fa6b45d262d90edf14b Command failed on smithi150 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [5035144] rgw/website/{clusters/fixed-2.yaml frontend/civetweb.yaml http.yaml overrides.yaml tasks/s3tests-website.yaml ubuntu_latest.yaml} ----------------------------------------------------------------- time: 00:13:17 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035144/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035144/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=d59d9a5651d34a0090fdb590802315d6 Command failed on smithi077 with status 2: 'cd /home/ubuntu/cephtest/s3-tests && ./bootstrap' [5035146] rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml objectstore/bluestore-bitmap.yaml overrides.yaml rgw_pool_type/ec.yaml tasks/rgw_bucket_quota.yaml} ----------------------------------------------------------------- time: 00:12:35 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035146/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035146/ "2020-05-08T19:29:46.866737+0000 mon.b (mon.0) 183 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log [5035151] rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml objectstore/filestore-xfs.yaml overrides.yaml rgw_pool_type/replicated.yaml tasks/rgw_multipart_upload.yaml} ----------------------------------------------------------------- time: 00:11:34 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035151/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035151/ "2020-05-08T19:31:29.177824+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY)" in cluster log [5035152] rgw/singleton/{all/radosgw-admin.yaml frontend/civetweb.yaml objectstore/filestore-xfs.yaml overrides.yaml rgw_pool_type/replicated.yaml supported-random-distro$/{ubuntu_latest.yaml}} ----------------------------------------------------------------- time: 00:12:33 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035152/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035152/ "2020-05-08T19:34:38.521768+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY)" in cluster log Pass ================================================================= [5035118] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/bluestore-bitmap.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_user_quota.yaml} time: 00:13:06 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035118/ [5035120] rgw/hadoop-s3a/{clusters/fixed-2.yaml hadoop/default.yaml overrides.yaml s3a-hadoop.yaml} time: 00:17:02 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035120/ [5035121] rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml objectstore/bluestore-bitmap.yaml overrides.yaml rgw_pool_type/ec-profile.yaml tasks/rgw_multipart_upload.yaml} time: 00:09:43 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035121/ [5035123] rgw/singleton/{all/radosgw-admin.yaml frontend/civetweb.yaml objectstore/bluestore-bitmap.yaml overrides.yaml rgw_pool_type/ec-profile.yaml supported-random-distro$/{centos_8.yaml}} time: 00:12:00 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035123/ [5035125] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/bluestore-bitmap.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_bucket_quota.yaml} time: 00:13:28 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035125/ [5035128] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/filestore-xfs.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_multipart_upload.yaml} time: 00:13:46 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035128/ [5035131] rgw/crypt/{0-cluster/fixed-1.yaml 1-ceph-install/install.yaml 2-kms/testing.yaml 3-rgw/rgw.yaml 4-tests/{s3tests.yaml}} time: 00:15:14 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035131/ [5035133] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/bluestore-bitmap.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_s3tests.yaml} time: 00:21:30 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035133/ [5035134] rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml objectstore/bluestore-bitmap.yaml overrides.yaml rgw_pool_type/replicated.yaml tasks/rgw_s3tests.yaml} time: 00:14:40 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035134/ [5035135] rgw/singleton/{all/radosgw-admin.yaml frontend/civetweb.yaml objectstore/bluestore-bitmap.yaml overrides.yaml rgw_pool_type/replicated.yaml supported-random-distro$/{centos_8.yaml}} time: 00:10:57 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035135/ [5035136] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/filestore-xfs.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_user_quota.yaml} time: 00:14:49 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035136/ [5035137] rgw/crypt/{0-cluster/fixed-1.yaml 1-ceph-install/install.yaml 2-kms/vault_kv.yaml 3-rgw/rgw.yaml 4-tests/{s3tests.yaml}} time: 00:14:40 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035137/ [5035138] rgw/hadoop-s3a/{clusters/fixed-2.yaml hadoop/v32.yaml overrides.yaml s3a-hadoop.yaml} time: 00:14:34 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035138/ [5035139] rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml objectstore/filestore-xfs.yaml overrides.yaml rgw_pool_type/ec-profile.yaml tasks/rgw_user_quota.yaml} time: 00:11:13 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035139/ [5035141] rgw/singleton/{all/radosgw-admin.yaml frontend/civetweb.yaml objectstore/filestore-xfs.yaml overrides.yaml rgw_pool_type/ec-profile.yaml supported-random-distro$/{centos_8.yaml}} time: 00:10:55 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035141/ [5035143] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/filestore-xfs.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_bucket_quota.yaml} time: 00:15:27 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035143/ [5035145] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/bluestore-bitmap.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_multipart_upload.yaml} time: 00:14:41 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035145/ [5035147] rgw/singleton/{all/radosgw-admin.yaml frontend/civetweb.yaml objectstore/bluestore-bitmap.yaml overrides.yaml rgw_pool_type/ec.yaml supported-random-distro$/{ubuntu_latest.yaml}} time: 00:13:25 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035147/ [5035148] rgw/crypt/{0-cluster/fixed-1.yaml 1-ceph-install/install.yaml 2-kms/vault_transit.yaml 3-rgw/rgw.yaml 4-tests/{s3tests.yaml}} time: 00:13:33 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035148/ [5035149] rgw/multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/two-zonegroup.yaml tasks/test_multi.yaml valgrind.yaml} time: 00:29:16 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035149/ [5035150] rgw/thrash/{civetweb.yaml clusters/fixed-2.yaml install.yaml objectstore/filestore-xfs.yaml thrasher/default.yaml thrashosds-health.yaml workload/rgw_s3tests.yaml} time: 00:17:57 info: http://pulpito.ceph.com/ivancich-2020-05-08_18:25:10-rgw-wip-eric-testing-distro-basic-smithi/5035150/ 2020-05-09T07:11:24.265 INFO:teuthology.results:starting coverage generation 2020-05-09T07:11:24.270 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results generate_coverage(archive_dir, name) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage archive_dir, File "/usr/lib/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'