2020-08-25T02:30:56.379 INFO:root:teuthology version: 1.0.0-1a72666 2020-08-25T02:30:56.381 INFO:teuthology.results:seed: '9542' 2020-08-25T02:49:15.519 INFO:root:teuthology version: 1.0.0-1a72666 2020-08-25T02:49:15.546 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-08-25T14:49:33.586 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2020-08-25T14:49:44.915 INFO:scrape:Found 16 jobs 2020-08-25T14:49:47.046 INFO:scrape:Found 2 distinct failure reasons 2020-08-25T14:49:47.048 INFO:scrape:Dead: b'2020-08-25T14:47:30.558 DEBUG:teuthology.exit:Finished running handlers' 2020-08-25T14:49:47.048 INFO:scrape:1 jobs: ['5374271'] 2020-08-25T14:49:47.049 INFO:scrape:suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack'] 2020-08-25T14:49:47.049 INFO:scrape: 2020-08-25T14:49:47.049 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=85434552de3c4223cb4a22d42c5d77ef734781e3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/rados/test.sh' 2020-08-25T14:49:47.050 INFO:scrape:1 jobs: ['5374267'] 2020-08-25T14:49:47.050 INFO:scrape:suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-octopus', '5-msgr2', '6-final-workload/{blogbench', 'ec-rados-default', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_latest}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack'] 2020-08-25T14:49:47.051 INFO:scrape: 2020-08-25T14:49:47.054 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/ failed: 1 dead: 0 running: 1 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [5374267] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 02:33:49 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374267/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374267/ sentry: https://sentry.ceph.com/sepia/teuthology/?q=e57bdcb9bbfc48288bb8d42a84eca273 Command failed (workunit test rados/test.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=85434552de3c4223cb4a22d42c5d77ef734781e3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/rados/test.sh' Running ================================================================= [5374271] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374271/ Pass ================================================================= [5374246] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 03:02:59 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374246/ [5374247] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 05:21:46 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374247/ [5374249] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 00:52:57 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374249/ [5374251] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:51:36 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374251/ [5374253] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 03:03:25 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374253/ [5374255] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:21:13 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374255/ [5374257] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 00:52:10 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374257/ [5374259] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:57:06 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374259/ [5374261] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 03:00:41 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374261/ [5374263] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 05:29:42 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374263/ [5374265] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 00:50:17 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374265/ [5374269] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 03:00:45 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374269/ [5374273] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 00:50:01 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374273/ [5374275] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:54:45 info: http://pulpito.ceph.com/teuthology-2020-08-25_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5374275/