2021-11-28T05:13:40.516 INFO:root:teuthology version: 1.1.0-a0bc3188 2021-11-28T05:13:40.520 INFO:teuthology.results:seed: '8227' 2021-11-28T19:08:02.492 INFO:root:teuthology version: 1.1.0-a0bc3188 2021-11-28T19:08:02.567 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-11-28T19:09:02.663 INFO:teuthology.results:Tests finished! gathering results... 2021-11-28T19:09:04.557 INFO:scrape:Found 24 jobs 2021-11-28T19:09:05.352 INFO:scrape:Found 3 distinct failure reasons 2021-11-28T19:09:05.353 INFO:scrape:Failure: "2021-11-28T09:08:25.944109+0000 mon.a (mon.0) 154 : cluster [WRN] Health check failed: Degraded data redundancy: 26/17378 objects degraded (0.150%), 1 pg degraded (PG_DEGRADED)" in cluster log 2021-11-28T19:09:05.354 INFO:scrape:1 jobs: ['6530996'] 2021-11-28T19:09:05.354 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{centos_8.stream}', 'tasks/{0-install', 'test/kclient_workunit_suites_fsstress}}'] 2021-11-28T19:09:05.355 INFO:scrape: 2021-11-28T19:09:05.355 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on gibba045 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69b04de2932d00fc7fcaa14c718595ec42f18e67 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2021-11-28T19:09:05.355 INFO:scrape:1 jobs: ['6530998'] 2021-11-28T19:09:05.356 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{rhel_8}', 'tasks/{0-install', 'test/libcephfs_interface_tests}}'] 2021-11-28T19:09:05.356 INFO:scrape: 2021-11-28T19:09:05.357 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2021-11-28T19:09:05.357 INFO:scrape:1 jobs: ['6530999'] 2021-11-28T19:09:05.358 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{centos_8.stream}', 'tasks/{0-install', 'test/mon_thrash}}'] 2021-11-28T19:09:05.358 INFO:scrape: 2021-11-28T19:09:05.359 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba ================================================================= info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [6530996] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/kclient_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:12:51 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530996/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530996/ "2021-11-28T09:08:25.944109+0000 mon.a (mon.0) 154 : cluster [WRN] Health check failed: Degraded data redundancy: 26/17378 objects degraded (0.150%), 1 pg degraded (PG_DEGRADED)" in cluster log [6530998] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/libcephfs_interface_tests}} ----------------------------------------------------------------- time: 00:20:26 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530998/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=435137cc76704f05a0f3a77e6e9a0852 Command failed (workunit test libcephfs/test.sh) on gibba045 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69b04de2932d00fc7fcaa14c718595ec42f18e67 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [6530999] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/mon_thrash}} ----------------------------------------------------------------- time: 03:13:49 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530999/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530999/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b635a9595d164be59fe20c1a234a4cab Command failed (workunit test rados/test.sh) on gibba032 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69b04de2932d00fc7fcaa14c718595ec42f18e67 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' Pass ================================================================= [6530990] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/cfuse_workunit_suites_blogbench}} time: 00:18:22 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530990/ [6530991] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/cfuse_workunit_suites_fsstress}} time: 00:13:29 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530991/ [6530992] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_iozone}} time: 00:20:38 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530992/ [6530993] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/cfuse_workunit_suites_pjd}} time: 00:13:03 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530993/ [6530994] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_direct_io}} time: 00:19:46 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530994/ [6530995] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/kclient_workunit_suites_dbench}} time: 01:23:28 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530995/ [6530997] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_suites_pjd}} time: 00:11:20 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6530997/ [6531000] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_api_tests}} time: 00:26:30 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531000/ [6531001] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_bench}} time: 01:54:09 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531001/ [6531002] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_cache_snaps}} time: 00:27:24 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531002/ [6531003] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/rados_cls_all}} time: 00:18:40 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531003/ [6531004] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rados_ec_snaps}} time: 00:26:26 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531004/ [6531005] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_python}} time: 00:24:50 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531005/ [6531006] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_workunit_loadgen_mix}} time: 00:21:59 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531006/ [6531007] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/rbd_api_tests}} time: 00:28:01 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531007/ [6531008] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_cli_import_export}} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531008/ [6531009] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_fsx}} time: 00:20:10 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531009/ [6531010] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_python_api_tests}} time: 00:29:43 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531010/ [6531011] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_workunit_suites_iozone}} time: 00:14:27 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531011/ [6531012] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rgw_ec_s3tests}} time: 00:20:10 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531012/ [6531013] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/rgw_s3tests}} time: 00:20:56 info: http://pulpito.ceph.com/teuthology-2021-11-28_05:00:03-smoke-master-distro-default-gibba/6531013/