2023-06-09T20:58:54.526 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd 2023-06-09T20:58:54.531 INFO:teuthology.results:subset: '111/120000' 2023-06-09T20:58:54.531 INFO:teuthology.results:seed: '514' 2023-06-09T20:59:56.153 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd 2023-06-09T20:59:56.157 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-10T09:00:54.993 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-06-10T09:00:56.860 INFO:scrape:Found 24 jobs 2023-06-10T09:00:57.668 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299974/teuthology.log 2023-06-10T09:00:57.726 INFO:scrape:Found 4 distinct failure reasons 2023-06-10T09:00:57.727 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi125 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5937c68aed42cde57f938ad8f8bb264f13176009 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-06-10T09:00:57.727 INFO:scrape:1 jobs: ['7299959'] 2023-06-10T09:00:57.728 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 's3tests-branch', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/{0-install', 'test/libcephfs_interface_tests}}'] 2023-06-10T09:00:57.729 INFO:scrape: 2023-06-10T09:00:57.729 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5937c68aed42cde57f938ad8f8bb264f13176009 2023-06-10T09:00:57.729 INFO:scrape:3 jobs: ['7299972', '7299950', '7299982'] 2023-06-10T09:00:57.730 INFO:scrape:suites intersection: ['objectstore/bluestore-bitmap', 'openstack}', 's3tests-branch', 'smoke/basic/{clusters/{fixed-3-cephfs', 'tasks/{0-install'] 2023-06-10T09:00:57.730 INFO:scrape:suites union: ['objectstore/bluestore-bitmap', 'openstack}', 's3tests-branch', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/{0-install', 'test/cfuse_workunit_suites_fsstress}}', 'test/rados_python}}', 'test/rbd_python_api_tests}}'] 2023-06-10T09:00:57.730 INFO:scrape: 2023-06-10T09:00:57.731 INFO:scrape:Dead: b'2023-06-09T21:40:42.274 INFO:teuthology.orchestra.run.smithi103.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 9 21:40 /home/ubuntu/cephtest' 2023-06-10T09:00:57.731 INFO:scrape:1 jobs: ['7299955'] 2023-06-10T09:00:57.731 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 's3tests-branch', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/{0-install', 'test/kclient_workunit_suites_fsstress}}'] 2023-06-10T09:00:57.731 INFO:scrape: 2023-06-10T09:00:57.732 INFO:scrape:Dead: None 2023-06-10T09:00:57.732 INFO:scrape:1 jobs: ['7299974'] 2023-06-10T09:00:57.732 INFO:scrape: 2023-06-10T09:00:57.734 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/ failed: 4 dead: 1 running: 1 waiting: 0 queued: 0 passed: 18 Fail ================================================================= [7299950] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install test/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:14:54 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299950/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299950/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3238c170d13a4d189ad202d80f80ac3e Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=593 7c68aed42cde57f938ad8f8bb264f13176009 [7299959] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/libcephfs_interface_tests}} ----------------------------------------------------------------- time: 00:18:21 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299959/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299959/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e7a53b87977400ca1c265897c6422f3 Command failed (workunit test libcephfs/test.sh) on smithi125 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5937c68aed42cde57f938ad8f8bb264f13176009 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7299972] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install test/rados_python}} ----------------------------------------------------------------- time: 00:10:58 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299972/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299972/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7633add0d0c4c44a0fbca484aa4e69b Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=593 7c68aed42cde57f938ad8f8bb264f13176009 [7299982] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_python_api_tests}} ----------------------------------------------------------------- time: 00:09:08 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299982/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=614dcba70c89457b8d85f57e4d3bf5ed Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=593 7c68aed42cde57f938ad8f8bb264f13176009 Dead ================================================================= [7299974] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install test/rados_workunit_loadgen_mix}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299974/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299974/ Error reimaging machines: Expected smithi193's OS to be centos 8 but found rhel 8.6 Running ================================================================= [7299955] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/kclient_workunit_suites_fsstress}} info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299955/ Pass ================================================================= [7299949] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/cfuse_workunit_suites_blogbench}} time: 00:23:05 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299949/ [7299951] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_iozone}} time: 00:24:05 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299951/ [7299952] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_pjd}} time: 00:27:35 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299952/ [7299953] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_direct_io}} time: 00:18:40 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299953/ [7299954] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_suites_dbench}} time: 00:45:08 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299954/ [7299956] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_suites_pjd}} time: 00:20:08 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299956/ [7299961] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/mon_thrash}} time: 00:35:33 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299961/ [7299963] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/rados_api_tests}} time: 00:30:58 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299963/ [7299965] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/rados_bench}} time: 00:49:22 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299965/ [7299967] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_cache_snaps}} time: 00:29:06 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299967/ [7299969] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/rados_cls_all}} time: 00:24:36 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299969/ [7299971] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install test/rados_ec_snaps}} time: 00:34:29 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299971/ [7299976] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/rbd_api_tests}} time: 00:49:14 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299976/ [7299978] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install test/rbd_cli_import_export}} time: 00:16:51 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299978/ [7299980] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rbd_fsx}} time: 00:19:33 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299980/ [7299984] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_workunit_suites_iozone}} time: 00:29:19 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299984/ [7299986] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rgw_ec_s3tests}} time: 00:30:36 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299986/ [7299988] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rgw_s3tests}} time: 00:29:06 info: http://pulpito.ceph.com/yuriw-2023-06-09_20:58:16-smoke-reef-distro-default-smithi/7299988/