2022-01-11T05:01:17.961 INFO:root:teuthology version: 1.1.0-00e09f05 2022-01-11T05:01:17.965 INFO:teuthology.results:seed: '5123' 2022-01-11T05:21:01.967 INFO:root:teuthology version: 1.1.0-00e09f05 2022-01-11T05:21:02.001 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-01-11T06:40:08.069 INFO:teuthology.results:Tests finished! gathering results... 2022-01-11T06:40:10.403 INFO:scrape:Found 24 jobs 2022-01-11T06:40:11.292 INFO:scrape:Found 4 distinct failure reasons 2022-01-11T06:40:11.293 INFO:scrape:Failure: Command failed on smithi175 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' 2022-01-11T06:40:11.294 INFO:scrape:2 jobs: ['6607279', '6607281'] 2022-01-11T06:40:11.294 INFO:scrape:suites intersection: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{0-install'] 2022-01-11T06:40:11.295 INFO:scrape:suites union: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{0-install', 'test/kclient_workunit_direct_io}}', 'test/kclient_workunit_suites_fsstress}}'] 2022-01-11T06:40:11.295 INFO:scrape: 2022-01-11T06:40:11.296 INFO:scrape:Failure: Command failed on smithi104 with status 1: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' 2022-01-11T06:40:11.296 INFO:scrape:2 jobs: ['6607294', '6607285'] 2022-01-11T06:40:11.297 INFO:scrape:suites intersection: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'tasks/{0-install'] 2022-01-11T06:40:11.297 INFO:scrape:suites union: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{0-install', 'test/rados_api_tests}}', 'test/rbd_fsx}}'] 2022-01-11T06:40:11.298 INFO:scrape: 2022-01-11T06:40:11.298 INFO:scrape:Failure: failed to install new kernel version within timeout 2022-01-11T06:40:11.298 INFO:scrape:1 jobs: ['6607295'] 2022-01-11T06:40:11.299 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{centos_8.stream}', 'tasks/{0-install', 'test/rbd_python_api_tests}}'] 2022-01-11T06:40:11.299 INFO:scrape: 2022-01-11T06:40:11.300 INFO:scrape:Failure: SSH connection to smithi041 was lost: 'rpm -q kernel --last | head -n 1' 2022-01-11T06:40:11.300 INFO:scrape:1 jobs: ['6607284'] 2022-01-11T06:40:11.301 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{centos_8}', 'tasks/{0-install', 'test/mon_thrash}}'] 2022-01-11T06:40:11.302 INFO:scrape: 2022-01-11T06:40:11.304 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/ failed: 5 dead: 1 running: 0 waiting: 0 queued: 0 passed: 18 Fail ================================================================= [6607279] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_direct_io}} ----------------------------------------------------------------- time: 00:13:21 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607279/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607279/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18b664d24fb844fd820121857e8030be Command failed on smithi175 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' [6607281] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/kclient_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:10:27 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607281/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607281/ sentry: https://sentry.ceph.com/organizations/ceph/?query=993fe7f27bd74b48b15d44f81e8d6b94 Command failed on smithi196 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' [6607285] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rados_api_tests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607285/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607285/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af0ceb8758944c0b8bea62985b36cff3 Command failed on smithi104 with status 1: 'sudo yum install -y kernel' [6607294] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rbd_fsx}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607294/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607294/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2528297053684504b203db1e217759d6 Command failed on smithi104 with status 1: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' [6607295] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/rbd_python_api_tests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607295/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e3dd090c77b455e99d90754452b203c failed to install new kernel version within timeout Dead ================================================================= [6607284] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/mon_thrash}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607284/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdb3c8d135914e8081ab0eaae352c1f1 SSH connection to smithi041 was lost: 'rpm -q kernel --last | head -n 1' Pass ================================================================= [6607275] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/cfuse_workunit_suites_blogbench}} time: 00:19:30 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607275/ [6607276] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/cfuse_workunit_suites_fsstress}} time: 00:12:57 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607276/ [6607277] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/cfuse_workunit_suites_iozone}} time: 00:13:50 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607277/ [6607278] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_pjd}} time: 00:26:57 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607278/ [6607280] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/kclient_workunit_suites_dbench}} time: 01:12:04 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607280/ [6607282] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/kclient_workunit_suites_pjd}} time: 00:14:19 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607282/ [6607283] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/libcephfs_interface_tests}} time: 00:25:48 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607283/ [6607286] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/rados_bench}} time: 01:21:01 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607286/ [6607287] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_cache_snaps}} time: 00:22:36 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607287/ [6607288] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_cls_all}} time: 00:31:58 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607288/ [6607289] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_ec_snaps}} time: 00:24:03 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607289/ [6607290] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/rados_python}} time: 00:18:14 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607290/ [6607291] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} tasks/{0-install test/rados_workunit_loadgen_mix}} time: 00:23:52 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607291/ [6607292] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_api_tests}} time: 00:29:02 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607292/ [6607293] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_cli_import_export}} time: 00:14:23 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607293/ [6607296] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_workunit_suites_iozone}} time: 00:30:02 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607296/ [6607297] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rgw_ec_s3tests}} time: 00:23:30 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607297/ [6607298] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rgw_s3tests}} time: 00:36:50 info: http://pulpito.ceph.com/teuthology-2022-01-11_05:00:02-smoke-master-distro-default-smithi/6607298/