2022-07-02T10:49:17.759 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-02T10:49:17.760 INFO:teuthology.results:subset: '7/32' 2022-07-02T10:49:17.760 INFO:teuthology.results:seed: '4012' 2022-07-02T10:57:14.193 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-02T10:57:14.197 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-02T22:58:11.588 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-07-02T22:58:12.225 INFO:scrape:Found 6 jobs 2022-07-02T22:58:13.612 INFO:scrape:Found 3 distinct failure reasons 2022-07-02T22:58:13.613 INFO:scrape:Failure: Command failed on smithi150 with status 1: 'sudo yum upgrade -y linux-firmware' 2022-07-02T22:58:13.614 INFO:scrape:2 jobs: ['6910348', '6910352'] 2022-07-02T22:58:13.614 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-02T22:58:13.615 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/basic}}', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-02T22:58:13.615 INFO:scrape: 2022-07-02T22:58:13.616 INFO:scrape:Dead: b'2022-07-02T22:56:41.934 DEBUG:teuthology.exit:Finished running handlers' 2022-07-02T22:58:13.616 INFO:scrape:1 jobs: ['6910349'] 2022-07-02T22:58:13.617 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-02T22:58:13.617 INFO:scrape: 2022-07-02T22:58:13.617 INFO:scrape:Dead: b'2022-07-02T22:58:12.904 INFO:tasks.ceph.osd.5.smithi150.stderr:2022-07-02T22:58:12.901+0000 7f8b547cf700 -1 osd.5 438 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.371:2 54.b 54.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force e395)' 2022-07-02T22:58:13.618 INFO:scrape:1 jobs: ['6910353'] 2022-07-02T22:58:13.618 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-02T22:58:13.619 INFO:scrape: 2022-07-02T22:58:13.620 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/ failed: 2 dead: 0 running: 2 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [6910348] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910348/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f74a4c81521746678c6ae2083c7eb0e1 Command failed on smithi150 with status 1: 'sudo yum upgrade -y linux- firmware' [6910352] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910352/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33691e9f522f496ea1ade7ddd9006ca5 Command failed on smithi115 with status 1: 'sudo yum upgrade -y linux- firmware' Running ================================================================= [6910349] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} info: http://pulpito.ceph.com/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910349/ [6910353] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} info: http://pulpito.ceph.com/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910353/ Pass ================================================================= [6910350] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} time: 00:27:18 info: http://pulpito.ceph.com/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910350/ [6910351] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volume-client} time: 00:26:32 info: http://pulpito.ceph.com/vshankar-2022-07-02_10:48:35-fs:volumes-pacific-sec-testing-default-smithi/6910351/