2022-07-02T06:49:14.158 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-07-02T06:49:14.160 INFO:teuthology.results:subset: '3/4'
2022-07-02T06:49:14.160 INFO:teuthology.results:seed: '9718'
2022-07-02T07:37:21.923 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-07-02T07:37:21.948 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-07-02T19:38:18.142 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-07-02T19:38:19.758 INFO:scrape:Found 18 jobs
2022-07-02T19:38:22.400 INFO:scrape:Found 5 distinct failure reasons
2022-07-02T19:38:22.402 INFO:scrape:Dead: b'2022-07-02T19:10:39.244 DEBUG:teuthology.exit:Finished running handlers'
2022-07-02T19:38:22.402 INFO:scrape:2 jobs: ['6910268', '6910277']
2022-07-02T19:38:22.403 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides']
2022-07-02T19:38:22.404 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}', 'test/snapshot}}']
2022-07-02T19:38:22.405 INFO:scrape:
2022-07-02T19:38:22.406 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_metadata_after_snapshot_remove (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)
2022-07-02T19:38:22.407 INFO:scrape:1 jobs: ['6910278']
2022-07-02T19:38:22.408 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/snapshot}}']
2022-07-02T19:38:22.408 INFO:scrape:
2022-07-02T19:38:22.409 INFO:scrape:Failure: Test failure: test_multitenant_subvolumes (tasks.cephfs.test_volumes.TestSubvolumes)
2022-07-02T19:38:22.412 INFO:scrape:1 jobs: ['6910270']
2022-07-02T19:38:22.413 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}']
2022-07-02T19:38:22.413 INFO:scrape:
2022-07-02T19:38:22.414 INFO:scrape:Failure: {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}}
2022-07-02T19:38:22.414 INFO:scrape:2 jobs: ['6910269', '6910276']
2022-07-02T19:38:22.415 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides']
2022-07-02T19:38:22.415 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}', 'test/misc}}']
2022-07-02T19:38:22.416 INFO:scrape:
2022-07-02T19:38:22.416 INFO:scrape:Dead: b'2022-07-02T19:38:20.799 INFO:tasks.ceph.osd.5.smithi070.stderr:2022-07-02T19:38:20.796+0000 7f759c9b1700 -1 osd.5 317 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.160:4 37.6 37.4b2c82a6 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e271)'
2022-07-02T19:38:22.417 INFO:scrape:1 jobs: ['6910279']
2022-07-02T19:38:22.417 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/snapshot}}']
2022-07-02T19:38:22.418 INFO:scrape:
2022-07-02T19:38:22.420 INFO:teuthology.results:Sending results to nshelke@redhat.com: Test Run: nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/
failed:  2
dead:    3
running: 2
waiting: 0
queued:  0
passed:  11


Fail
=================================================================
[6910270]  fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   01:03:49
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910270/
log:    http://qa-proxy.ceph.com/teuthology/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910270/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7b95df80f8474dc5ae1e36f39080d0cc

    Test failure: test_multitenant_subvolumes
    (tasks.cephfs.test_volumes.TestSubvolumes)

[6910278]  fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
-----------------------------------------------------------------
time:   00:39:04
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910278/
log:    http://qa-proxy.ceph.com/teuthology/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910278/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9f12712f61844b369ca924c2007a3abd

    Test failure: test_subvolume_snapshot_metadata_after_snapshot_remove
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshots)



Dead
=================================================================
[6910268]  fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910268/
log:    http://qa-proxy.ceph.com/teuthology/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910268/

    hit max job timeout

[6910269]  fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:10:54
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910269/
log:    http://qa-proxy.ceph.com/teuthology/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910269/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b9e8dd30d8af43af9107962c18ee6a25

    {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}

[6910276]  fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:06:54
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910276/
log:    http://qa-proxy.ceph.com/teuthology/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910276/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c874aebeaa1546128a6962f43dd2d843

    {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output
    has been hidden due to the fact that 'no_log: true' was specified for this
    result", 'changed': True}}



Running
=================================================================
[6910277] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910277/

[6910279] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910279/



Pass
=================================================================
[6910262] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:17:09
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910262/

[6910263] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:17:11
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910263/

[6910264] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:17:22
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910264/

[6910265] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:29:16
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910265/

[6910266] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:28:56
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910266/

[6910267] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:28:51
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910267/

[6910271] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:50:00
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910271/

[6910272] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:50:26
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910272/

[6910273] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
time:   00:49:16
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910273/

[6910274] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:17:39
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910274/

[6910275] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:17:24
info:   http://pulpito.ceph.com/nshelke-2022-07-02_04:40:52-fs:volumes-wip-nshelke-in-progress-clone-testing-distro-default-smithi/6910275/