2021-03-15T15:25:20.963 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-15T15:25:20.965 INFO:teuthology.results:subset: '1/8' 2021-03-15T15:25:20.965 INFO:teuthology.results:seed: '414' 2021-03-15T15:37:47.706 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-15T15:37:47.754 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-15T16:09:50.081 INFO:teuthology.results:Tests finished! gathering results... 2021-03-15T16:09:51.231 INFO:scrape:Found 11 jobs 2021-03-15T16:09:57.688 INFO:scrape:Found 6 distinct failure reasons 2021-03-15T16:09:57.689 INFO:scrape:Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 2021-03-15T16:09:57.689 INFO:scrape:2 jobs: ['5967514', '5967520'] 2021-03-15T16:09:57.690 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.690 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.691 INFO:scrape: 2021-03-15T16:09:57.691 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5e888220931ade392ca9f4bd16a258d2ba8bd02 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 2021-03-15T16:09:57.692 INFO:scrape:1 jobs: ['5967517'] 2021-03-15T16:09:57.692 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/workunit/quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.693 INFO:scrape: 2021-03-15T16:09:57.693 INFO:scrape:Failure: Test failure: test_mirroring_command_idempotency (tasks.cephfs.test_admin.TestMirroringCommands) 2021-03-15T16:09:57.694 INFO:scrape:2 jobs: ['5967518', '5967512'] 2021-03-15T16:09:57.694 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'tasks/admin}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.694 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/admin}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.695 INFO:scrape: 2021-03-15T16:09:57.695 INFO:scrape:Failure: Test failure: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2021-03-15T16:09:57.696 INFO:scrape:1 jobs: ['5967522'] 2021-03-15T16:09:57.696 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.697 INFO:scrape: 2021-03-15T16:09:57.697 INFO:scrape:Failure: Test failure: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2021-03-15T16:09:57.697 INFO:scrape:1 jobs: ['5967516'] 2021-03-15T16:09:57.698 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.698 INFO:scrape: 2021-03-15T16:09:57.699 INFO:scrape:Failure: "2021-03-15T15:56:01.121084+0000 mds.a (mds.0) 4 : cluster [WRN] Corrupt dentry 'shadow' in dir frag 0x10000000000: Invalid tag char '4' pos 2: Malformed input(/parent)" in cluster log 2021-03-15T16:09:57.700 INFO:scrape:2 jobs: ['5967521', '5967515'] 2021-03-15T16:09:57.700 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.700 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs:functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'rhel_8}', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-03-15T16:09:57.701 INFO:scrape: 2021-03-15T16:09:57.702 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/ failed: 9 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [5967512] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} ----------------------------------------------------------------- time: 00:20:28 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967512/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967512/ sentry: https://sentry.ceph.com/organizations/ceph/?query=694db070b8544f7b968b8c8a489bae61 Test failure: test_mirroring_command_idempotency (tasks.cephfs.test_admin.TestMirroringCommands) [5967514] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} ----------------------------------------------------------------- time: 00:16:13 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967514/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967514/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eb1a8e47811e480ea91b4d649f582cd7 Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) [5967515] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} ----------------------------------------------------------------- time: 00:24:14 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967515/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967515/ "2021-03-15T16:03:10.760089+0000 mds.c (mds.0) 4 : cluster [WRN] Corrupt dentry 'shadow' in dir frag 0x10000000000: Invalid tag char '4' pos 2: Malformed input(/parent)" in cluster log [5967516] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:14:53 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967516/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967516/ sentry: https://sentry.ceph.com/organizations/ceph/?query=597374d53a1e43f58c70b88ff3fb6a69 Test failure: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) [5967517] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:10:50 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967517/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967517/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50b6a1a73e294a0fba147299b0d5139a Command failed (workunit test fs/quota/quota.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5e888220931ade392ca9f4bd16a258d2ba8bd02 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' [5967518] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} ----------------------------------------------------------------- time: 00:24:52 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967518/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967518/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc95430520b2411c8da9e2cd57691fb1 Test failure: test_mirroring_command_idempotency (tasks.cephfs.test_admin.TestMirroringCommands) [5967520] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} ----------------------------------------------------------------- time: 00:19:21 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967520/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967520/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d88c23e65f24c13a7ea558e1efd0e86 Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) [5967521] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} ----------------------------------------------------------------- time: 00:17:17 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967521/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967521/ "2021-03-15T15:56:01.121084+0000 mds.a (mds.0) 4 : cluster [WRN] Corrupt dentry 'shadow' in dir frag 0x10000000000: Invalid tag char '4' pos 2: Malformed input(/parent)" in cluster log [5967522] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:14:45 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967522/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967522/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2dbd293727843c989551ed5f07c9893 Test failure: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Pass ================================================================= [5967513] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} time: 00:23:36 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967513/ [5967519] fs:functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} time: 00:24:18 info: http://pulpito.ceph.com/pdonnell-2021-03-15_15:24:43-fs:functional-wip-pdonnell-testing-20210312.221823-distro-basic-smithi/5967519/