2020-11-28T22:33:58.004 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-11-28T22:33:58.005 INFO:teuthology.results:subset: '24/9999' 2020-11-28T22:33:58.006 INFO:teuthology.results:seed: '7660' 2020-11-28T22:46:26.297 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-11-28T22:46:26.331 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-29T02:05:29.840 INFO:teuthology.results:Tests finished! gathering results... 2020-11-29T02:05:37.313 INFO:scrape:Found 24 jobs 2020-11-29T02:05:38.093 INFO:scrape:Found 2 distinct failure reasons 2020-11-29T02:05:38.094 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2020-11-29T02:05:38.095 INFO:scrape:1 jobs: ['5657174'] 2020-11-29T02:05:38.095 INFO:scrape:suites: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}'] 2020-11-29T02:05:38.096 INFO:scrape: 2020-11-29T02:05:38.096 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/fs/norstats/kernel_untar_tar.sh 2020-11-29T02:05:38.097 INFO:scrape:1 jobs: ['5657179'] 2020-11-29T02:05:38.097 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_norstats}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-11-29T02:05:38.098 INFO:scrape: 2020-11-29T02:05:38.099 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [5657174] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:48:19 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657174/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657174/ reached maximum tries (50) after waiting for 300 seconds [5657179] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats} ----------------------------------------------------------------- time: 03:15:26 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657179/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e33873c0544a43528563bd3f940a4037 Command failed (workunit test fs/norstats/kernel_untar_tar.sh) on smithi159 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a83a3572a957356977342f461361c1cd3ec06dbd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/fs/norstats/kernel_untar_tar.sh' Pass ================================================================= [5657170] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep} time: 01:00:15 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657170/ [5657171] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} time: 00:40:56 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657171/ [5657172] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots} time: 00:21:42 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657172/ [5657173] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:13:45 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657173/ [5657175] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build} time: 02:33:18 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657175/ [5657176] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc} time: 00:42:32 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657176/ [5657177] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:19:15 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657177/ [5657178] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep} time: 01:01:00 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657178/ [5657180] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench} time: 00:43:03 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657180/ [5657181] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:15:48 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657181/ [5657182] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind} time: 02:27:25 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657182/ [5657183] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench} time: 01:08:35 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657183/ [5657184] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb} time: 01:50:12 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657184/ [5657185] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:17:00 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657185/ [5657186] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} time: 00:18:30 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657186/ [5657187] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress} time: 00:17:26 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657187/ [5657188] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx} time: 00:27:11 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657188/ [5657189] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:18:31 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657189/ [5657190] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} time: 00:56:04 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657190/ [5657191] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd} time: 00:13:38 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657191/ [5657192] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} time: 00:42:03 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657192/ [5657193] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:19:04 info: http://pulpito.ceph.com/teuthology-2020-11-25_13:15:03-multimds-octopus-distro-basic-smithi/5657193/