2022-10-24T20:08:01.963 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-24T20:08:01.966 INFO:teuthology.results:subset: '111/120000'
2022-10-24T20:08:01.967 INFO:teuthology.results:seed: '6855'
2022-10-24T22:10:24.419 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-24T22:10:24.448 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-25T05:53:05.163 INFO:teuthology.results:Tests finished! gathering results...
2022-10-25T05:53:14.219 INFO:scrape:Found 93 jobs
2022-10-25T05:54:27.441 INFO:scrape:Found 47 distinct failure reasons
2022-10-25T05:54:27.443 INFO:scrape:Failure: Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7954e164-53db-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''
2022-10-25T05:54:27.444 INFO:scrape:8 jobs: ['7079865', '7080032', '7079838', '7079992', '7080056', '7079902', '7079880', '7079953']
2022-10-25T05:54:27.444 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2022-10-25T05:54:27.445 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2022-10-25T05:54:27.445 INFO:scrape:
2022-10-25T05:54:27.446 INFO:scrape:Failure: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation)
2022-10-25T05:54:27.446 INFO:scrape:2 jobs: ['7079856', '7080066']
2022-10-25T05:54:27.447 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}']
2022-10-25T05:54:27.447 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/fragment}']
2022-10-25T05:54:27.447 INFO:scrape:
2022-10-25T05:54:27.448 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)
2022-10-25T05:54:27.448 INFO:scrape:1 jobs: ['7079868']
2022-10-25T05:54:27.449 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/mds-flush}']
2022-10-25T05:54:27.450 INFO:scrape:
2022-10-25T05:54:27.450 INFO:scrape:Failure: Test failure: test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2)
2022-10-25T05:54:27.451 INFO:scrape:1 jobs: ['7079886']
2022-10-25T05:54:27.452 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/multimds_misc}']
2022-10-25T05:54:27.453 INFO:scrape:
2022-10-25T05:54:27.454 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)
2022-10-25T05:54:27.454 INFO:scrape:1 jobs: ['7080042']
2022-10-25T05:54:27.455 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/client-readahead}']
2022-10-25T05:54:27.456 INFO:scrape:
2022-10-25T05:54:27.457 INFO:scrape:Failure: Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks)
2022-10-25T05:54:27.457 INFO:scrape:1 jobs: ['7079941']
2022-10-25T05:54:27.458 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/scrub}']
2022-10-25T05:54:27.458 INFO:scrape:
2022-10-25T05:54:27.459 INFO:scrape:Failure: Test failure: test_basic (tasks.cephfs.test_dump_tree.TestDumpTree)
2022-10-25T05:54:27.459 INFO:scrape:1 jobs: ['7080006']
2022-10-25T05:54:27.460 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/asok_dump_tree}']
2022-10-25T05:54:27.460 INFO:scrape:
2022-10-25T05:54:27.460 INFO:scrape:Failure: Test failure: test_barrier (tasks.cephfs.test_full.TestClusterFull)
2022-10-25T05:54:27.461 INFO:scrape:1 jobs: ['7079874']
2022-10-25T05:54:27.461 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2022-10-25T05:54:27.462 INFO:scrape:
2022-10-25T05:54:27.462 INFO:scrape:Failure: Test failure: test_fstop (tasks.cephfs.test_fstop.TestFSTop)
2022-10-25T05:54:27.463 INFO:scrape:1 jobs: ['7079980']
2022-10-25T05:54:27.463 INFO:scrape:suites: ['cluster/{1-node}', 'fs/top/{begin', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_8}', 'tasks/fstop}']
2022-10-25T05:54:27.464 INFO:scrape:
2022-10-25T05:54:27.465 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2022-10-25T05:54:27.465 INFO:scrape:4 jobs: ['7079905', '7080058', '7080010', '7079871']
2022-10-25T05:54:27.466 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2022-10-25T05:54:27.467 INFO:scrape:suites union: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'fs/verify/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/lockdep}']
2022-10-25T05:54:27.468 INFO:scrape:
2022-10-25T05:54:27.468 INFO:scrape:Failure: Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)
2022-10-25T05:54:27.468 INFO:scrape:3 jobs: ['7079974', '7079892', '7080050']
2022-10-25T05:54:27.469 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2022-10-25T05:54:27.470 INFO:scrape:suites union: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2022-10-25T05:54:27.470 INFO:scrape:
2022-10-25T05:54:27.471 INFO:scrape:Failure: Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair)
2022-10-25T05:54:27.471 INFO:scrape:2 jobs: ['7079862', '7080070']
2022-10-25T05:54:27.472 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/journal-repair}']
2022-10-25T05:54:27.472 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/journal-repair}']
2022-10-25T05:54:27.473 INFO:scrape:
2022-10-25T05:54:27.473 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2022-10-25T05:54:27.473 INFO:scrape:1 jobs: ['7079994']
2022-10-25T05:54:27.474 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}']
2022-10-25T05:54:27.474 INFO:scrape:
2022-10-25T05:54:27.475 INFO:scrape:Failure: Test failure: test_dir_deletion (tasks.cephfs.test_strays.TestStrays)
2022-10-25T05:54:27.476 INFO:scrape:1 jobs: ['7079962']
2022-10-25T05:54:27.476 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/strays}']
2022-10-25T05:54:27.477 INFO:scrape:
2022-10-25T05:54:27.477 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)
2022-10-25T05:54:27.477 INFO:scrape:2 jobs: ['7079968', '7079984']
2022-10-25T05:54:27.478 INFO:scrape:suites intersection: ['mount/fuse']
2022-10-25T05:54:27.478 INFO:scrape:suites union: ['centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}']
2022-10-25T05:54:27.479 INFO:scrape:
2022-10-25T05:54:27.480 INFO:scrape:Failure: Test failure: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop)
2022-10-25T05:54:27.480 INFO:scrape:2 jobs: ['7080068', '7079971']
2022-10-25T05:54:27.481 INFO:scrape:suites intersection: ['conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}']
2022-10-25T05:54:27.481 INFO:scrape:suites union: ['clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}']
2022-10-25T05:54:27.481 INFO:scrape:
2022-10-25T05:54:27.482 INFO:scrape:Failure: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh'
2022-10-25T05:54:27.482 INFO:scrape:1 jobs: ['7079844']
2022-10-25T05:54:27.483 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-10-25T05:54:27.483 INFO:scrape:
2022-10-25T05:54:27.484 INFO:scrape:Failure: Test failure: test_abort_conn (tasks.cephfs.test_client_recovery.TestClientRecovery)
2022-10-25T05:54:27.484 INFO:scrape:1 jobs: ['7080048']
2022-10-25T05:54:27.485 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/client-recovery}']
2022-10-25T05:54:27.485 INFO:scrape:
2022-10-25T05:54:27.486 INFO:scrape:Failure: Test failure: test_cap_acquisition_throttle_readdir (tasks.cephfs.test_client_limits.TestClientLimits)
2022-10-25T05:54:27.487 INFO:scrape:1 jobs: ['7080038']
2022-10-25T05:54:27.487 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}']
2022-10-25T05:54:27.488 INFO:scrape:
2022-10-25T05:54:27.488 INFO:scrape:Failure: Test failure: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc)
2022-10-25T05:54:27.489 INFO:scrape:2 jobs: ['7080062', '7079877']
2022-10-25T05:54:27.489 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/misc}}']
2022-10-25T05:54:27.490 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/misc}}']
2022-10-25T05:54:27.490 INFO:scrape:
2022-10-25T05:54:27.491 INFO:scrape:Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable)
2022-10-25T05:54:27.491 INFO:scrape:1 jobs: ['7079889']
2022-10-25T05:54:27.491 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/openfiletable}']
2022-10-25T05:54:27.492 INFO:scrape:
2022-10-25T05:54:27.492 INFO:scrape:Failure: Test failure: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2022-10-25T05:54:27.493 INFO:scrape:2 jobs: ['7080064', '7079853']
2022-10-25T05:54:27.493 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-10-25T05:54:27.494 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-10-25T05:54:27.494 INFO:scrape:
2022-10-25T05:54:27.495 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
2022-10-25T05:54:27.495 INFO:scrape:1 jobs: ['7079988']
2022-10-25T05:54:27.496 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/test_journal_migration}']
2022-10-25T05:54:27.496 INFO:scrape:
2022-10-25T05:54:27.496 INFO:scrape:Failure: Test failure: test_15303 (tasks.cephfs.test_volume_client.TestVolumeClient)
2022-10-25T05:54:27.497 INFO:scrape:1 jobs: ['7079986']
2022-10-25T05:54:27.498 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volume-client}']
2022-10-25T05:54:27.498 INFO:scrape:
2022-10-25T05:54:27.499 INFO:scrape:Failure: Test failure: test_subvolume_group_snapshot_unsupported_status (tasks.cephfs.test_volumes.TestSubvolumeGroupSnapshots)
2022-10-25T05:54:27.499 INFO:scrape:1 jobs: ['7079923']
2022-10-25T05:54:27.500 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/snapshot}}']
2022-10-25T05:54:27.501 INFO:scrape:
2022-10-25T05:54:27.502 INFO:scrape:Failure: Test failure: test_allow_new_snaps_config (tasks.cephfs.test_snapshots.TestSnapshots)
2022-10-25T05:54:27.502 INFO:scrape:1 jobs: ['7079959']
2022-10-25T05:54:27.503 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/snapshots}']
2022-10-25T05:54:27.503 INFO:scrape:
2022-10-25T05:54:27.504 INFO:scrape:Failure: Test failure: test_ephemeral_pin_dist_conf_off (tasks.cephfs.test_exports.TestEphemeralPins)
2022-10-25T05:54:27.504 INFO:scrape:2 jobs: ['7079850', '7080060']
2022-10-25T05:54:27.505 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/exports}']
2022-10-25T05:54:27.505 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}']
2022-10-25T05:54:27.505 INFO:scrape:
2022-10-25T05:54:27.506 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
2022-10-25T05:54:27.506 INFO:scrape:2 jobs: ['7079841', '7080054']
2022-10-25T05:54:27.507 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}']
2022-10-25T05:54:27.507 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/data-scan}']
2022-10-25T05:54:27.508 INFO:scrape:
2022-10-25T05:54:27.509 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)
2022-10-25T05:54:27.509 INFO:scrape:1 jobs: ['7079977']
2022-10-25T05:54:27.510 INFO:scrape:suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/shell/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2022-10-25T05:54:27.511 INFO:scrape:
2022-10-25T05:54:27.511 INFO:scrape:Failure: Test failure: test_replay_create (tasks.cephfs.test_cap_flush.TestCapFlush)
2022-10-25T05:54:27.512 INFO:scrape:1 jobs: ['7080030']
2022-10-25T05:54:27.512 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/cap-flush}']
2022-10-25T05:54:27.513 INFO:scrape:
2022-10-25T05:54:27.513 INFO:scrape:Failure: Test failure: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity)
2022-10-25T05:54:27.513 INFO:scrape:2 jobs: ['7080004', '7079859']
2022-10-25T05:54:27.514 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}']
2022-10-25T05:54:27.514 INFO:scrape:suites union: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mon-debug}', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/failover}']
2022-10-25T05:54:27.515 INFO:scrape:
2022-10-25T05:54:27.515 INFO:scrape:Failure: Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
2022-10-25T05:54:27.516 INFO:scrape:1 jobs: ['7080002']
2022-10-25T05:54:27.516 INFO:scrape:suites: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2022-10-25T05:54:27.517 INFO:scrape:
2022-10-25T05:54:27.518 INFO:scrape:Failure: Command failed on smithi047 with status 5: 'sudo systemctl stop ceph-bdeaeaba-53e2-11ed-8438-001a4aab830c@mon.smithi047'
2022-10-25T05:54:27.518 INFO:scrape:1 jobs: ['7079965']
2022-10-25T05:54:27.519 INFO:scrape:suites: ['1-mount', '2-workload/dbench', 'distro/centos_latest}', 'fs/cephadm/multivolume/{0-start']
2022-10-25T05:54:27.520 INFO:scrape:
2022-10-25T05:54:27.520 INFO:scrape:Failure: Test failure: test_client_metrics_and_metadata (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2022-10-25T05:54:27.521 INFO:scrape:1 jobs: ['7079883']
2022-10-25T05:54:27.521 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/metrics}']
2022-10-25T05:54:27.522 INFO:scrape:
2022-10-25T05:54:27.522 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2022-10-25T05:54:27.523 INFO:scrape:1 jobs: ['7080034']
2022-10-25T05:54:27.523 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2022-10-25T05:54:27.524 INFO:scrape:
2022-10-25T05:54:27.524 INFO:scrape:Failure: Test failure: test_dump_subtrees (tasks.cephfs.test_admin.TestAdminCommandDumpTree)
2022-10-25T05:54:27.525 INFO:scrape:1 jobs: ['7079996']
2022-10-25T05:54:27.525 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/admin}']
2022-10-25T05:54:27.526 INFO:scrape:
2022-10-25T05:54:27.526 INFO:scrape:Failure: Test failure: test_recover_fs_after_fsmap_removal (tasks.cephfs.test_recovery_fs.TestFSRecovery)
2022-10-25T05:54:27.527 INFO:scrape:1 jobs: ['7079929']
2022-10-25T05:54:27.527 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/recovery-fs}']
2022-10-25T05:54:27.528 INFO:scrape:
2022-10-25T05:54:27.528 INFO:scrape:Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool)
2022-10-25T05:54:27.529 INFO:scrape:1 jobs: ['7080000']
2022-10-25T05:54:27.529 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/alternate-pool}']
2022-10-25T05:54:27.530 INFO:scrape:
2022-10-25T05:54:27.530 INFO:scrape:Failure: Test failure: test_volume_create (tasks.cephfs.test_volumes.TestVolumes)
2022-10-25T05:54:27.531 INFO:scrape:1 jobs: ['7079998']
2022-10-25T05:54:27.531 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}']
2022-10-25T05:54:27.531 INFO:scrape:
2022-10-25T05:54:27.532 INFO:scrape:Failure: Test failure: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules)
2022-10-25T05:54:27.532 INFO:scrape:1 jobs: ['7079956']
2022-10-25T05:54:27.533 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2022-10-25T05:54:27.533 INFO:scrape:
2022-10-25T05:54:27.534 INFO:scrape:Failure: Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2022-10-25T05:54:27.534 INFO:scrape:2 jobs: ['7079847', '7080036']
2022-10-25T05:54:27.534 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}']
2022-10-25T05:54:27.535 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}']
2022-10-25T05:54:27.535 INFO:scrape:
2022-10-25T05:54:27.536 INFO:scrape:Failure: Test failure: test_mount_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap)
2022-10-25T05:54:27.536 INFO:scrape:1 jobs: ['7079950']
2022-10-25T05:54:27.537 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/sessionmap}']
2022-10-25T05:54:27.537 INFO:scrape:
2022-10-25T05:54:27.538 INFO:scrape:Failure: Test failure: test_backtrace_repair (tasks.cephfs.test_auto_repair.TestMDSAutoRepair)
2022-10-25T05:54:27.538 INFO:scrape:1 jobs: ['7080012']
2022-10-25T05:54:27.538 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/auto-repair}']
2022-10-25T05:54:27.539 INFO:scrape:
2022-10-25T05:54:27.539 INFO:scrape:Failure: Test failure: test_remote_update_df (tasks.cephfs.test_quota.TestQuota)
2022-10-25T05:54:27.540 INFO:scrape:1 jobs: ['7079917']
2022-10-25T05:54:27.540 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/quota}']
2022-10-25T05:54:27.541 INFO:scrape:
2022-10-25T05:54:27.541 INFO:scrape:Failure: Test failure: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm)
2022-10-25T05:54:27.541 INFO:scrape:1 jobs: ['7079899']
2022-10-25T05:54:27.542 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/pool-perm}']
2022-10-25T05:54:27.542 INFO:scrape:
2022-10-25T05:54:27.543 INFO:scrape:Failure: Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)
2022-10-25T05:54:27.543 INFO:scrape:1 jobs: ['7080026']
2022-10-25T05:54:27.544 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/backtrace}']
2022-10-25T05:54:27.544 INFO:scrape:
2022-10-25T05:54:27.545 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)
2022-10-25T05:54:27.545 INFO:scrape:1 jobs: ['7080052']
2022-10-25T05:54:27.545 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/damage}']
2022-10-25T05:54:27.546 INFO:scrape: