2023-03-13T05:38:55.303 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-13T05:38:55.304 INFO:teuthology.results:subset: '1007/1024' 2023-03-13T05:38:55.304 INFO:teuthology.results:seed: '2881' 2023-03-13T07:01:48.756 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-13T07:01:48.793 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-13T18:53:45.568 INFO:teuthology.results:Tests finished! gathering results... 2023-03-13T18:53:47.499 INFO:scrape:Found 23 jobs 2023-03-13T18:53:48.229 WARNING:scrape:Misunderstood line: 2023-03-13T18:53:51.871 WARNING:scrape:Misunderstood line: 2023-03-13T18:53:57.586 INFO:scrape:Found 10 distinct failure reasons 2023-03-13T18:53:57.587 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-03-13T18:53:57.587 INFO:scrape:1 jobs: ['7205242'] 2023-03-13T18:53:57.587 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-03-13T18:53:57.588 INFO:scrape: 2023-03-13T18:53:57.588 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-03-13T18:53:57.588 INFO:scrape:2 jobs: ['7205229', '7205246'] 2023-03-13T18:53:57.589 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-03-13T18:53:57.589 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-03-13T18:53:57.589 INFO:scrape: 2023-03-13T18:53:57.590 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-03-13T18:53:57.590 INFO:scrape:1 jobs: ['7205226'] 2023-03-13T18:53:57.590 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-03-13T18:53:57.590 INFO:scrape: 2023-03-13T18:53:57.591 INFO:scrape:Dead: b'2023-03-13T18:47:34.901 DEBUG:teuthology.exit:Finished running handlers' 2023-03-13T18:53:57.591 INFO:scrape:1 jobs: ['7205233'] 2023-03-13T18:53:57.591 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-03-13T18:53:57.592 INFO:scrape: 2023-03-13T18:53:57.592 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' 2023-03-13T18:53:57.592 INFO:scrape:1 jobs: ['7205232'] 2023-03-13T18:53:57.592 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-03-13T18:53:57.593 INFO:scrape: 2023-03-13T18:53:57.593 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2023-03-13T18:53:57.593 INFO:scrape:1 jobs: ['7205238'] 2023-03-13T18:53:57.594 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2023-03-13T18:53:57.594 INFO:scrape: 2023-03-13T18:53:57.594 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-03-13T18:53:57.594 INFO:scrape:1 jobs: ['7205225'] 2023-03-13T18:53:57.595 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-03-13T18:53:57.595 INFO:scrape: 2023-03-13T18:53:57.595 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-13T18:53:57.596 INFO:scrape:1 jobs: ['7205230'] 2023-03-13T18:53:57.596 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}'] 2023-03-13T18:53:57.596 INFO:scrape: 2023-03-13T18:53:57.596 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-03-13T18:53:57.597 INFO:scrape:1 jobs: ['7205236'] 2023-03-13T18:53:57.597 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2023-03-13T18:53:57.597 INFO:scrape: 2023-03-13T18:53:57.598 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-03-13T18:53:57.598 INFO:scrape:1 jobs: ['7205239'] 2023-03-13T18:53:57.598 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-03-13T18:53:57.598 INFO:scrape: 2023-03-13T18:53:57.600 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/ failed: 10 dead: 1 running: 0 waiting: 0 queued: 0 passed: 12 Fail ================================================================= [7205225] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:18:54 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205225/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205225/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea49426e20e942cb9fd8cc41cf6e4903 Command failed (workunit test libcephfs/test.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7205226] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:48:04 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205226/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205226/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7794ac944b5941cdb818c89a1b2c2d4f Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7205229] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:00:21 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205229/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb1f1299ccfc4682a6f8c44a7d9de276 saw valgrind issues [7205230] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:32:28 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205230/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205230/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8cf3becc8cd141b3a39db866543a389a Command failed (workunit test suites/pjd.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7205232] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:13:58 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205232/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205232/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e0cc090aad1412ab886d2a1fa95ac6a Command failed (workunit test fs/quota/quota.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' [7205236] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:16:48 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205236/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205236/ error during scrub thrashing: rank damage found: {'backtrace'} [7205238] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 01:57:42 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205238/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205238/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6294afe043db43c3819e9e0fa60f047c Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7205239] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:39:46 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205239/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205239/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19157a792b204bbc9d9fd1b1bb80f7d7 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7205242] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:45:59 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205242/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205242/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7057aedf96d547688e6eecd3632301c2 Command failed (workunit test suites/dbench.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=637824fef7a77d200c39de4a5b8cc03180ff20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7205246] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:01:38 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205246/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205246/ sentry: https://sentry.ceph.com/organizations/ceph/?query=838bf810c82a4f2dbaf6dd48b9e97c80 saw valgrind issues Dead ================================================================= [7205233] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205233/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205233/ hit max job timeout Pass ================================================================= [7205227] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:49:35 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205227/ [7205228] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:23:20 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205228/ [7205231] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:38:24 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205231/ [7205234] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 01:14:45 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205234/ [7205235] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:53:56 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205235/ [7205237] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:24:18 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205237/ [7205240] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:30:42 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205240/ [7205241] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:41:38 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205241/ [7205243] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:48:01 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205243/ [7205244] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:33:52 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205244/ [7205245] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:40:07 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205245/ [7205247] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:19:05 info: http://pulpito.ceph.com/vshankar-2023-03-13_05:27:30-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205247/