2023-03-14T10:54:40.393 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-14T10:54:40.395 INFO:teuthology.results:subset: '111/120000' 2023-03-14T10:54:40.395 INFO:teuthology.results:seed: '5279' 2023-03-14T11:10:50.128 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-14T11:10:50.154 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-14T13:25:00.940 INFO:teuthology.results:Tests finished! gathering results... 2023-03-14T13:25:02.106 INFO:scrape:Found 15 jobs 2023-03-14T13:25:02.296 WARNING:scrape:Misunderstood line: 2023-03-14T13:25:06.916 INFO:scrape:Found 5 distinct failure reasons 2023-03-14T13:25:06.917 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-03-14T13:25:06.918 INFO:scrape:1 jobs: ['7206411'] 2023-03-14T13:25:06.918 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-03-14T13:25:06.918 INFO:scrape: 2023-03-14T13:25:06.919 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-03-14T13:25:06.919 INFO:scrape:1 jobs: ['7206404'] 2023-03-14T13:25:06.919 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-03-14T13:25:06.920 INFO:scrape: 2023-03-14T13:25:06.920 INFO:scrape:Failure: Test failure: test_dump_dir (tasks.cephfs.test_misc.TestMisc) 2023-03-14T13:25:06.920 INFO:scrape:2 jobs: ['7206413', '7206403'] 2023-03-14T13:25:06.920 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-03-14T13:25:06.921 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-03-14T13:25:06.921 INFO:scrape: 2023-03-14T13:25:06.921 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d26e5144f0edfa28a654282890c41d63c46b17ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-14T13:25:06.922 INFO:scrape:1 jobs: ['7206408'] 2023-03-14T13:25:06.923 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}'] 2023-03-14T13:25:06.923 INFO:scrape: 2023-03-14T13:25:06.924 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2023-03-14T13:25:06.924 INFO:scrape:1 jobs: ['7206412'] 2023-03-14T13:25:06.924 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2023-03-14T13:25:06.925 INFO:scrape: 2023-03-14T13:25:06.926 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7206403] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:24:40 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206403/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41c5eb0deac445658714a5ef5d788f11 Test failure: test_dump_dir (tasks.cephfs.test_misc.TestMisc) [7206404] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:40:16 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206404/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d76aac3242d41d5aebfb41363bf3584 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7206408] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:40:39 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206408/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a987a452e44429db52049f91aaf309c Command failed (workunit test suites/pjd.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d26e5144f0edfa28a654282890c41d63c46b17ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7206411] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 02:05:54 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206411/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02132e1966b64ee8b52535238224c0cc saw valgrind issues [7206412] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:28:56 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206412/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206412/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7206413] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:25:59 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206413/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=511dd89c73034f6d8b2a1ca0808b3f09 Test failure: test_dump_dir (tasks.cephfs.test_misc.TestMisc) Pass ================================================================= [7206400] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:18:14 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206400/ [7206401] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:20:43 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206401/ [7206402] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} time: 02:14:06 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206402/ [7206405] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 01:02:18 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206405/ [7206406] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:19:01 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206406/ [7206407] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:18:17 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206407/ [7206409] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:24:14 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206409/ [7206410] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:13 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206410/ [7206414] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:18:44 info: http://pulpito.ceph.com/vshankar-2023-03-14_10:53:00-fs-main-testing-default-smithi/7206414/