2023-03-13T07:01:48.943 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-13T07:01:48.945 INFO:teuthology.results:subset: '1007/1024' 2023-03-13T07:01:48.945 INFO:teuthology.results:seed: '2881' 2023-03-13T08:01:08.550 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-13T08:01:08.588 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-13T14:08:38.196 INFO:teuthology.results:Tests finished! gathering results... 2023-03-13T14:08:39.781 INFO:scrape:Found 23 jobs 2023-03-13T14:08:40.027 WARNING:scrape:Misunderstood line: 2023-03-13T14:08:41.156 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205313/teuthology.log 2023-03-13T14:08:45.011 WARNING:scrape:Misunderstood line: 2023-03-13T14:08:47.889 INFO:scrape:Found 8 distinct failure reasons 2023-03-13T14:08:47.890 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-03-13T14:08:47.890 INFO:scrape:2 jobs: ['7205333', '7205316'] 2023-03-13T14:08:47.890 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-03-13T14:08:47.891 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-03-13T14:08:47.891 INFO:scrape: 2023-03-13T14:08:47.891 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-03-13T14:08:47.892 INFO:scrape:2 jobs: ['7205323', '7205321'] 2023-03-13T14:08:47.892 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/3', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-03-13T14:08:47.893 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2023-03-13T14:08:47.893 INFO:scrape: 2023-03-13T14:08:47.893 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edd2928cf0d13c787a6557c78431fa56604eeffd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-03-13T14:08:47.894 INFO:scrape:1 jobs: ['7205312'] 2023-03-13T14:08:47.894 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-03-13T14:08:47.894 INFO:scrape: 2023-03-13T14:08:47.895 INFO:scrape:Dead: None 2023-03-13T14:08:47.895 INFO:scrape:3 jobs: ['7205313', '7205318', '7205328'] 2023-03-13T14:08:47.896 INFO:scrape: 2023-03-13T14:08:47.896 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edd2928cf0d13c787a6557c78431fa56604eeffd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-03-13T14:08:47.896 INFO:scrape:1 jobs: ['7205329'] 2023-03-13T14:08:47.896 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-03-13T14:08:47.897 INFO:scrape: 2023-03-13T14:08:47.897 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-03-13T14:08:47.897 INFO:scrape:1 jobs: ['7205320'] 2023-03-13T14:08:47.898 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-03-13T14:08:47.898 INFO:scrape: 2023-03-13T14:08:47.898 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edd2928cf0d13c787a6557c78431fa56604eeffd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-13T14:08:47.899 INFO:scrape:1 jobs: ['7205317'] 2023-03-13T14:08:47.899 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}'] 2023-03-13T14:08:47.899 INFO:scrape: 2023-03-13T14:08:47.899 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-03-13T14:08:47.900 INFO:scrape:1 jobs: ['7205326'] 2023-03-13T14:08:47.900 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-03-13T14:08:47.900 INFO:scrape: 2023-03-13T14:08:47.902 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/ failed: 9 dead: 3 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [7205312] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:18:36 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205312/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31a323c3ec8345a8b4c33159c72a92a9 Command failed (workunit test libcephfs/test.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edd2928cf0d13c787a6557c78431fa56604eeffd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7205316] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:00:52 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205316/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f25125b745a34e79b9e9eaed8b18b159 saw valgrind issues [7205317] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:32:41 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205317/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205317/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1bc076ee699d476ebb128fa43c105e5a Command failed (workunit test suites/pjd.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edd2928cf0d13c787a6557c78431fa56604eeffd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7205320] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:27:56 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205320/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205320/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4749894f71a441389f9129239f6397e Command failed (workunit test suites/fsstress.sh) on smithi077 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edd2928cf0d13c787a6557c78431fa56604eeffd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7205321] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:09:11 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205321/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205321/ error during scrub thrashing: rank damage found: {'backtrace'} [7205323] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:14:11 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205323/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205323/ error during scrub thrashing: rank damage found: {'backtrace'} [7205326] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:40:11 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205326/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68fff2c7170c4b44bd32f0623e942ac9 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7205329] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:57:14 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205329/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205329/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ba0238207e047dab2bbf7461b541e57 Command failed (workunit test suites/dbench.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edd2928cf0d13c787a6557c78431fa56604eeffd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7205333] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:00:27 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205333/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205333/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c00816658ae94dd8a9dfc31956e2a20c saw valgrind issues Dead ================================================================= [7205313] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205313/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205313/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7205318] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205318/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205318/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7205328] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205328/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205328/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds Pass ================================================================= [7205314] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:49:50 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205314/ [7205315] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:23:19 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205315/ [7205319] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:14:09 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205319/ [7205322] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:53:55 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205322/ [7205324] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:24:03 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205324/ [7205325] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} time: 02:15:56 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205325/ [7205327] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:30:27 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205327/ [7205330] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:49:25 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205330/ [7205331] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:32:44 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205331/ [7205332] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:40:35 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205332/ [7205334] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:19:05 info: http://pulpito.ceph.com/vshankar-2023-03-13_06:28:31-fs-wip-vshankar-testing-20230309.101043-testing-default-smithi/7205334/