2023-01-10T07:55:54.056 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-10T07:55:54.057 INFO:teuthology.results:subset: '16/32' 2023-01-10T07:55:54.057 INFO:teuthology.results:seed: '3653' 2023-01-10T10:37:09.570 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-10T10:37:09.593 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-01-10T14:12:24.793 INFO:teuthology.results:Tests finished! gathering results... 2023-01-10T14:12:30.966 INFO:scrape:Found 79 jobs 2023-01-10T14:12:48.858 INFO:scrape:Found 12 distinct failure reasons 2023-01-10T14:12:48.859 INFO:scrape:Crash: Command failed (workunit test suites/fsstress.sh) on smithi052 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2023-01-10T14:12:48.860 INFO:scrape:ceph version 18.0.0-1698-gc75a7cc3 (c75a7cc3c0325db6b834355648236f5ee8368cb5) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f40878e1420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7f4087c4728f] 5: /usr/lib/ceph/libceph-common.so.2(+0x31c3f1) [0x7f4087c473f1] 6: ceph-mds(+0x2b8fb3) [0x55b5c6feafb3] 7: (Server::handle_peer_rename_prep(boost::intrusive_ptr&)+0x1cea) [0x55b5c6fd348a] 8: (MDSContext::complete(int)+0x5f) [0x55b5c725e24f] 9: (MDSRank::_advance_queues()+0x9f) [0x55b5c6ef12cf] 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x1c5) [0x55b5c6ef1b55] 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55b5c6ef252c] 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55b5c6edc32f] 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x468) [0x7f4087eba7e8] 14: (DispatchQueue::entry()+0x657) [0x7f4087eb7d87] 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4087f7c0c1] 16: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f40878d5609] 17: clone() 2023-01-10T14:12:48.860 INFO:scrape:10 jobs: ['7132873', '7132875', '7132904', '7132937', '7132877', '7132939', '7132881', '7132879', '7132876', '7132918'] 2023-01-10T14:12:48.861 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2023-01-10T14:12:48.861 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/valgrind}'] 2023-01-10T14:12:48.862 INFO:scrape: 2023-01-10T14:12:48.862 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-01-10T14:12:48.863 INFO:scrape:1 jobs: ['7132870'] 2023-01-10T14:12:48.863 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-01-10T14:12:48.863 INFO:scrape: 2023-01-10T14:12:48.864 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-01-10T14:12:48.864 INFO:scrape:9 jobs: ['7132922', '7132914', '7132893', '7132912', '7132926', '7132883', '7132892', '7132899', '7132921'] 2023-01-10T14:12:48.865 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'mount/fuse', 'osd}'] 2023-01-10T14:12:48.865 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-01-10T14:12:48.866 INFO:scrape: 2023-01-10T14:12:48.866 INFO:scrape:Failure: Command failed on smithi085 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c75a7cc3c0325db6b834355648236f5ee8368cb5' 2023-01-10T14:12:48.866 INFO:scrape:2 jobs: ['7132897', '7132895'] 2023-01-10T14:12:48.867 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'ranks/5'] 2023-01-10T14:12:48.867 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/dbench', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'validater/lockdep}'] 2023-01-10T14:12:48.868 INFO:scrape: 2023-01-10T14:12:48.868 INFO:scrape:Failure: Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) 2023-01-10T14:12:48.868 INFO:scrape:1 jobs: ['7132894'] 2023-01-10T14:12:48.869 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/journal-repair}'] 2023-01-10T14:12:48.869 INFO:scrape: 2023-01-10T14:12:48.870 INFO:scrape:Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-01-10T14:12:48.870 INFO:scrape:1 jobs: ['7132884'] 2023-01-10T14:12:48.870 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] 2023-01-10T14:12:48.871 INFO:scrape: 2023-01-10T14:12:48.871 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi092 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-01-10T14:12:48.872 INFO:scrape:3 jobs: ['7132905', '7132909', '7132889'] 2023-01-10T14:12:48.872 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'races', 'ranks/1', 'session_timeout', 'thrashosds-health}'] 2023-01-10T14:12:48.872 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-01-10T14:12:48.873 INFO:scrape: 2023-01-10T14:12:48.873 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi083 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2023-01-10T14:12:48.874 INFO:scrape:1 jobs: ['7132907'] 2023-01-10T14:12:48.874 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-01-10T14:12:48.874 INFO:scrape: 2023-01-10T14:12:48.875 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) 2023-01-10T14:12:48.875 INFO:scrape:1 jobs: ['7132935'] 2023-01-10T14:12:48.876 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2023-01-10T14:12:48.876 INFO:scrape: 2023-01-10T14:12:48.876 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi167 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh unlocked dbench' 2023-01-10T14:12:48.877 INFO:scrape:1 jobs: ['7132885'] 2023-01-10T14:12:48.877 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}'] 2023-01-10T14:12:48.878 INFO:scrape: 2023-01-10T14:12:48.878 INFO:scrape:Failure: reached maximum tries (60) after waiting for 600 seconds 2023-01-10T14:12:48.879 INFO:scrape:2 jobs: ['7132898', '7132865'] 2023-01-10T14:12:48.879 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'objectstore/bluestore-bitmap'] 2023-01-10T14:12:48.879 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/mirror-ha/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/volumes/{overrides', 'test/misc}}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-01-10T14:12:48.880 INFO:scrape: 2023-01-10T14:12:48.880 INFO:scrape:Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2023-01-10T14:12:48.880 INFO:scrape:1 jobs: ['7132872'] 2023-01-10T14:12:48.881 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/alternate-pool}'] 2023-01-10T14:12:48.881 INFO:scrape: 2023-01-10T14:12:48.885 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/ failed: 33 dead: 0 running: 0 waiting: 0 queued: 0 passed: 46 Fail ================================================================= [7132865] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:37:04 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132865/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132865/ reached maximum tries (50) after waiting for 300 seconds [7132870] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 01:19:12 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132870/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132870/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7682e89ae854480826fdc5a521822a6 Command failed (workunit test suites/pjd.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7132872] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:22:11 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132872/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132872/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9270753e4d1f44dd8765d860cc7ef7e8 Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7132873] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:30:14 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132873/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc0b43c5a3304ed6bc69af9e4a4eafd1 Command failed (workunit test suites/fsstress.sh) on smithi052 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7132875] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:26:55 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132875/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5b2ee130cb942e39008ad1f84de453a Command failed (workunit test suites/pjd.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7132876] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:23:24 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132876/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132876/ Command failed on smithi026 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7132877] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:30:02 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132877/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132877/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e578b660a96a404da638937ec3186294 Command failed (workunit test suites/fsstress.sh) on smithi032 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7132879] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:03:16 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132879/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132879/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c893cb27b8d437e9fd55cf1f9299c13 Command failed (workunit test suites/fsstress.sh) on smithi007 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7132881] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:29:07 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132881/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132881/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1c49c7b523c494983b1cf122a2279c8 Command failed (workunit test suites/pjd.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7132883] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:46:59 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132883/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132883/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f744c98a39374fbfa0401026fcdec4bc Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132884] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:27:33 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132884/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9180f44cefcf49de95b3bee63ec47282 Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) [7132885] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 00:45:35 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132885/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132885/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6c2487461424d66b49f37f98a984bc5 Command failed (workunit test fs/fscrypt.sh) on smithi167 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh unlocked dbench' [7132889] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:22:27 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132889/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132889/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be315b7130a0428a9a7a0889bb29f7ae Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi012 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7132892] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:35:43 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132892/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132892/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fcae1b91c0c84921964183101aa46dbc Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132893] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:46:52 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132893/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132893/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dd51ffe510f47c59709cd7539654a17 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132894] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:17:01 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132894/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66b0c891599e410c8f8d533952650c04 Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) [7132895] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:18:19 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132895/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132895/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b0518ecc32f4b36bb1c267fdd3cc31a Command failed on smithi115 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c75a7cc3c0325db6b834355648236f5ee8368cb5' [7132897] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:15:11 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132897/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132897/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f846b1ebb1ea4fad88f188ca70ac9935 Command failed on smithi085 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c75a7cc3c0325db6b834355648236f5ee8368cb5' [7132898] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 01:09:55 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132898/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132898/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e4c6f76c4954c21a54aff3460ab6d2d reached maximum tries (60) after waiting for 600 seconds [7132899] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:50:32 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132899/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cbf215ccb2f2431f906185371fc087de Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132904] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 01:56:21 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132904/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132904/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2dcc6b1d3aa14960aed4c921f31a4ded Command failed (workunit test suites/fsstress.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7132905] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:59:23 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132905/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132905/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c270c1bd8cfc42f3bf6769c98c90d3f0 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi092 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7132907] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 02:13:34 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132907/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d36f827d99b14cc78218eedb87b394df Command failed (workunit test suites/fsstress.sh) on smithi083 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7132909] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:50:40 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132909/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132909/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d37562e782e4efc8dc7aa230e10d8f5 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi035 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7132912] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 02:03:51 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132912/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ebc492636d94a7c935249c4f0c6140a Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132914] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:05:21 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132914/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132914/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04cb3845753a4dde8cb8447352817272 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132918] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 01:37:10 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132918/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132918/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b4182c3f57f4edd92d737c3d3864f28 Command failed (workunit test suites/fsstress.sh) on smithi026 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7132921] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 01:45:57 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132921/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87c4554642c3408bad76abf10c664ac8 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132922] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:26:25 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132922/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132922/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cff9332db4eb4d2383e9604de7e246d9 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132926] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:51:35 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132926/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=911de82a13f74f94a3b08e852d937320 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7132935] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:14:09 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132935/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132935/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d54968e01274ae8bf132cf4aa8788d8 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) [7132937] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:28:25 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132937/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132937/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a57fc0c1e84344d68adebdadf707c051 Command failed (workunit test suites/fsstress.sh) on smithi100 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7132939] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:29:11 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132939/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132939/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa7970353b6f49caa236291fcb0d4303 Command failed (workunit test suites/pjd.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c75a7cc3c0325db6b834355648236f5ee8368cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' Pass ================================================================= [7132861] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:27:31 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132861/ [7132862] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:57 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132862/ [7132863] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} time: 00:39:38 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132863/ [7132864] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:12:37 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132864/ [7132866] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:58:16 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132866/ [7132867] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:47:21 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132867/ [7132868] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:48:33 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132868/ [7132869] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:38:50 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132869/ [7132871] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:16:05 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132871/ [7132874] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:52:02 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132874/ [7132878] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:15:46 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132878/ [7132880] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:17:29 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132880/ [7132882] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:50:42 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132882/ [7132886] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:10 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132886/ [7132887] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:22:12 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132887/ [7132888] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:22:59 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132888/ [7132890] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:08 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132890/ [7132891] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:21:12 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132891/ [7132896] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:14:52 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132896/ [7132900] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 01:46:26 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132900/ [7132901] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:32:52 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132901/ [7132902] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:43:32 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132902/ [7132903] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:47:23 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132903/ [7132906] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 01:42:21 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132906/ [7132908] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 01:41:02 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132908/ [7132910] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:38:46 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132910/ [7132911] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 02:17:12 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132911/ [7132913] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 01:28:49 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132913/ [7132915] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 01:29:30 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132915/ [7132916] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 01:24:35 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132916/ [7132917] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 01:35:44 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132917/ [7132919] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 01:20:11 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132919/ [7132920] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 01:21:02 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132920/ [7132923] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 01:20:44 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132923/ [7132924] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 01:15:47 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132924/ [7132925] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 01:23:04 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132925/ [7132927] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:08:55 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132927/ [7132928] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 01:10:35 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132928/ [7132929] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:36:54 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132929/ [7132930] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:24:12 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132930/ [7132931] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:20:24 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132931/ [7132932] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:32:29 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132932/ [7132933] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:13:40 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132933/ [7132934] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:47:56 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132934/ [7132936] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:21:37 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132936/ [7132938] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:14:13 info: http://pulpito.ceph.com/vshankar-2023-01-10_07:55:48-fs-wip-vshankar-testing-20230103.131950-testing-default-smithi/7132938/