Found 202 jobs Misunderstood line: Misunderstood line: Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393458/teuthology.log Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-11_09:35:04-fs-wip-vshankar-testing-20230911.055957-testing-default-smithi/7393458/teuthology.log Found 23 distinct failure reasons Failure: Command failed (workunit test fs/test_python.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 1 jobs: ['7393538'] suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] Failure: Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6ebdfb72-5098-11ee-9ab7-7b867c8bd7da -e sha1=02cf814a2fe196b91c36545e55f7fa303a1a5367 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2 jobs: ['7393451', '7393527'] suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] Failure: Command failed on smithi151 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 3 jobs: ['7393452', '7393552', '7393434'] suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'tasks/mdtest}', 'validater/valgrind}'] Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 1 jobs: ['7393512'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] Failure: error during scrub thrashing: rank damage found: {'backtrace'} 1 jobs: ['7393549'] suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] Failure: No module named 'tasks' 3 jobs: ['7393483', '7393471', '7393484'] suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/yes', '3-inline/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/suites/iozone}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fail_fs/yes', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/5', 'ranks/multi/{export-check', 'replication/always}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/fsstress', 'tasks/{0-from/pacific', 'tasks/{0-subvolume/{with-quota}', 'validater/lockdep}', 'wsync/no}'] Crash: Test failure: test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover), test_connect_bootstrapping (tasks.cephfs.test_failover.TestFailover) ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9265bb6420] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x55b1289ba973] 3: (Beacon::notify_health(MDSRank const*)+0x2321) [0x55b12866d6b1] 4: (Context::complete(int)+0xd) [0x55b12861dadd] 5: (CommonSafeTimer::timer_thread()+0x18e) [0x7f926601b86e] 6: (CommonSafeTimerThread::entry()+0x11) [0x7f926601cde1] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f9265baa609] 4 jobs: ['7393453', '7393500', '7393443', '7393529'] suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] suites union: ['1-ceph', '2-logrotate}', '2-workunit/ffsb}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/pjd}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/multifs/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'session_timeout', 'tasks/failover}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] Dead: b'2023-09-12T01:53:55.330 DEBUG:teuthology.exit:Finished running handlers' ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x560f0fef0d53] 2: (KernelDevice::_aio_thread()+0x1686) [0x560f10b30136] 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x560f10b35b51] 4: /lib64/libpthread.so.0(+0x81ca) [0x7fd3970081ca] 1 jobs: ['7393556'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 1 jobs: ['7393390'] suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] Failure: "2023-09-11T10:14:37.456922+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 5 jobs: ['7393382', '7393460', '7393514', '7393533', '7393469'] suites intersection: ['1-ceph', '2-logrotate}'] suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'session_timeout', 'supported-random-distros$/{centos_latest}', 'tasks/forward-scrub}', 'tasks/mirror}', 'tasks/scrub}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] Dead: b'2023-09-11T23:42:32.704 DEBUG:teuthology.exit:Finished running handlers' 4 jobs: ['7393467', '7393394', '7393385', '7393567'] suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] suites union: ['1-ceph', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fail_fs/no', 'fail_fs/yes', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/failover}', 'tasks/xfstests-dev}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4'] Crash: Test failure: test_extra_data_pool_rebuild_few_files (tasks.cephfs.test_data_scan.TestDataScan) ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fb9c6c12cf0] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x5596d68824c3] 3: (Beacon::notify_health(MDSRank const*)+0x21cf) [0x5596d6527daf] 4: (MDSRank::damaged()+0x53) [0x5596d64de2e3] 5: (MDSTable::load_2(int, ceph::buffer::v15_2_0::list&, Context*)+0x351) [0x5596d6814701] 6: (MDSContext::complete(int)+0x5f) [0x5596d68717cf] 7: (MDSIOContextBase::complete(int)+0x574) [0x5596d6871fb4] 8: (Finisher::finisher_thread_entry()+0x19d) [0x7fb9c8953e4d] 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb9c6c081ca] 2 jobs: ['7393450', '7393468'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}', 'tasks/journal-repair}'] Failure: The following counters failed to be set on mds daemons: {'mds_server.req_rmsnap_latency.avgcount', 'mds_server.req_mksnap_latency.avgcount', 'mds.root_rsnaps'} 5 jobs: ['7393531', '7393404', '7393442', '7393423', '7393572'] suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'session_timeout}', 'standby-replay', 'wsync/no}'] suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] Crash: Command failed (workunit test fs/misc/trivial_sync.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/trivial_sync.sh' ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fcea419ace0] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x56202c9154c3] 3: (Beacon::notify_health(MDSRank const*)+0x21cf) [0x56202c5badaf] 4: (Context::complete(int)+0xd) [0x56202c5676cd] 5: (CommonSafeTimer::timer_thread()+0x181) [0x7fcea54efc91] 6: (CommonSafeTimerThread::entry()+0x11) [0x7fcea54f0fa1] 7: /lib64/libpthread.so.0(+0x81cf) [0x7fcea41901cf] 2 jobs: ['7393493', '7393523'] suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/stock/{k-stock', 'rhel_8}'] suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/failover}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] Failure: Command failed on smithi130 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6013-g02cf814a-1focal ceph-mds=18.0.0-6013-g02cf814a-1focal ceph-common=18.0.0-6013-g02cf814a-1focal ceph-fuse=18.0.0-6013-g02cf814a-1focal ceph-test=18.0.0-6013-g02cf814a-1focal radosgw=18.0.0-6013-g02cf814a-1focal python-ceph=18.0.0-6013-g02cf814a-1focal libcephfs1=18.0.0-6013-g02cf814a-1focal libcephfs-java=18.0.0-6013-g02cf814a-1focal libcephfs-jni=18.0.0-6013-g02cf814a-1focal librados2=18.0.0-6013-g02cf814a-1focal librbd1=18.0.0-6013-g02cf814a-1focal rbd-fuse=18.0.0-6013-g02cf814a-1focal python3-cephfs=18.0.0-6013-g02cf814a-1focal cephfs-shell=18.0.0-6013-g02cf814a-1focal cephfs-top=18.0.0-6013-g02cf814a-1focal cephfs-mirror=18.0.0-6013-g02cf814a-1focal' 1 jobs: ['7393474'] suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] Timeout 3h running clone.client.0/qa/workunits/suites/iozone.sh 1 jobs: ['7393555'] suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] Failure: Command failed on smithi184 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ecd8e44-50b1-11ee-9ab7-7b867c8bd7da -- ceph mon dump -f json' 1 jobs: ['7393546'] suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 1 jobs: ['7393503'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] Crash: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7f95f6925afb] 2: (CDentry::check_corruption(bool)+0x6fc) [0x5630e797cc3c] 3: ceph-mds(+0x47d7a2) [0x5630e79967a2] 4: (CDir::_omap_commit(int)+0x30a) [0x5630e7996d5a] 5: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x12d) [0x5630e7997f7d] 6: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x257e) [0x5630e773fabe] 7: (MDLog::try_expire(LogSegment*, int)+0x68) [0x5630e7a6f828] 8: (MDLog::trim_all()+0x6cd) [0x5630e7a70fcd] 9: (C_Flush_Journal::trim_mdlog()+0x14b) [0x5630e770f13b] 10: (Context::complete(int)+0xd) [0x5630e76cbadd] 11: (MDSContext::complete(int)+0x5f) [0x5630e7a581af] 12: (C_IO_Wrapper::finish(int)+0x12) [0x5630e7a59062] 13: (MDSContext::complete(int)+0x5f) [0x5630e7a581af] 14: (MDSIOContextBase::complete(int)+0x55c) [0x5630e7a5896c] 15: (Finisher::finisher_thread_entry()+0x195) [0x7f95f69cdde5] 16: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f95f73b0609] 1 jobs: ['7393447'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/damage}'] Dead: None 1 jobs: ['7393458'] Failure: Command failed (workunit test fs/quota/quota.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=02cf814a2fe196b91c36545e55f7fa303a1a5367 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 2 jobs: ['7393396', '7393564'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] Failure: Test failure: test_ephemeral_randomness (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_randomness (tasks.cephfs.test_exports.TestEphemeralPins) 1 jobs: ['7393457'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}'] Crash: Command failed on smithi094 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0' ceph version 18.0.0-6013-g02cf814a (02cf814a2fe196b91c36545e55f7fa303a1a5367) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f708e454db0] 2: (MDLog::get_replay_estimated_complete_time()+0x33) [0x55ed75cfd5f3] 3: (Beacon::notify_health(MDSRank const*)+0x245a) [0x55ed75a87a1a] 4: ceph-mds(+0x123d9d) [0x55ed75a1dd9d] 5: (CommonSafeTimer::timer_thread()+0x15e) [0x7f708ee54dce] 6: /usr/lib64/ceph/libceph-common.so.2(+0x255691) [0x7f708ee55691] 7: /lib64/libc.so.6(+0x9f802) [0x7f708e49f802] 8: /lib64/libc.so.6(+0x3f450) [0x7f708e43f450] 1 jobs: ['7393417'] suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']