2023-05-21T03:03:00.667 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e 2023-05-21T03:03:00.668 INFO:teuthology.results:subset: '144/512' 2023-05-21T03:03:00.669 INFO:teuthology.results:seed: '3179' 2023-05-21T03:50:46.702 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e 2023-05-21T03:50:46.756 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-21T15:20:43.730 INFO:teuthology.results:Tests finished! gathering results... 2023-05-21T15:20:48.124 INFO:scrape:Found 51 jobs 2023-05-21T15:21:03.687 WARNING:scrape:Misunderstood line: 2023-05-21T15:22:06.469 INFO:scrape:Found 29 distinct failure reasons 2023-05-21T15:22:06.539 INFO:scrape:Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) 2023-05-21T15:22:06.540 INFO:scrape:1 jobs: ['7281420'] 2023-05-21T15:22:06.540 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/openfiletable}'] 2023-05-21T15:22:06.540 INFO:scrape: 2023-05-21T15:22:06.541 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-05-21T15:22:06.541 INFO:scrape:2 jobs: ['7281380', '7281424'] 2023-05-21T15:22:06.541 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-05-21T15:22:06.542 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-05-21T15:22:06.542 INFO:scrape: 2023-05-21T15:22:06.542 INFO:scrape:Crash: Test failure: test_shutdown_killpoint_SHUTDOWN_EMPTYSUBTREES (tasks.cephfs.test_failover.TestShutdownKillpoints) 2023-05-21T15:22:06.543 INFO:scrape:ceph version 18.0.0-4092-gf704846c (f704846cae03d45131067a3ffbcec6231d97621a) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2499642520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2499f6173b] 6: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7f2499f61895] 7: (MDCache::shutdown_pass()+0x1297) [0x55a6542994f7] 8: (MDSRankDispatcher::tick()+0x32c) [0x55a65414bb2c] 9: ceph-mds(+0x1230ed) [0x55a65411e0ed] 10: (CommonSafeTimer::timer_thread()+0x146) [0x7f249a046846] 11: /usr/lib/ceph/libceph-common.so.2(+0x2472c1) [0x7f249a0472c1] 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2499694b43] 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2499726a00] 2023-05-21T15:22:06.543 INFO:scrape:1 jobs: ['7281378'] 2023-05-21T15:22:06.543 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/failover}'] 2023-05-21T15:22:06.544 INFO:scrape: 2023-05-21T15:22:06.544 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2023-05-21T15:22:06.544 INFO:scrape:1 jobs: ['7281383'] 2023-05-21T15:22:06.545 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2023-05-21T15:22:06.545 INFO:scrape: 2023-05-21T15:22:06.545 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/dbench.sh 2023-05-21T15:22:06.546 INFO:scrape:4 jobs: ['7281415', '7281399', '7281418', '7281397'] 2023-05-21T15:22:06.546 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}', 'session_timeout}'] 2023-05-21T15:22:06.546 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'frag', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/3', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/dbench', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-no-extra-options}', 'validater/lockdep}', 'validater/valgrind}', 'wsync/no}', 'wsync/yes}'] 2023-05-21T15:22:06.547 INFO:scrape: 2023-05-21T15:22:06.547 INFO:scrape:Failure: Test failure: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) 2023-05-21T15:22:06.548 INFO:scrape:2 jobs: ['7281412', '7281385'] 2023-05-21T15:22:06.548 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-05-21T15:22:06.548 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-05-21T15:22:06.549 INFO:scrape: 2023-05-21T15:22:06.549 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-05-21T15:22:06.549 INFO:scrape:1 jobs: ['7281395'] 2023-05-21T15:22:06.549 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-05-21T15:22:06.550 INFO:scrape: 2023-05-21T15:22:06.550 INFO:scrape:Failure: Command failed on smithi052 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-05-21T15:22:06.550 INFO:scrape:7 jobs: ['7281377', '7281416', '7281398', '7281410', '7281403', '7281391', '7281422'] 2023-05-21T15:22:06.551 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}'] 2023-05-21T15:22:06.551 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/yes', '3-inline/yes', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/postgres}}', '5-workunit/suites/ffsb}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distros/ubuntu_latest', 'fail_fs/no', 'frag', 'fs/multiclient/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/multi/{export-check', 'replication/always}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-from/v16.2.4', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-05-21T15:22:06.551 INFO:scrape: 2023-05-21T15:22:06.552 INFO:scrape:Failure: Cannot connect to remote host smithi092 2023-05-21T15:22:06.552 INFO:scrape:4 jobs: ['7281414', '7281389', '7281419', '7281426'] 2023-05-21T15:22:06.552 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-05-21T15:22:06.553 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-logrotate}', '2-scrub/yes', '3-inline/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/fs/misc}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/fsstress}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fail_fs/yes', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-05-21T15:22:06.553 INFO:scrape: 2023-05-21T15:22:06.553 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) 2023-05-21T15:22:06.553 INFO:scrape:1 jobs: ['7281390'] 2023-05-21T15:22:06.554 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/test_journal_migration}'] 2023-05-21T15:22:06.554 INFO:scrape: 2023-05-21T15:22:06.554 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/dbench.sh 2023-05-21T15:22:06.555 INFO:scrape:2 jobs: ['7281392', '7281421'] 2023-05-21T15:22:06.555 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_dbench_iozone}'] 2023-05-21T15:22:06.555 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}'] 2023-05-21T15:22:06.556 INFO:scrape: 2023-05-21T15:22:06.556 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/kernel_untar_build.sh 2023-05-21T15:22:06.556 INFO:scrape:1 jobs: ['7281408'] 2023-05-21T15:22:06.556 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-05-21T15:22:06.557 INFO:scrape: 2023-05-21T15:22:06.557 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/blogbench.sh 2023-05-21T15:22:06.557 INFO:scrape:1 jobs: ['7281393'] 2023-05-21T15:22:06.558 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-05-21T15:22:06.558 INFO:scrape: 2023-05-21T15:22:06.558 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-05-21T15:22:06.559 INFO:scrape:1 jobs: ['7281386'] 2023-05-21T15:22:06.559 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-05-21T15:22:06.559 INFO:scrape: 2023-05-21T15:22:06.559 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2023-05-21T15:22:06.560 INFO:scrape:1 jobs: ['7281384'] 2023-05-21T15:22:06.560 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-05-21T15:22:06.560 INFO:scrape: 2023-05-21T15:22:06.561 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-05-21T15:22:06.561 INFO:scrape:1 jobs: ['7281402'] 2023-05-21T15:22:06.561 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-05-21T15:22:06.562 INFO:scrape: 2023-05-21T15:22:06.562 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/pjd.sh 2023-05-21T15:22:06.562 INFO:scrape:2 jobs: ['7281427', '7281401'] 2023-05-21T15:22:06.562 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2023-05-21T15:22:06.563 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '2-workunit/suites/pjd}}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'msgr-failures/none', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'wsync/yes}'] 2023-05-21T15:22:06.563 INFO:scrape: 2023-05-21T15:22:06.563 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi052 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-05-21T15:22:06.564 INFO:scrape:1 jobs: ['7281411'] 2023-05-21T15:22:06.564 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-05-21T15:22:06.564 INFO:scrape: 2023-05-21T15:22:06.565 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' 2023-05-21T15:22:06.565 INFO:scrape:1 jobs: ['7281396'] 2023-05-21T15:22:06.565 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-05-21T15:22:06.565 INFO:scrape: 2023-05-21T15:22:06.566 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi151 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-05-21T15:22:06.566 INFO:scrape:1 jobs: ['7281381'] 2023-05-21T15:22:06.566 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-05-21T15:22:06.567 INFO:scrape: 2023-05-21T15:22:06.567 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi182 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none ffsb' 2023-05-21T15:22:06.567 INFO:scrape:2 jobs: ['7281406', '7281400'] 2023-05-21T15:22:06.567 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}'] 2023-05-21T15:22:06.568 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}'] 2023-05-21T15:22:06.568 INFO:scrape: 2023-05-21T15:22:06.568 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-05-21T15:22:06.569 INFO:scrape:1 jobs: ['7281425'] 2023-05-21T15:22:06.569 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-05-21T15:22:06.569 INFO:scrape: 2023-05-21T15:22:06.570 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi118 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-05-21T15:22:06.570 INFO:scrape:ceph version 18.0.0-4092-gf704846c (f704846cae03d45131067a3ffbcec6231d97621a) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb2330ffce0] 2: (CInode::get_dirfrags() const+0x26) [0x55bfc75a51f6] 3: (MDCache::advance_stray()+0x1fa) [0x55bfc7516d8a] 4: (MDLog::_start_new_segment(SegmentBoundary*)+0x465) [0x55bfc775bb65] 5: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0xba) [0x55bfc775bcfa] 6: (MDLog::submit_entry(LogEvent*, MDSLogContextBase*)+0xbf) [0x55bfc743340f] 7: (MDCache::shutdown_pass()+0xe9f) [0x55bfc756cb1f] 8: (MDSRankDispatcher::tick()+0x2c0) [0x55bfc73d8ff0] 9: (Context::complete(int)+0xd) [0x55bfc73ae99d] 10: (CommonSafeTimer::timer_thread()+0x16d) [0x7fb23443d26d] 11: (CommonSafeTimerThread::entry()+0x11) [0x7fb23443e341] 12: /lib64/libpthread.so.0(+0x81cf) [0x7fb2330f51cf] 13: clone() 2023-05-21T15:22:06.570 INFO:scrape:1 jobs: ['7281387'] 2023-05-21T15:22:06.570 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-21T15:22:06.571 INFO:scrape: 2023-05-21T15:22:06.571 INFO:scrape:Failure: Test failure: test_evicted_caps (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-05-21T15:22:06.571 INFO:scrape:1 jobs: ['7281405'] 2023-05-21T15:22:06.572 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] 2023-05-21T15:22:06.572 INFO:scrape: 2023-05-21T15:22:06.572 INFO:scrape:Failure: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2023-05-21T15:22:06.572 INFO:scrape:1 jobs: ['7281388'] 2023-05-21T15:22:06.573 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/strays}'] 2023-05-21T15:22:06.573 INFO:scrape: 2023-05-21T15:22:06.573 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-05-21T15:22:06.574 INFO:scrape:1 jobs: ['7281413'] 2023-05-21T15:22:06.574 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-05-21T15:22:06.574 INFO:scrape: 2023-05-21T15:22:06.575 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan), test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2023-05-21T15:22:06.575 INFO:scrape:1 jobs: ['7281407'] 2023-05-21T15:22:06.575 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2023-05-21T15:22:06.575 INFO:scrape: 2023-05-21T15:22:06.576 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev), test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-05-21T15:22:06.576 INFO:scrape:1 jobs: ['7281394'] 2023-05-21T15:22:06.576 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-05-21T15:22:06.577 INFO:scrape: 2023-05-21T15:22:06.577 INFO:scrape:Failure: SSH connection to smithi033 was lost: 'sudo PATH=/usr/sbin:$PATH lsof ; ps auxf' 2023-05-21T15:22:06.577 INFO:scrape:1 jobs: ['7281382'] 2023-05-21T15:22:06.578 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-05-21T15:22:06.578 INFO:scrape: 2023-05-21T15:22:06.582 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/ failed: 46 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7281377] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:15 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281377/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281377/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bd7acf94bbf4ef3800ca7500d843642 Command failed on smithi052 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7281378] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:40:38 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281378/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eaa5523b278b4a839e899d4aa93c7f2d Test failure: test_shutdown_killpoint_SHUTDOWN_EMPTYSUBTREES (tasks.cephfs.test_failover.TestShutdownKillpoints) [7281380] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:11:50 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281380/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bc5c2f310ef48be9b0c2dd08ecad7e6 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7281381] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:26:22 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281381/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c99047a4cd34058ad097b757102712d Command failed (workunit test kernel_untar_build.sh) on smithi151 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7281382] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 11:01:49 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281382/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281382/ SSH connection to smithi033 was lost: 'sudo PATH=/usr/sbin:$PATH lsof ; ps auxf' [7281383] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:14:06 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281383/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2962c71eef01489fb26b54dab7457434 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7281384] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:32:51 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281384/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281384/ reached maximum tries (50) after waiting for 300 seconds [7281385] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:18:53 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281385/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba0d6621898f4563bfdff40b5954b5d6 Test failure: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) [7281386] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:58:04 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281386/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71924d71a0d3462e919bdc029ca254e3 saw valgrind issues [7281387] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:15:40 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281387/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281387/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8dbda4b8423342a68afc9f870356d91d Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi118 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7281388] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:20:13 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281388/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92179973a9cd4e5794060e4bb1015f8b Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) [7281389] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 10:52:26 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281389/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2150c9f60ee64c75bf1563874e49da8b Cannot connect to remote host smithi107 [7281390] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 03:20:52 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281390/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d28440280934ecdb7c7f5ab6ea4092a Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) [7281391] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 10:51:49 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281391/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281391/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4003659f66164150bc8620f35f4aecad SSH connection to smithi072 was lost: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" [7281392] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:14:34 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281392/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281392/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4fb36184edf407082af4ab6df2fe16d Command failed (workunit test suites/dbench.sh) on smithi062 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7281393] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 10:53:28 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281393/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281393/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca8c0e31a602494f96dc7843fd67529e Command failed (workunit test suites/blogbench.sh) on smithi039 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' [7281394] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 10:49:03 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281394/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0e2b88226814333a7a7e3270114164a Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev), test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7281395] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:26:34 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281395/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4552201484534dd195a9a87515cbca22 Command failed (workunit test suites/fsstress.sh) on smithi089 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7281396] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:12:58 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281396/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281396/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a22c65a59de048c89b1c4307879dc631 Command failed (workunit test fs/quota/quota.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' [7281397] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 03:27:23 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281397/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe6f1b3d28724552af52c26f9f1bfd6c Command failed (workunit test suites/dbench.sh) on smithi070 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7281398] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:49 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281398/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7beedab2c1174907bbabf2afa865558d Command failed on smithi117 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7281399] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 03:16:20 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281399/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b1408b6b5a549f084f27cfaf76c5596 Command failed (workunit test suites/dbench.sh) on smithi135 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7281400] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 00:14:39 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281400/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281400/ sentry: https://sentry.ceph.com/organizations/ceph/?query=30916aba2b074b3cb4e3402a297feffc Command failed (workunit test fs/fscrypt.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench' [7281401] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 06:35:59 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281401/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef7816b4e48449c5a8f0237ca6f6d65c Command failed (workunit test suites/pjd.sh) on smithi032 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7281402] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:30:57 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281402/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94e816b32adf4e6c8fe349c65eeb0d33 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7281403] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:06 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281403/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43a57f6af355417c8b9e228c879854a9 Command failed on smithi177 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7281405] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:24:58 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281405/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a356f7177ff471f9f3469fc0c12fe38 Test failure: test_evicted_caps (tasks.cephfs.test_client_recovery.TestClientRecovery) [7281406] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 00:14:43 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281406/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281406/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3966ecab47c9453da930db0165f5133f Command failed (workunit test fs/fscrypt.sh) on smithi182 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none ffsb' [7281407] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 10:38:45 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281407/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d99e356148c141a194661107d9616356 Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan), test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [7281408] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 04:25:01 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281408/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=803f4cbda5be4ba0ab26f8962c56c43f Command failed (workunit test kernel_untar_build.sh) on smithi026 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7281410] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 11:41:28 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281410/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281410/ Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4904f7b2-f78a-11ed-9b02-001a4aab830c --force' [7281411] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:12:34 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281411/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=74efb8517bfd4ff6abe1495a9727474d Command failed (workunit test libcephfs/test.sh) on smithi052 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7281412] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:19:28 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281412/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281412/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71ac1d7397a444b3a34983028feab073 Test failure: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) [7281413] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:41:43 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281413/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15176332528d4d3ca78719534fafd01c Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7281414] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 10:31:19 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281414/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281414/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb53839f761f4c4e93c830bd535b21a0 Cannot connect to remote host smithi092 [7281415] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 03:36:11 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281415/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281415/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de014f0577284b8a8f720a962645c780 Command failed (workunit test suites/dbench.sh) on smithi002 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7281416] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:14:43 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281416/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281416/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20afa2668a2249bd87a289a7a3271c40 Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:f704846cae03d45131067a3ffbcec6231d97621a pull' [7281418] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 03:29:43 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281418/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65c83611445d47d88e821b0ee0d5a694 Command failed (workunit test suites/dbench.sh) on smithi016 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7281419] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 10:23:58 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281419/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281419/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1a26eec2f03477d9d8e8fd56928f0f5 Cannot connect to remote host smithi100 [7281420] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:19:04 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281420/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281420/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9487085a1ffd4873b52197800fe0f11b Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) [7281421] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:19:08 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281421/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281421/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9b2f93a7f0e479f8c06ac5e7398ca20 Command failed (workunit test suites/dbench.sh) on smithi112 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7281422] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:00 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281422/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281422/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9a58815be644dad84ba9cf6c7b9c37d Command failed on smithi159 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7281424] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:17:12 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281424/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281424/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a3e10fc6254493486cfb18d6e4123b1 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7281425] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:12:09 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281425/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281425/ sentry: https://sentry.ceph.com/organizations/ceph/?query=125d938de19c40e9ac9f667ff18a2680 Command failed (workunit test fs/test_python.sh) on smithi131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7281426] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 10:13:55 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281426/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281426/ Cannot connect to remote host smithi027 [7281427] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 06:45:48 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281427/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281427/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ebd38dbea5740f28832444631fad6f8 Command failed (workunit test suites/pjd.sh) on smithi053 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de3630ce7f2548b9c1af22318e8b8acbdfa6ee83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' Pass ================================================================= [7281379] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:17 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281379/ [7281404] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:38:05 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281404/ [7281409] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 01:06:14 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281409/ [7281417] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:23:08 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281417/ [7281423] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:30:50 info: http://pulpito.ceph.com/pdonnell-2023-05-21_03:01:34-fs-wip-pdonnell-testing-20230519.164114-distro-default-smithi/7281423/