2023-03-06T09:04:20.481 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T09:04:20.483 INFO:teuthology.results:subset: '802/1024' 2023-03-06T09:04:20.483 INFO:teuthology.results:seed: '7337' 2023-03-06T09:05:24.199 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-06T09:05:24.205 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-06T21:06:17.144 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-06T21:06:20.440 INFO:scrape:Found 39 jobs 2023-03-06T21:07:08.556 INFO:scrape:Found 25 distinct failure reasons 2023-03-06T21:07:08.557 INFO:scrape:Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2023-03-06T21:07:08.558 INFO:scrape:1 jobs: ['7195055'] 2023-03-06T21:07:08.558 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/alternate-pool}'] 2023-03-06T21:07:08.558 INFO:scrape: 2023-03-06T21:07:08.559 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi077 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-03-06T21:07:08.559 INFO:scrape:1 jobs: ['7195060'] 2023-03-06T21:07:08.559 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-03-06T21:07:08.559 INFO:scrape: 2023-03-06T21:07:08.560 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 2023-03-06T21:07:08.560 INFO:scrape:2 jobs: ['7195083', '7195052'] 2023-03-06T21:07:08.560 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-03-06T21:07:08.561 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-03-06T21:07:08.561 INFO:scrape: 2023-03-06T21:07:08.561 INFO:scrape:Failure: Test failure: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2023-03-06T21:07:08.562 INFO:scrape:1 jobs: ['7195071'] 2023-03-06T21:07:08.562 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}'] 2023-03-06T21:07:08.562 INFO:scrape: 2023-03-06T21:07:08.562 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-03-06T21:07:08.563 INFO:scrape:1 jobs: ['7195049'] 2023-03-06T21:07:08.563 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-03-06T21:07:08.563 INFO:scrape: 2023-03-06T21:07:08.564 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-03-06T21:07:08.564 INFO:scrape:1 jobs: ['7195074'] 2023-03-06T21:07:08.564 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2023-03-06T21:07:08.565 INFO:scrape: 2023-03-06T21:07:08.565 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi083 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-03-06T21:07:08.565 INFO:scrape:ceph version 18.0.0-2462-g802de851 (802de851643c97960b33dacc1e3ce8486e6d7d3c) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd409412cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd40b0a7e15] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a7f81) [0x7fd40b0a7f81] 6: ceph-mds(+0x2c2263) [0x5653a9bb1263] 7: (Server::handle_peer_rename_prep(boost::intrusive_ptr&)+0x1ccf) [0x5653a9b945df] 8: (MDSContext::complete(int)+0x5f) [0x5653a9e2d42f] 9: (MDSRank::_advance_queues()+0xaa) [0x5653a9ab213a] 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x1ce) [0x5653a9ab2c2e] 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5653a9ab361c] 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5653a9a9d95f] 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd40b31d398] 14: (DispatchQueue::entry()+0x50f) [0x7fd40b31a53f] 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd40b3df281] 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd4094081ca] 17: clone() 2023-03-06T21:07:08.566 INFO:scrape:2 jobs: ['7195051', '7195057'] 2023-03-06T21:07:08.566 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'objectstore-ec/bluestore-ec-root', 'osd}', 'ranks/3'] 2023-03-06T21:07:08.566 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu/{latest', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'session_timeout}', 'tasks/fsstress', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'validater/lockdep}'] 2023-03-06T21:07:08.566 INFO:scrape: 2023-03-06T21:07:08.567 INFO:scrape:Failure: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2023-03-06T21:07:08.567 INFO:scrape:2 jobs: ['7195045', '7195081'] 2023-03-06T21:07:08.567 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-03-06T21:07:08.568 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-03-06T21:07:08.568 INFO:scrape: 2023-03-06T21:07:08.569 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-06T21:07:08.569 INFO:scrape:2 jobs: ['7195080', '7195046'] 2023-03-06T21:07:08.569 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2023-03-06T21:07:08.569 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-03-06T21:07:08.570 INFO:scrape: 2023-03-06T21:07:08.570 INFO:scrape:Failure: Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2023-03-06T21:07:08.570 INFO:scrape:1 jobs: ['7195059'] 2023-03-06T21:07:08.571 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/damage}'] 2023-03-06T21:07:08.571 INFO:scrape: 2023-03-06T21:07:08.571 INFO:scrape:Failure: Test failure: test_down (tasks.cephfs.test_failover.TestClusterResize) 2023-03-06T21:07:08.572 INFO:scrape:3 jobs: ['7195077', '7195048', '7195062'] 2023-03-06T21:07:08.572 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}'] 2023-03-06T21:07:08.572 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/failover}'] 2023-03-06T21:07:08.573 INFO:scrape: 2023-03-06T21:07:08.573 INFO:scrape:Failure: Test failure: test_ephemeral_pin_grow_mds (tasks.cephfs.test_exports.TestEphemeralPins) 2023-03-06T21:07:08.573 INFO:scrape:1 jobs: ['7195063'] 2023-03-06T21:07:08.573 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/exports}'] 2023-03-06T21:07:08.574 INFO:scrape: 2023-03-06T21:07:08.574 INFO:scrape:Dead: b'2023-03-06T21:06:27.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-03-06T21:07:08.574 INFO:scrape:ceph version 18.0.0-2462-g802de851 (802de851643c97960b33dacc1e3ce8486e6d7d3c) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x580d420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x4d6c2af] 5: /usr/lib/ceph/libceph-common.so.2(+0x31d411) [0x4d6c411] 6: ceph-mds(+0x2b96f3) [0x3c16f3] 7: (Server::handle_peer_rename_prep(boost::intrusive_ptr&)+0x1cea) [0x3a8f5a] 8: (MDSContext::complete(int)+0x5f) [0x634a0f] 9: (MDSRank::_advance_queues()+0x9f) [0x2c7cef] 10: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x1c5) [0x2c8575] 11: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x2c8f4c] 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x2b353f] 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x468) [0x4fe0458] 14: (DispatchQueue::entry()+0x657) [0x4fdd9f7] 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x50a1d31] 16: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x5801609] 17: clone() 2023-03-06T21:07:08.575 INFO:scrape:1 jobs: ['7195064'] 2023-03-06T21:07:08.575 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-03-06T21:07:08.575 INFO:scrape: 2023-03-06T21:07:08.575 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-03-06T21:07:08.576 INFO:scrape:1 jobs: ['7195070'] 2023-03-06T21:07:08.576 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-03-06T21:07:08.576 INFO:scrape: 2023-03-06T21:07:08.577 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-03-06T21:07:08.577 INFO:scrape:1 jobs: ['7195058'] 2023-03-06T21:07:08.577 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-03-06T21:07:08.577 INFO:scrape: 2023-03-06T21:07:08.578 INFO:scrape:Failure: Command failed on smithi038 with status 5: 'sudo systemctl stop ceph-2b02baa0-bc03-11ed-9aea-001a4aab830c@mon.b' 2023-03-06T21:07:08.578 INFO:scrape:1 jobs: ['7195067'] 2023-03-06T21:07:08.578 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-03-06T21:07:08.579 INFO:scrape: 2023-03-06T21:07:08.579 INFO:scrape:Failure: Test failure: test_parallel_execution (tasks.cephfs.test_data_scan.TestDataScan) 2023-03-06T21:07:08.579 INFO:scrape:1 jobs: ['7195061'] 2023-03-06T21:07:08.579 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}'] 2023-03-06T21:07:08.580 INFO:scrape: 2023-03-06T21:07:08.580 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2023-03-06T21:07:08.580 INFO:scrape:1 jobs: ['7195053'] 2023-03-06T21:07:08.581 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/acls}'] 2023-03-06T21:07:08.581 INFO:scrape: 2023-03-06T21:07:08.581 INFO:scrape:Failure: Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c731000-bc03-11ed-9aea-001a4aab830c -e sha1=802de851643c97960b33dacc1e3ce8486e6d7d3c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-03-06T21:07:08.581 INFO:scrape:1 jobs: ['7195079'] 2023-03-06T21:07:08.582 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-03-06T21:07:08.582 INFO:scrape: 2023-03-06T21:07:08.582 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi151 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-03-06T21:07:08.583 INFO:scrape:1 jobs: ['7195047'] 2023-03-06T21:07:08.583 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-03-06T21:07:08.583 INFO:scrape: 2023-03-06T21:07:08.584 INFO:scrape:Failure: Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" 2023-03-06T21:07:08.584 INFO:scrape:1 jobs: ['7195056'] 2023-03-06T21:07:08.584 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-03-06T21:07:08.584 INFO:scrape: 2023-03-06T21:07:08.585 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-03-06T21:07:08.585 INFO:scrape:1 jobs: ['7195066'] 2023-03-06T21:07:08.585 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}'] 2023-03-06T21:07:08.586 INFO:scrape: 2023-03-06T21:07:08.586 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-03-06T21:07:08.586 INFO:scrape:1 jobs: ['7195054'] 2023-03-06T21:07:08.586 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}'] 2023-03-06T21:07:08.587 INFO:scrape: 2023-03-06T21:07:08.587 INFO:scrape:Crash: Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) 2023-03-06T21:07:08.587 INFO:scrape:ceph version 18.0.0-2462-g802de851 (802de851643c97960b33dacc1e3ce8486e6d7d3c) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1799612cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f179b2a7e15] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a7f81) [0x7f179b2a7f81] 6: (CDir::log_mark_dirty()+0xfa) [0x563e0a7de3ea] 7: (CDir::_omap_fetched(ceph::buffer::v15_2_0::list&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >&, bool, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, int)+0x1776) [0x563e0a7e5756] 8: (C_IO_Dir_OMAP_Fetched::finish(int)+0xbb) [0x563e0a7f7a1b] 9: (MDSContext::complete(int)+0x5f) [0x563e0a89a42f] 10: (MDSIOContextBase::complete(int)+0x524) [0x563e0a89abb4] 11: (Finisher::finisher_thread_entry()+0x18d) [0x7f179b34af2d] 12: /lib64/libpthread.so.0(+0x81ca) [0x7f17996081ca] 13: clone() 2023-03-06T21:07:08.588 INFO:scrape:1 jobs: ['7195065'] 2023-03-06T21:07:08.588 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/journal-repair}'] 2023-03-06T21:07:08.588 INFO:scrape: 2023-03-06T21:07:08.588 INFO:scrape:Failure: Test failure: test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups), test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2023-03-06T21:07:08.589 INFO:scrape:1 jobs: ['7195078'] 2023-03-06T21:07:08.589 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-03-06T21:07:08.589 INFO:scrape: 2023-03-06T21:07:08.592 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/ failed: 30 dead: 0 running: 1 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7195045] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:32:15 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195045/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195045/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cafb8e9dc6fc460f807ab5dda3eda570 Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) [7195046] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:32:41 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195046/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195046/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3b1fc9e9bda4489b2576024c932975b Command failed (workunit test suites/pjd.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7195047] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:31:14 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195047/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195047/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34d56340b1f84869afcad99b98e89201 Command failed (workunit test kernel_untar_build.sh) on smithi151 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7195048] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:30:01 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195048/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195048/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a99cc529d864fd4a7d1fd59e21c318c Test failure: test_down (tasks.cephfs.test_failover.TestClusterResize) [7195049] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:40:03 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195049/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195049/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48acf1cd859546a598690fdf3140f35f Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7195051] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:46:05 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195051/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195051/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76271a0bd47e445ba887e4edaca0e924 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi083 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7195052] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:14:50 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195052/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195052/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e4c203d295e4490baacccc60ccb17d7 Command failed (workunit test fs/quota/quota.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' [7195053] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:19:50 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195053/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195053/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a188ceb992ea43b2affe70b2b3798ddb Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [7195054] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:18:23 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195054/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195054/ error during scrub thrashing: rank damage found: {'backtrace'} [7195055] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:17:20 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195055/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195055/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fed9d99331c343ce93677272d52ab3a3 Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7195056] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:40:47 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195056/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195056/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f23ddc1fae7419a8f5b94fb37a2bbbb Command failed on smithi008 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" [7195057] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:24:37 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195057/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195057/ sentry: https://sentry.ceph.com/organizations/ceph/?query=810ac08b0da04940b5d4219a6432baa3 Command failed (workunit test suites/fsstress.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7195058] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:04:08 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195058/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195058/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aba530bb922c41b49ef8ca37012d4e13 Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7195059] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:36:50 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195059/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195059/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b322ad9275fb4a2e9d932e46a19a862b Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) [7195060] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:35:56 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195060/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195060/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f2bb03065e04fec8b008bce23cd30a3 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi077 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7195061] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:19:35 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195061/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195061/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f9f8735091d47d9b6bde1bc55032d6f Test failure: test_parallel_execution (tasks.cephfs.test_data_scan.TestDataScan) [7195062] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:29:26 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195062/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195062/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c31bd802062458a960b4e5899476fa5 Test failure: test_down (tasks.cephfs.test_failover.TestClusterResize) [7195063] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:33:53 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195063/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195063/ sentry: https://sentry.ceph.com/organizations/ceph/?query=671b1c908bed42768e8ef2d418ecabdc Test failure: test_ephemeral_pin_grow_mds (tasks.cephfs.test_exports.TestEphemeralPins) [7195065] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:19:28 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195065/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195065/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b0b22a8fadc4438b456672027b540d7 Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair) [7195066] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:29:08 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195066/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195066/ sentry: https://sentry.ceph.com/organizations/ceph/?query=869eae1830924f31b2adfa34121d6f45 Command failed (workunit test suites/fsstress.sh) on smithi047 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7195067] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:19:20 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195067/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195067/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb16fd6d06f946d3b921147ad252346f Command failed on smithi038 with status 5: 'sudo systemctl stop ceph-2b02baa0-bc03-11ed-9aea-001a4aab830c@mon.b' [7195070] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:54:26 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195070/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195070/ sentry: https://sentry.ceph.com/organizations/ceph/?query=550a937881b746409923af217d62d4aa Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7195071] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:33:37 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195071/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195071/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c39fad74e368412e9f44037bb8fac097 Test failure: test_metrics_mds_grow_and_shrink (tasks.cephfs.test_mds_metrics.TestMDSMetrics) [7195074] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:22:23 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195074/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195074/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f2a61aecc804eab80d38eb0ad10501c Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7195077] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:21:01 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195077/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195077/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b613c98adad34f0f98cc367f68469895 Test failure: test_down (tasks.cephfs.test_failover.TestClusterResize) [7195078] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:03:00 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195078/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195078/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5941b0cc09a4a3bb6f977d830765b55 Test failure: test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups), test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups) [7195079] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:18:09 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195079/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195079/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8fda274484046c7abe4b606d4749a35 Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c731000-bc03-11ed-9aea-001a4aab830c -e sha1=802de851643c97960b33dacc1e3ce8486e6d7d3c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7195080] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:46 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195080/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195080/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fa587bd45fe4ec698de0aef37ddc673 Command failed (workunit test suites/pjd.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7195081] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:32:12 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195081/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195081/ sentry: https://sentry.ceph.com/organizations/ceph/?query=977251352f6d4da588a31bed37cdaaf9 Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) [7195083] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:20:19 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195083/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195083/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14e1c8dc5f6b4b7f9083156a0aefa714 Command failed (workunit test fs/quota/quota.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=802de851643c97960b33dacc1e3ce8486e6d7d3c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' Running ================================================================= [7195064] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195064/ Pass ================================================================= [7195050] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:31:58 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195050/ [7195068] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:41 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195068/ [7195069] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:21:58 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195069/ [7195072] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:37 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195072/ [7195073] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:19:29 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195073/ [7195075] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:23:42 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195075/ [7195076] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:42:58 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195076/ [7195082] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} time: 01:03:16 info: http://pulpito.ceph.com/vshankar-2023-03-06_09:04:06-fs-wip-vshankar-testing-20230222.044949-testing-default-smithi/7195082/