2023-07-12T01:24:11.177 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-12T01:24:11.178 INFO:teuthology.results:subset: '199/512' 2023-07-12T01:24:11.178 INFO:teuthology.results:seed: '948' 2023-07-12T01:25:07.546 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-12T01:25:07.549 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-12T13:26:02.618 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-07-12T13:26:04.469 INFO:scrape:Found 22 jobs 2023-07-12T13:26:08.898 WARNING:scrape:Misunderstood line: 2023-07-12T13:26:11.471 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514/teuthology.log 2023-07-12T13:26:11.477 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514/teuthology.log 2023-07-12T13:26:19.849 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334509/teuthology.log 2023-07-12T13:26:20.230 INFO:scrape:Found 12 distinct failure reasons 2023-07-12T13:26:20.231 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2023-07-12T13:26:20.232 INFO:scrape:1 jobs: ['7334522'] 2023-07-12T13:26:20.232 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{ubuntu_20.04}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-07-12T13:26:20.232 INFO:scrape: 2023-07-12T13:26:20.233 INFO:scrape:Dead: b'2023-07-12T13:26:02.345 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-07-12T13:26:20.233 INFO:scrape:ceph version 18.0.0-4828-gb5891ed6 (b5891ed62e7aba6227aed8ab09a4dd0853a1caf2) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fb68155ccf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb6827aae19] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a8f85) [0x7fb6827aaf85] 6: (SimpleLock::put_xlock()+0x332) [0x5587f0ce3bc2] 7: (Locker::xlock_finish(std::_Rb_tree_const_iterator const&, MutationImpl*, bool*)+0x8c) [0x5587f0ccbe5c] 8: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0x407) [0x5587f0cd4a37] 9: (Locker::drop_non_rdlocks(MutationImpl*, std::set, std::allocator >*)+0x5f) [0x5587f0cd92af] 10: (Server::reply_client_request(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x497) [0x5587f0b2b137] 11: (Server::respond_to_request(boost::intrusive_ptr&, int)+0x350) [0x5587f0b2bfe0] 12: (Server::_link_local_finish(boost::intrusive_ptr&, CDentry*, CInode*, unsigned long, unsigned long, bool)+0xf5) [0x5587f0b2dee5] 13: (C_MDS_link_local_finish::finish(int)+0x35) [0x5587f0bb4585] 14: (MDSContext::complete(int)+0x5f) [0x5587f0e1279f] 15: (MDSIOContextBase::complete(int)+0x524) [0x5587f0e12f24] 16: (MDSLogContextBase::complete(int)+0x5b) [0x5587f0e132db] 17: (Finisher::finisher_thread_entry()+0x18d) [0x7fb68284e93d] 18: /lib64/libpthread.so.0(+0x81ca) [0x7fb6815521ca] 19: clone() 2023-07-12T13:26:20.233 INFO:scrape:2 jobs: ['7334513', '7334517'] 2023-07-12T13:26:20.233 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health'] 2023-07-12T13:26:20.234 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/xfstests-dev}', 'tasks/{0-client'] 2023-07-12T13:26:20.234 INFO:scrape: 2023-07-12T13:26:20.234 INFO:scrape:Failure: Command failed on smithi155 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-07-12T13:26:20.235 INFO:scrape:3 jobs: ['7334525', '7334528', '7334519'] 2023-07-12T13:26:20.235 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-07-12T13:26:20.235 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-07-12T13:26:20.236 INFO:scrape: 2023-07-12T13:26:20.236 INFO:scrape:Failure: Stale jobs detected, aborting. 2023-07-12T13:26:20.236 INFO:scrape:2 jobs: ['7334507', '7334508'] 2023-07-12T13:26:20.237 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/3', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-07-12T13:26:20.237 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-07-12T13:26:20.237 INFO:scrape: 2023-07-12T13:26:20.238 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-07-12T13:26:20.238 INFO:scrape:3 jobs: ['7334526', '7334523', '7334511'] 2023-07-12T13:26:20.238 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-07-12T13:26:20.239 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/suites/dbench}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-07-12T13:26:20.239 INFO:scrape: 2023-07-12T13:26:20.239 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6a59b159e6baf6cc2ce7ca365bda9e61847ec2a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 2023-07-12T13:26:20.239 INFO:scrape:1 jobs: ['7334518'] 2023-07-12T13:26:20.240 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-07-12T13:26:20.240 INFO:scrape: 2023-07-12T13:26:20.240 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-07-12T13:26:20.241 INFO:scrape:1 jobs: ['7334515'] 2023-07-12T13:26:20.241 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-07-12T13:26:20.241 INFO:scrape: 2023-07-12T13:26:20.241 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-07-12T13:26:20.242 INFO:scrape:1 jobs: ['7334512'] 2023-07-12T13:26:20.242 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-07-12T13:26:20.242 INFO:scrape: 2023-07-12T13:26:20.243 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6a59b159e6baf6cc2ce7ca365bda9e61847ec2a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-07-12T13:26:20.243 INFO:scrape:1 jobs: ['7334516'] 2023-07-12T13:26:20.243 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-07-12T13:26:20.244 INFO:scrape: 2023-07-12T13:26:20.244 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi082 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6a59b159e6baf6cc2ce7ca365bda9e61847ec2a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-07-12T13:26:20.244 INFO:scrape:2 jobs: ['7334524', '7334510'] 2023-07-12T13:26:20.244 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'session_timeout', 'thrashosds-health}'] 2023-07-12T13:26:20.245 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-07-12T13:26:20.245 INFO:scrape: 2023-07-12T13:26:20.245 INFO:scrape:Dead: None 2023-07-12T13:26:20.246 INFO:scrape:2 jobs: ['7334514', '7334509'] 2023-07-12T13:26:20.246 INFO:scrape: 2023-07-12T13:26:20.246 INFO:scrape:Failure: SSH connection to smithi081 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2023-07-12T13:26:20.247 INFO:scrape:1 jobs: ['7334521'] 2023-07-12T13:26:20.247 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-07-12T13:26:20.247 INFO:scrape: 2023-07-12T13:26:20.249 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/ failed: 16 dead: 2 running: 2 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7334507] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334507/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fe3b212473d46eea494784c46219603 Stale jobs detected, aborting. [7334508] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334508/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334508/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23713cdc804f49c2bfce395413841ef3 Stale jobs detected, aborting. [7334510] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:21:12 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334510/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334510/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd0f56dab67747d7b1363054a52d9466 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6a59b159e6baf6cc2ce7ca365bda9e61847ec2a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7334511] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:07:06 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334511/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334511/ error during scrub thrashing: rank damage found: {'backtrace'} [7334512] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:14:40 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334512/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334512/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3a113c4889647389860156a547fedf5 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7334515] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:59:37 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334515/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334515/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd4f31f40c494bcab0353b6e33e036a7 saw valgrind issues [7334516] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:14:12 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334516/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334516/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc73e98af6b944b1a2451813d8a0515a Command failed (workunit test fs/test_python.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6a59b159e6baf6cc2ce7ca365bda9e61847ec2a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7334518] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:15:42 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334518/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334518/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94dd1c2a67444e74b24b88b20372f484 Command failed (workunit test fs/quota/quota.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6a59b159e6baf6cc2ce7ca365bda9e61847ec2a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' [7334519] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:26 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334519/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334519/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2cf59c8eb5c14f4db088952dbef0b037 Command failed on smithi143 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7334521] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:50:53 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334521/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334521/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1c278f4c1104b22883be466ba4be3e0 SSH connection to smithi081 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [7334522] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_20.04} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:30:41 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334522/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334522/ reached maximum tries (51) after waiting for 300 seconds [7334523] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:37:30 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334523/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334523/ error during scrub thrashing: rank damage found: {'backtrace'} [7334524] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:28:13 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334524/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334524/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8cbe5f57b73847928b73477fbe51fb5f Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi082 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6a59b159e6baf6cc2ce7ca365bda9e61847ec2a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7334525] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:56 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334525/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334525/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c01b9697734417686f6a483229dcec8 Command failed on smithi155 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7334526] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:50:09 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334526/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334526/ error during scrub thrashing: rank damage found: {'backtrace'} [7334528] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:14 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334528/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33847e205fca4571a96619e277883842 Command failed on smithi102 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' Dead ================================================================= [7334509] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334509/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334509/ Error reimaging machines: Failed to power on smithi027 [7334514] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514/ Error reimaging machines: Failed to power on smithi123 Running ================================================================= [7334513] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334513/ [7334517] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334517/ Pass ================================================================= [7334520] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:18:56 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334520/ [7334527] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:53:28 info: http://pulpito.ceph.com/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334527/