2023-05-03T05:33:23.473 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-03T05:33:23.476 INFO:teuthology.results:subset: '663/1024' 2023-05-03T05:33:23.476 INFO:teuthology.results:seed: '244' 2023-05-03T09:25:41.661 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-03T09:25:41.701 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-03T21:20:47.383 INFO:teuthology.results:Tests finished! gathering results... 2023-05-03T21:21:05.135 INFO:scrape:Found 204 jobs 2023-05-03T21:21:20.356 WARNING:scrape:Misunderstood line: 2023-05-03T21:21:33.675 WARNING:scrape:Misunderstood line: 2023-05-03T21:21:59.338 INFO:scrape:Found 21 distinct failure reasons 2023-05-03T21:21:59.340 INFO:scrape:Failure: Command failed on smithi177 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2023-05-03T21:21:59.340 INFO:scrape:36 jobs 2023-05-03T21:21:59.341 INFO:scrape:suites intersection: [] 2023-05-03T21:21:59.341 INFO:scrape:suites union: ['1-ceph', '1-client-sanity}}}', '1-mount/mount/fuse', '1-mount/mount/kclient/{mount-syntax/{v1}', '1-tests/blogbench}}}', '1-tests/dbench}}}', '1-tests/fsstress}}}', '1-tests/iozone}}}', '1-tests/kernel_untar_build}}}', '2-logrotate}', '2-workload/new_ops/{0-clients/fuse-upgrade', '2-workload/new_ops/{0-clients/kclient', '2-workload/stress_tests/{0-client-upgrade', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/fs/snaps}}', '2-workunit/iozone}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', 'bluestore-bitmap', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1-mds-1-client', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'debug', 'distro/{centos_8}', 'distro/{centos_latest}', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides', 'overrides/whitelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/acls}', 'tasks/cap-flush}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/dbench', 'tasks/forward-scrub}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-pjd}', 'tasks/fsstress', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/mds-flush}', 'tasks/mds_creation_retry}', 'tasks/mgr-osd-full}', 'tasks/mirror}}', 'tasks/quota}', 'tasks/recovery-fs}', 'tasks/trim-i24137}', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{0-from/nautilus', 'tasks/{0-from/pacific', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}'] 2023-05-03T21:21:59.342 INFO:scrape: 2023-05-03T21:21:59.342 INFO:scrape:Crash: Test failure: test_volume_ls (tasks.cephfs.test_volumes.TestVolumes), test_volume_ls (tasks.cephfs.test_volumes.TestVolumes) 2023-05-03T21:21:59.342 INFO:scrape:ceph version 18.0.0-3658-g5efb30c6 (5efb30c69d6138881536a78b7fbd0dbc6d59d97d) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb843e7f420] 2: gsignal() 3: abort() 4: /usr/lib/ceph/libceph-common.so.2(+0x3692b7) [0x7fb8433692b7] 5: (__cxxabiv1::__terminate(void (*)())+0xc) [0x7fb843a1ea2c] 6: /usr/lib/ceph/libceph-common.so.2(+0xa1ea97) [0x7fb843a1ea97] 7: /usr/lib/ceph/libceph-common.so.2(+0xa1ebf9) [0x7fb843a1ebf9] 8: (std::__throw_out_of_range(char const*)+0x41) [0x7fb84336ad42] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3bc) [0x557022cdbaac] 10: (MDSMonitor::tick()+0x18a) [0x557022cea41a] 11: (Monitor::tick()+0xc0) [0x557022aa1840] 12: (Context::complete(int)+0xd) [0x557022ac19dd] 13: (CommonSafeTimer::timer_thread()+0x12c) [0x7fb84341006c] 14: (CommonSafeTimerThread::entry()+0x11) [0x7fb843411101] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb843e73609] 16: clone() 2023-05-03T21:21:59.343 INFO:scrape:8 jobs: ['7260563', '7260587', '7260632', '7260520', '7260647', '7260539', '7260672', '7260583'] 2023-05-03T21:21:59.343 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health'] 2023-05-03T21:21:59.343 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'fs/shell/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/auto-repair}', 'tasks/cephfs-shell}', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/volumes/{overrides', 'test/basic}}', 'test/clone}}', 'test/misc}}', 'test/snapshot}}'] 2023-05-03T21:21:59.344 INFO:scrape: 2023-05-03T21:21:59.344 INFO:scrape:Crash: Found coredumps on ubuntu@smithi027.front.sepia.ceph.com 2023-05-03T21:21:59.344 INFO:scrape:ceph version 18.0.0-3658-g5efb30c6 (5efb30c69d6138881536a78b7fbd0dbc6d59d97d) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fafef9bdce0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7fafeefd409b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7fafeefda53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7fafeefda597] 7: /lib64/libstdc++.so.6(+0x967f8) [0x7fafeefda7f8] 8: /lib64/libstdc++.so.6(+0x9204b) [0x7fafeefd604b] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3cc) [0x55f6e4d9ea8c] 10: (MDSMonitor::tick()+0x18a) [0x55f6e4dad3fa] 11: (Monitor::tick()+0xc0) [0x55f6e4b59b30] 12: (Context::complete(int)+0xd) [0x55f6e4b7d0ad] 13: (CommonSafeTimer::timer_thread()+0x127) [0x7faff2294307] 14: (CommonSafeTimerThread::entry()+0x11) [0x7faff2295431] 15: /lib64/libpthread.so.0(+0x81cf) [0x7fafef9b31cf] 16: clone() 2023-05-03T21:21:59.345 INFO:scrape:17 jobs: ['7260688', '7260626', '7260561', '7260607', '7260530', '7260495', '7260536', '7260515', '7260597', '7260511', '7260665', '7260578', '7260650', '7260644', '7260627', '7260659', '7260630'] 2023-05-03T21:21:59.345 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-05-03T21:21:59.345 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_trivial_sync}}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'rhel_8}', 'session_timeout', 'supported-random-distros$/{rhel_8}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-limits}', 'tasks/client}', 'tasks/failover}', 'tasks/mds-full}', 'tasks/metrics}', 'tasks/mirror}', 'tasks/multifs-auth}', 'tasks/scrub}', 'tasks/sessionmap}', 'tasks/snap_schedule_snapdir}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mds', 'test/snapshot}}', 'thrashosds-health}'] 2023-05-03T21:21:59.346 INFO:scrape: 2023-05-03T21:21:59.346 INFO:scrape:Failure: Command failed on smithi150 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c34a39c8-e98a-11ed-9b01-001a4aab830c -- ceph mon dump -f json' 2023-05-03T21:21:59.346 INFO:scrape:1 jobs: ['7260599'] 2023-05-03T21:21:59.347 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-05-03T21:21:59.347 INFO:scrape: 2023-05-03T21:21:59.347 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-05-03T21:21:59.347 INFO:scrape:2 jobs: ['7260498', '7260531'] 2023-05-03T21:21:59.348 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'n/3', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-05-03T21:21:59.348 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}', 'wsync/yes}'] 2023-05-03T21:21:59.349 INFO:scrape: 2023-05-03T21:21:59.349 INFO:scrape:Failure: Test failure: test_standby_replay_singleton (tasks.cephfs.test_failover.TestStandbyReplay) 2023-05-03T21:21:59.349 INFO:scrape:1 jobs: ['7260527'] 2023-05-03T21:21:59.349 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}'] 2023-05-03T21:21:59.350 INFO:scrape: 2023-05-03T21:21:59.350 INFO:scrape:Dead: b'2023-05-03T21:16:32.255 DEBUG:teuthology.exit:Finished running handlers' 2023-05-03T21:21:59.350 INFO:scrape:4 jobs: ['7260673', '7260555', '7260540', '7260628'] 2023-05-03T21:21:59.351 INFO:scrape:suites intersection: ['2-logrotate}'] 2023-05-03T21:21:59.351 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/postgres}}', 'begin/{0-install', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'frag', 'fs/functional/{begin/{0-install', 'fs/mirror-ha/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'ranks/1', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{ubuntu_latest}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'workloads/cephfs-mirror-ha-workunit}', 'wsync/no}'] 2023-05-03T21:21:59.351 INFO:scrape: 2023-05-03T21:21:59.351 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed. 2023-05-03T21:21:59.352 INFO:scrape:1 jobs: ['7260538'] 2023-05-03T21:21:59.352 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-05-03T21:21:59.352 INFO:scrape: 2023-05-03T21:21:59.353 INFO:scrape:Crash: Test failure: test_scrub_non_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2), test_scrub_non_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) 2023-05-03T21:21:59.353 INFO:scrape:ceph version 18.0.0-3658-g5efb30c6 (5efb30c69d6138881536a78b7fbd0dbc6d59d97d) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fbc00642520] 2: pthread_kill() 3: raise() 4: abort() 5: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa2bbe) [0x7fbc00aa2bbe] 6: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae24c) [0x7fbc00aae24c] 7: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae2b7) [0x7fbc00aae2b7] 8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae518) [0x7fbc00aae518] 9: (std::__throw_out_of_range(char const*)+0x40) [0x7fbc00aa54e6] 10: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0x3ac) [0x5586393abb7c] 11: (MDSMonitor::tick()+0x182) [0x5586393b9722] 12: (Monitor::tick()+0xb0) [0x55863922f540] 13: ceph-mon(+0xb0816d) [0x5586391d916d] 14: (CommonSafeTimer::timer_thread()+0x116) [0x7fbc01445726] 15: /usr/lib/ceph/libceph-common.so.2(+0x245f71) [0x7fbc01445f71] 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fbc00694b43] 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fbc00726a00] 2023-05-03T21:21:59.353 INFO:scrape:19 jobs: ['7260490', '7260642', '7260509', '7260606', '7260634', '7260535', '7260594', '7260639', '7260581', '7260569', '7260617', '7260595', '7260624', '7260507', '7260502', '7260526', '7260500', '7260546', '7260552'] 2023-05-03T21:21:59.354 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-05-03T21:21:59.354 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/ffsb}}', '2-workunit/iozone}}', '2-workunit/suites/pjd}}', 'bluestore-bitmap', 'cluster/{1-node}', 'clusters/1-mds-1-client', 'clusters/1-mds-2-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/ignorelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'pg-warn}', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'supported-random-distros$/{ubuntu_latest}', 'tasks/admin}', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-recovery}', 'tasks/exports}', 'tasks/failover}', 'tasks/fscrypt-common}', 'tasks/fscrypt-iozone}', 'tasks/fstop}', 'tasks/journal-repair}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/mdtest}', 'tasks/multimds_misc}', 'tasks/snapshots}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'test/misc}}', 'thrashosds-health}'] 2023-05-03T21:21:59.354 INFO:scrape: 2023-05-03T21:21:59.355 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh 2023-05-03T21:21:59.355 INFO:scrape:1 jobs: ['7260572'] 2023-05-03T21:21:59.355 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-05-03T21:21:59.356 INFO:scrape: 2023-05-03T21:21:59.356 INFO:scrape:Failure: Command failed on smithi082 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-05-03T21:21:59.356 INFO:scrape:6 jobs: ['7260494', '7260616', '7260560', '7260664', '7260579', '7260532'] 2023-05-03T21:21:59.357 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}'] 2023-05-03T21:21:59.357 INFO:scrape:suites union: ['1-ceph', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fail_fs/yes', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn', 'rhel_8}', 'roles', 'session_timeout', 'syntax}', 'tasks/asok_dump_tree}', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-from/pacific', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-05-03T21:21:59.357 INFO:scrape: 2023-05-03T21:21:59.358 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi012 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-05-03T21:21:59.358 INFO:scrape:1 jobs: ['7260525'] 2023-05-03T21:21:59.359 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-05-03T21:21:59.359 INFO:scrape: 2023-05-03T21:21:59.360 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-05-03T21:21:59.360 INFO:scrape:2 jobs: ['7260533', '7260692'] 2023-05-03T21:21:59.360 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-05-03T21:21:59.361 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-05-03T21:21:59.361 INFO:scrape: 2023-05-03T21:21:59.361 INFO:scrape:Failure: Test failure: test_file_filesystem_sync_crash (tasks.cephfs.test_misc.TestMisc), test_file_filesystem_sync_crash (tasks.cephfs.test_misc.TestMisc) 2023-05-03T21:21:59.362 INFO:scrape:1 jobs: ['7260629'] 2023-05-03T21:21:59.362 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-05-03T21:21:59.362 INFO:scrape: 2023-05-03T21:21:59.362 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-05-03T21:21:59.363 INFO:scrape:1 jobs: ['7260577'] 2023-05-03T21:21:59.363 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-05-03T21:21:59.363 INFO:scrape: 2023-05-03T21:21:59.364 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2023-05-03T21:21:59.364 INFO:scrape:1 jobs: ['7260663'] 2023-05-03T21:21:59.364 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-05-03T21:21:59.364 INFO:scrape: 2023-05-03T21:21:59.365 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-05-03T21:21:59.365 INFO:scrape:1 jobs: ['7260678'] 2023-05-03T21:21:59.365 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}'] 2023-05-03T21:21:59.366 INFO:scrape: 2023-05-03T21:21:59.366 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi135 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-05-03T21:21:59.366 INFO:scrape:1 jobs: ['7260677'] 2023-05-03T21:21:59.367 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-03T21:21:59.367 INFO:scrape: 2023-05-03T21:21:59.367 INFO:scrape:Failure: Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan) 2023-05-03T21:21:59.367 INFO:scrape:1 jobs: ['7260614'] 2023-05-03T21:21:59.368 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}'] 2023-05-03T21:21:59.368 INFO:scrape: 2023-05-03T21:21:59.368 INFO:scrape:Failure: Command failed on smithi116 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5efb30c69d6138881536a78b7fbd0dbc6d59d97d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea14d9b6-e98d-11ed-9b01-001a4aab830c -- ceph mon dump -f json' 2023-05-03T21:21:59.369 INFO:scrape:1 jobs: ['7260623'] 2023-05-03T21:21:59.369 INFO:scrape:suites: ['1-mount', '2-workload/dbench', 'distro/single-container-host}', 'fs/cephadm/multivolume/{0-start'] 2023-05-03T21:21:59.369 INFO:scrape: 2023-05-03T21:21:59.369 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-05-03T21:21:59.370 INFO:scrape:1 jobs: ['7260670'] 2023-05-03T21:21:59.370 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2023-05-03T21:21:59.370 INFO:scrape: 2023-05-03T21:21:59.384 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/ failed: 103 dead: 4 running: 0 waiting: 0 queued: 0 passed: 97 Fail ================================================================= [7260490] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:21:54 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260490/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260490/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8330177cb0a245c697ce6967eda4a8e1 Test failure: test_scrub_non_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2), test_scrub_non_mds0 (tasks.cephfs.test_multimds_misc.TestScrub2) [7260491] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:13:47 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260491/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260491/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73d44ce90d7d429a9ade0aa7a4fc29fc Command failed on smithi067 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260493] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} ----------------------------------------------------------------- time: 00:09:30 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260493/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260493/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bd1cb3fd3504f4683ebbc77e2430f67 Command failed on smithi112 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260494] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:28 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260494/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260494/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a20dd5089a114174941830edbc7be834 Command failed on smithi082 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7260495] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:21:45 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260495/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260495/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28c2452e0b15448f985c3bfb73cfb12f Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) [7260498] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:31:02 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260498/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260498/ error during scrub thrashing: rank damage found: {'backtrace'} [7260500] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:26:12 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260500/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=30318a5f4fd2473989a898f030ac6b14 Command failed (workunit test suites/ffsb.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7260502] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:19:53 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260502/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260502/ Command failed on smithi083 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7260505] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:09:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260505/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=692cd430c7b749caa4a6bbff54d974ab Command failed on smithi037 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260506] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 00:09:17 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260506/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1ee3ce56fa24fca98dba95166aaa683 Command failed on smithi097 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260507] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:19:58 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260507/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=724811b9e9c74da7a17459bceb1c82bd Test failure: test_malicious_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc) [7260509] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:22:09 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260509/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260509/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e907ec1e8e2d41d282e6da899d6bd249 Command failed on smithi086 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp' [7260510] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:09:29 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260510/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260510/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5427cf4b07064222b3c214cab9bfc6c4 Command failed on smithi171 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260511] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:26:15 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260511/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260511/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c68a61cc743d43ad8694fcf8e368c039 Test failure: test_scrub_task_status_on_mds_failover (tasks.cephfs.test_scrub_checks.TestScrubControls) [7260515] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:42:31 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260515/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260515/ Command failed on smithi072 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' [7260518] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} ----------------------------------------------------------------- time: 00:09:28 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260518/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260518/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd8cf61c8bb24b6bb4d3df031763fc88 Command failed on smithi130 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260519] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:09:26 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260519/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260519/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e453056e199540b8948c78c2566bf9a4 Command failed on smithi097 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260520] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:31:53 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260520/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260520/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d567f272549940b686eda017f730eeac Test failure: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) [7260522] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:24 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260522/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260522/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df7874d2bfc14666aea06461f9f1d4e6 Command failed on smithi123 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260525] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:29:05 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260525/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260525/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0cbfeb81f27547de8800490a7d4c555a Command failed (workunit test libcephfs/test.sh) on smithi012 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7260526] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:14:46 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260526/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260526/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f2b869b6a0a45b7b3435e2d492ea886 Test failure: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) [7260527] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:52:50 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260527/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9688a07ce2be45fe8f394ab79c48a46f Test failure: test_standby_replay_singleton (tasks.cephfs.test_failover.TestStandbyReplay) [7260528] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:32 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260528/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b51529b1fd0488b86929f0477111a87 Command failed on smithi195 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260529] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:09:33 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260529/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260529/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5cc25cb2ea61481692d5f0ee0f7c01ce Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260530] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:20:14 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260530/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260530/ Found coredumps on ubuntu@smithi138.front.sepia.ceph.com [7260531] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:40:32 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260531/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260531/ error during scrub thrashing: rank damage found: {'backtrace'} [7260532] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:17:49 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260532/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260532/ Command failed on smithi177 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [7260533] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:59:17 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260533/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260533/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e8fa66ecb2348098709039bc31242c4 saw valgrind issues [7260535] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:23:36 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260535/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260535/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9300d6e07aad4d81aee7b5aa89d61f4e Test failure: test_mds_max_snaps_per_dir_with_reduced_limit (tasks.cephfs.test_snapshots.TestSnapshots) [7260536] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:32:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260536/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260536/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d571a65b8214d22a78ad9d610613fdf Test failure: test_subvolume_snapshot_in_group (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) [7260538] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 03:07:45 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260538/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260538/ Exiting scrub checking -- not all pgs scrubbed. [7260539] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:23:20 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260539/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260539/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f8994cb243845b498eefd6a3d49f3bd Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) [7260541] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}} ----------------------------------------------------------------- time: 00:09:21 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260541/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260541/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3c69155e5e34dc095d902f773fce7fa Command failed on smithi023 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260546] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} ----------------------------------------------------------------- time: 06:18:27 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260546/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260546/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1cce5696688b40a2bd16dd1e09ca1a68 Command failed (workunit test fs/fscrypt.sh) on smithi112 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh unlocked iozone' [7260547] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:09:30 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260547/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260547/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91a2882d134a4f2f81b45a2556ac6bdf Command failed on smithi017 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260548] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} ----------------------------------------------------------------- time: 00:09:26 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260548/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260548/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3a98491a33848518391006ec93226dc Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260551] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:09:23 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260551/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260551/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d970dd67975346a482cd01b11fef7cb3 Command failed on smithi125 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260552] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:17:50 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260552/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260552/ Found coredumps on ubuntu@smithi099.front.sepia.ceph.com [7260556] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:09:33 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260556/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260556/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99f29d01aa7e4810b2f7dd5e4d7d8f34 Command failed on smithi177 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260559] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:09:29 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260559/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260559/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93213ac4aa404444a8047c36d7aae066 Command failed on smithi107 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260560] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:24 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260560/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260560/ sentry: https://sentry.ceph.com/organizations/ceph/?query=597136ba64614191ab5fca1fa6c657c7 Command failed on smithi082 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7260561] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:22:05 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260561/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260561/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2b6af8d998749d49202d6efc0e9dfb8 Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7260563] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:26:54 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260563/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260563/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea58dabedc5d40d18d7b8a1f39b566dc Test failure: test_volume_ls (tasks.cephfs.test_volumes.TestVolumes), test_volume_ls (tasks.cephfs.test_volumes.TestVolumes) [7260567] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:09:27 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260567/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260567/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8018871914f34c27a40b469160f1d036 Command failed on smithi125 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260569] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:28:34 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260569/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260569/ sentry: https://sentry.ceph.com/organizations/ceph/?query=182709ac5c9c48bb8c23b6153a3db1cc Test failure: test_standby_incompat_reject_multifs (tasks.cephfs.test_admin.TestCompatCommands) [7260572] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 03:29:41 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260572/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260572/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc4592f3f81540f786a8b916c7d5ee89 Command failed (workunit test suites/ffsb.sh) on smithi029 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7260575] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} ----------------------------------------------------------------- time: 00:09:28 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260575/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260575/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e416e1cf0f0441c8ef085b79a2f68a7 Command failed on smithi187 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260576] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:27 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260576/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260576/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcdabbea4e2e4ae8884a69a15d6484a7 Command failed on smithi117 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260577] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:17:18 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260577/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260577/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5999e04a994449b7a3867dd41c5e838d Command failed (workunit test fs/test_python.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7260578] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:21:55 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260578/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260578/ Command failed on smithi101 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7260579] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:28:03 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260579/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260579/ Command failed on smithi177 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [7260581] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:23:54 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260581/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260581/ Command failed on smithi097 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' [7260583] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} ----------------------------------------------------------------- time: 00:16:59 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260583/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260583/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b50a0b86e82f4a79a4b133a15f879acb Test failure: test_mds_readonly (tasks.cephfs.test_auto_repair.TestMDSAutoRepair) [7260586] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} ----------------------------------------------------------------- time: 00:13:44 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260586/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260586/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f14d71e07e04fffaef92a544c3ca5a7 Command failed on smithi057 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260587] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:50:27 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260587/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260587/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c030fdd04c0345f49a32fb8919e3ec15 Test failure: test_subvolume_snapshot_clone_pool_layout (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7260590] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} ----------------------------------------------------------------- time: 00:09:44 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260590/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260590/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7cbac3e298c4f01b0dc6d1195f52aa6 Command failed on smithi037 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260594] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:16 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260594/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260594/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1e661d59eb2425db97517bcab2d69a8 Command failed on smithi136 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7260595] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:20:22 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260595/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260595/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc83d305d78c4f7cb0ea16deb16329dd Test failure: test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity), test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) [7260597] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:31:11 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260597/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260597/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43122d5db2844f8cba28d5f6576dd44d Test failure: test_client_pin (tasks.cephfs.test_client_limits.TestClientLimits) [7260599] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:58 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260599/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260599/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b261e11f91d42d189d301207f3402ad Command failed on smithi150 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c34a39c8-e98a-11ed-9b01-001a4aab830c -- ceph mon dump -f json' [7260606] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:23:59 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260606/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260606/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5cfcefe2d0a648c08273525ffc18ec83 Test failure: test_dir_fsync (tasks.cephfs.test_client_recovery.TestClientRecovery), test_dir_fsync (tasks.cephfs.test_client_recovery.TestClientRecovery) [7260607] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:20:40 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260607/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260607/ Command failed on smithi060 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7260608] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:09:31 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260608/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260608/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bc7c61c744a4781aecaf61045a444d3 Command failed on smithi183 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph- immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon- debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs- debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw- debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260614] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:33:19 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260614/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260614/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3f5a5e49427478386ae144c2c270125 Test failure: test_rebuild_inotable (tasks.cephfs.test_data_scan.TestDataScan) [7260615] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:09:29 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260615/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260615/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c53e0d3c05784782b4fe546000c1e0ab Command failed on smithi181 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260616] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:42:55 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260616/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260616/ Command failed on smithi177 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [7260617] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:22:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260617/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260617/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afa81914fc174b7e87f55ed54953d662 Test failure: test_ephemeral_pin_dist_override_after (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_pin_dist_override_after (tasks.cephfs.test_exports.TestEphemeralPins) [7260619] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:09:34 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260619/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260619/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4abe1c97e8f142939f325aae3cde25c1 Command failed on smithi169 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260621] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:09:26 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260621/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f96b8a80c8a148ab88e0ec43eb114b68 Command failed on smithi117 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260622] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:09:22 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260622/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f17736a561141f9a8e1635e76487a17 Command failed on smithi125 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260623] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:16:23 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260623/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17403cb8f7fa4c47bcb84000333b57ae Command failed on smithi116 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:5efb30c69d6138881536a78b7fbd0dbc6d59d97d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea14d9b6-e98d-11ed-9b01-001a4aab830c -- ceph mon dump -f json' [7260624] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:23:05 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260624/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb77e13f1e604da09bb7e4a0ac3705a5 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt), test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7260625] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260625/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260625/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ede21950bda645dcb054aaca68722885 Command failed on smithi026 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260626] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:21:07 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260626/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260626/ Found coredumps on ubuntu@smithi099.front.sepia.ceph.com [7260627] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} ----------------------------------------------------------------- time: 00:21:16 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260627/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c31bc7738e04f568daba0e8153ca6e8 Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) [7260629] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:48:36 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260629/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c22f84ee0ad48af9f508487e02a07a7 Test failure: test_file_filesystem_sync_crash (tasks.cephfs.test_misc.TestMisc), test_file_filesystem_sync_crash (tasks.cephfs.test_misc.TestMisc) [7260630] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:22:20 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260630/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260630/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e23e6e1adc9c46caae173061980bf9a2 Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7260632] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:18:36 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260632/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260632/ sentry: https://sentry.ceph.com/organizations/ceph/?query=286aae8055c4424a8437d497a311affc Test failure: test_df_for_valid_directory (tasks.cephfs.test_cephfs_shell.TestDF) [7260634] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:17:20 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260634/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260634/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd2bba6e045847e1b1824145a93bbae1 Test failure: test_dumpfs (tasks.cephfs.test_fstop.TestFSTop) [7260635] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:09:27 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260635/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260635/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f219a4f14784c87a7ed512b946d87e0 Command failed on smithi149 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph- immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon- debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs- debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw- debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [7260639] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:39:03 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260639/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260639/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2d4b0fbd478406698e6764248739b35 Command failed (workunit test suites/dbench.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7260642] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:22:52 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260642/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260642/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f31acb9d68d24a478d21c530a422f2d0 Test failure: test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair), test_reset (tasks.cephfs.test_journal_repair.TestJournalRepair) [7260644] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:22:21 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260644/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260644/ Command failed on smithi026 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' [7260645] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} ----------------------------------------------------------------- time: 00:09:28 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260645/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260645/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5d15c1cca7f4ee48dcf2e3ea70e5a7e Command failed on smithi184 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260646] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:09:37 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260646/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260646/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22dfa427132a43328f04c8cb27ef5818 Command failed on smithi176 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260647] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:18:10 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260647/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260647/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bcf795fadb14438c8917091dae8228f6 Test failure: test_names_can_only_be_goodchars (tasks.cephfs.test_volumes.TestMisc) [7260650] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:20:57 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260650/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260650/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8990bc28ca264d15b1ee9c43bcff5c5e Test failure: test_full_fclose (tasks.cephfs.test_full.TestQuotaFull) [7260653] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} ----------------------------------------------------------------- time: 00:16:10 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260653/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260653/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fcf79a58dd454b32b2fa96c8d228663e Command failed on smithi037 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260656] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:10:07 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260656/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260656/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f0d41812f2e47528d2db63ae5b62f33 Command failed on smithi149 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260659] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:33:30 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260659/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260659/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a65065a4cff4d71a9b5aafc2e68cb4a Test failure: test_for_invalid_client_ip (tasks.cephfs.test_mds_metrics.TestMDSMetrics) [7260662] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:09:41 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260662/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260662/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c0e0dc0c76745b4898d9b039a2995c4 Command failed on smithi167 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260663] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:44:11 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260663/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260663/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0390cc832484e418b879f0d4fb9af86 Command failed (workunit test suites/fsstress.sh) on smithi118 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7260664] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:28 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260664/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260664/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8056d1caaca442db39794ef4e5727f6 Command failed on smithi088 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7260665] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:23:11 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260665/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260665/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d32a31786d4b4adbb0e1af2759192a4a Test failure: test_down (tasks.cephfs.test_failover.TestClusterResize) [7260668] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} ----------------------------------------------------------------- time: 00:09:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260668/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260668/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb8f83098ca8462cbe64ff8b3f776c13 Command failed on smithi134 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260670] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:15:54 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260670/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260670/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d61c8dd439fe4789925c5415282712f4 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7260671] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 00:09:21 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260671/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260671/ sentry: https://sentry.ceph.com/organizations/ceph/?query=97a730332cd449eda7d2f69f94151247 Command failed on smithi169 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260672] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:24:14 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260672/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260672/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2814a1204bd6418aa6c7d7e3bc035444 Test failure: test_subvolume_retain_snapshot_with_snapshots (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) [7260677] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:36:59 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260677/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260677/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b54a64be831e4fdbaa47a568570a0842 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi135 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5efb30c69d6138881536a78b7fbd0dbc6d59d97d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7260678] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:15:16 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260678/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260678/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6327777df597459f8895def0ecb141d3 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7260679] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:32 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260679/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260679/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b8a70d1eef64f049af7139c68b96298 Command failed on smithi037 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260688] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:17:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260688/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260688/ Found coredumps on ubuntu@smithi027.front.sepia.ceph.com [7260692] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:58:52 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260692/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260692/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7de988826c974ab39b5606adc3376c2f saw valgrind issues Dead ================================================================= [7260540] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260540/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260540/ hit max job timeout [7260555] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260555/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260555/ hit max job timeout [7260628] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260628/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260628/ hit max job timeout [7260673] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260673/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260673/ hit max job timeout Pass ================================================================= [7260489] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:46:40 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260489/ [7260492] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:29:22 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260492/ [7260496] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:48:12 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260496/ [7260497] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:17:56 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260497/ [7260499] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:35:49 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260499/ [7260501] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:14:16 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260501/ [7260503] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:29 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260503/ [7260504] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:22 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260504/ [7260508] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:28:09 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260508/ [7260512] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:28:56 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260512/ [7260513] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:19 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260513/ [7260514] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 02:20:34 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260514/ [7260516] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:35:20 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260516/ [7260517] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:24:46 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260517/ [7260521] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:42:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260521/ [7260523] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:30 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260523/ [7260524] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:24:38 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260524/ [7260534] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:42:15 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260534/ [7260537] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:51 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260537/ [7260542] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:21:58 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260542/ [7260543] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:32:16 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260543/ [7260544] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:18 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260544/ [7260545] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:15:09 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260545/ [7260549] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:42:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260549/ [7260550] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:16:36 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260550/ [7260553] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:23:08 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260553/ [7260554] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:32:08 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260554/ [7260557] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:49:16 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260557/ [7260558] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:15 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260558/ [7260562] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:38:54 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260562/ [7260564] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 01:07:14 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260564/ [7260565] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:24 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260565/ [7260566] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:16:42 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260566/ [7260568] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:29:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260568/ [7260570] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:26:00 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260570/ [7260571] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:54:08 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260571/ [7260573] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:13:52 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260573/ [7260574] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:17:07 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260574/ [7260580] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:44:53 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260580/ [7260582] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:38:18 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260582/ [7260584] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:30:31 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260584/ [7260585] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:18:32 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260585/ [7260588] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:13:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260588/ [7260589] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 02:39:55 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260589/ [7260591] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:26:29 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260591/ [7260592] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:32:03 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260592/ [7260593] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:50:25 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260593/ [7260596] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:56:20 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260596/ [7260598] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 02:05:07 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260598/ [7260600] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:20:14 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260600/ [7260601] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:32:42 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260601/ [7260602] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:13 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260602/ [7260603] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:18:28 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260603/ [7260604] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:40:02 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260604/ [7260605] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} time: 00:49:15 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260605/ [7260609] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:32:07 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260609/ [7260610] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:30:21 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260610/ [7260611] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:17:30 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260611/ [7260612] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:29:54 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260612/ [7260613] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:35:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260613/ [7260618] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:44:57 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260618/ [7260620] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:19:02 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260620/ [7260631] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:20 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260631/ [7260633] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:12:48 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260633/ [7260636] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260636/ [7260637] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:21:37 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260637/ [7260638] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:16:47 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260638/ [7260640] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:21:03 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260640/ [7260641] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:15:04 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260641/ [7260643] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:38:14 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260643/ [7260648] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:19:09 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260648/ [7260649] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:44:56 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260649/ [7260651] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:46:44 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260651/ [7260652] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:30:10 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260652/ [7260654] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:53:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260654/ [7260655] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:41:23 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260655/ [7260657] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:06 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260657/ [7260658] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260658/ [7260660] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} time: 00:27:23 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260660/ [7260661] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:28:44 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260661/ [7260666] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:50:38 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260666/ [7260667] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:18:35 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260667/ [7260669] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:37:32 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260669/ [7260674] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:58 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260674/ [7260675] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:44:52 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260675/ [7260676] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:24:05 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260676/ [7260680] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:14:37 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260680/ [7260681] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:42:31 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260681/ [7260682] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260682/ [7260683] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:14:28 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260683/ [7260684] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:37 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260684/ [7260685] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} time: 00:53:11 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260685/ [7260686] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:10 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260686/ [7260687] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:35:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260687/ [7260689] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:36:17 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260689/ [7260690] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:21:01 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260690/ [7260691] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 02:50:27 info: http://pulpito.ceph.com/vshankar-2023-05-03_05:29:04-fs-wip-vshankar-testing-20230501.060318-testing-default-smithi/7260691/