2024-01-25T07:20:57.584 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-25T07:20:57.584 INFO:teuthology.results:subset: '48/1024' 2024-01-25T07:20:57.584 INFO:teuthology.results:seed: '2348' 2024-01-25T07:57:58.917 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-25T07:57:58.920 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-25T16:41:37.265 INFO:teuthology.results:Tests finished! gathering results... 2024-01-25T16:41:39.134 INFO:scrape:Found 21 jobs 2024-01-25T16:41:39.342 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.343 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.343 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.343 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.348 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.348 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.348 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.348 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.416 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.416 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.417 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.417 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.421 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.421 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.421 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.421 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.429 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.429 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.429 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.429 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.518 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.519 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.519 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.519 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.523 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.523 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.523 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.523 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.532 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.532 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.532 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.532 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.536 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.537 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.537 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.537 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.545 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.545 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.545 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.545 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.635 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.636 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.636 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.636 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.641 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.641 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.641 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.641 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.649 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.649 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.649 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.649 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.653 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.654 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.654 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.654 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.662 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.662 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.662 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:39.662 WARNING:scrape:Saw end of BT but not start 2024-01-25T16:41:58.908 INFO:scrape:Found 11 distinct failure reasons 2024-01-25T16:41:58.908 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37a6d7ab43c9ea40d231cf2a2c77e3f3bc163ca2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-01-25T16:41:58.909 INFO:scrape:1 jobs: ['7531248'] 2024-01-25T16:41:58.909 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-01-25T16:41:58.909 INFO:scrape: 2024-01-25T16:41:58.909 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37a6d7ab43c9ea40d231cf2a2c77e3f3bc163ca2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-01-25T16:41:58.909 INFO:scrape:1 jobs: ['7531254'] 2024-01-25T16:41:58.909 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-01-25T16:41:58.909 INFO:scrape: 2024-01-25T16:41:58.910 INFO:scrape:Failure: valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions 2024-01-25T16:41:58.910 INFO:scrape:3 jobs: ['7531256', '7531241', '7531250'] 2024-01-25T16:41:58.910 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2024-01-25T16:41:58.910 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-01-25T16:41:58.910 INFO:scrape: 2024-01-25T16:41:58.910 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_clone_with_no_wait_enabled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2024-01-25T16:41:58.910 INFO:scrape:2 jobs: ['7531246', '7531257'] 2024-01-25T16:41:58.910 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2024-01-25T16:41:58.910 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}'] 2024-01-25T16:41:58.911 INFO:scrape: 2024-01-25T16:41:58.911 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2024-01-25T16:41:58.911 INFO:scrape:1 jobs: ['7531240'] 2024-01-25T16:41:58.911 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-01-25T16:41:58.911 INFO:scrape: 2024-01-25T16:41:58.911 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-01-25T16:41:58.911 INFO:scrape:1 jobs: ['7531255'] 2024-01-25T16:41:58.911 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-01-25T16:41:58.911 INFO:scrape: 2024-01-25T16:41:58.911 INFO:scrape:Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) 2024-01-25T16:41:58.911 INFO:scrape:1 jobs: ['7531247'] 2024-01-25T16:41:58.911 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-01-25T16:41:58.912 INFO:scrape: 2024-01-25T16:41:58.912 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-01-25T16:41:58.912 INFO:scrape:1 jobs: ['7531244'] 2024-01-25T16:41:58.912 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-01-25T16:41:58.912 INFO:scrape: 2024-01-25T16:41:58.912 INFO:scrape:Failure: Command failed on smithi149 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1db5459e688287ea17e1bda532d3c81a51b40e23 pull' 2024-01-25T16:41:58.912 INFO:scrape:1 jobs: ['7531239'] 2024-01-25T16:41:58.912 INFO:scrape:suites: ['conf/{client', 'fs/nfs/{cluster/{1-node}', 'mds', 'mon', 'osd}', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_latest}', 'tasks/nfs}'] 2024-01-25T16:41:58.912 INFO:scrape: 2024-01-25T16:41:58.912 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-01-25T16:41:58.912 INFO:scrape:1 jobs: ['7531243'] 2024-01-25T16:41:58.913 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{balancer/automatic', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-01-25T16:41:58.913 INFO:scrape: 2024-01-25T16:41:58.913 INFO:scrape:Failure: Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) 2024-01-25T16:41:58.913 INFO:scrape:1 jobs: ['7531258'] 2024-01-25T16:41:58.913 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2024-01-25T16:41:58.913 INFO:scrape: 2024-01-25T16:41:58.914 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/ failed: 14 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7531239] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:09:34 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531239/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531239/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e290c8610ba84b90a752c74085261e1f Command failed on smithi149 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:1db5459e688287ea17e1bda532d3c81a51b40e23 pull' [7531240] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:28:31 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531240/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531240/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e56f8c4c29c4fdb806238250e23116b Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7531241] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:26:13 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531241/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531241/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7531243] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:46:04 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531243/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531243/ error during scrub thrashing: rank damage found: {'backtrace'} [7531244] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 09:01:11 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531244/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531244/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a061712c6ede4b8c94ef531d0837440d Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7531246] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:49:44 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531246/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531246/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55995a3e3e674db189b128b8d868a30d Test failure: test_subvolume_snapshot_clone_with_no_wait_enabled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7531247] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:34:07 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531247/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531247/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c0464e221414a4a95ca81f28c51ed43 Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) [7531248] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:57:00 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531248/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531248/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f63877e14b146ce84e34313f60f88a2 Command failed (workunit test suites/dbench.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37a6d7ab43c9ea40d231cf2a2c77e3f3bc163ca2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7531250] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:50:50 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531250/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531250/ valgrind error: Leak_StillReachable malloc malloc strdup [7531254] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:37:16 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531254/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531254/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e32ebecfd3044fc59ceb4b94b180483e Command failed (workunit test suites/ffsb.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=37a6d7ab43c9ea40d231cf2a2c77e3f3bc163ca2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7531255] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:12:05 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531255/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531255/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c7b8eb230924ffe9fad4e6f91b436d1 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7531256] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:26:41 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531256/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531256/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7531257] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:53:12 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531257/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531257/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1c8a9d5b3824365a3a3c501fcd2092c Test failure: test_subvolume_snapshot_clone_with_no_wait_enabled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7531258] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:14:10 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531258/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531258/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55f2daa28a794b0aaa146734aa4fc835 Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) Pass ================================================================= [7531238] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:28:05 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531238/ [7531242] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:22:11 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531242/ [7531245] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 00:44:57 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531245/ [7531249] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:10:35 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531249/ [7531251] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:18 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531251/ [7531252] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:21:10 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531252/ [7531253] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:17:37 info: https://pulpito.ceph.com/vshankar-2024-01-25_07:18:10-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7531253/