2024-02-21T06:29:11.524 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-21T06:29:11.524 INFO:teuthology.results:subset: '669/1024' 2024-02-21T06:29:11.524 INFO:teuthology.results:seed: '8780' 2024-02-21T07:12:05.944 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-21T07:12:05.971 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-21T18:39:56.038 INFO:teuthology.results:Tests finished! gathering results... 2024-02-21T18:40:03.582 INFO:scrape:Found 86 jobs 2024-02-21T18:40:05.562 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.562 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.562 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.563 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.797 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.797 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.797 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.797 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.976 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.976 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.976 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:05.977 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.158 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.159 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.159 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.159 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.345 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.346 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.346 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:06.346 WARNING:scrape:Saw end of BT but not start 2024-02-21T18:40:17.477 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569270/teuthology.log 2024-02-21T18:40:40.845 INFO:scrape:Found 17 distinct failure reasons 2024-02-21T18:40:40.846 INFO:scrape:Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) 2024-02-21T18:40:40.846 INFO:scrape:1 jobs: ['7569269'] 2024-02-21T18:40:40.846 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-02-21T18:40:40.846 INFO:scrape: 2024-02-21T18:40:40.846 INFO:scrape:Failure: Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' 2024-02-21T18:40:40.846 INFO:scrape:37 jobs 2024-02-21T18:40:40.847 INFO:scrape:suites intersection: [] 2024-02-21T18:40:40.847 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '1-ranks/1', '1-ranks/2', '1-rename', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-workload/dbench', '3-inline/no', '3-inline/yes', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host', 'distro/single-container-host}', 'export-check', 'fail_fs/no', 'fail_fs/yes', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/ignorelist_health', 'overrides/ignorelist_health}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-02-21T18:40:40.848 INFO:scrape: 2024-02-21T18:40:40.848 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-02-21T18:40:40.848 INFO:scrape:2 jobs: ['7569286', '7569294'] 2024-02-21T18:40:40.848 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2024-02-21T18:40:40.848 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-02-21T18:40:40.848 INFO:scrape: 2024-02-21T18:40:40.848 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-02-21T18:40:40.848 INFO:scrape:1 jobs: ['7569260'] 2024-02-21T18:40:40.849 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-02-21T18:40:40.849 INFO:scrape: 2024-02-21T18:40:40.849 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-02-21T18:40:40.849 INFO:scrape:1 jobs: ['7569255'] 2024-02-21T18:40:40.849 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-02-21T18:40:40.849 INFO:scrape: 2024-02-21T18:40:40.849 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi018.front.sepia.ceph.com: ['type=AVC msg=audit(1708500106.608:192): avc: denied { checkpoint_restore } for pid=1306 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-02-21T18:40:40.849 INFO:scrape:9 jobs: ['7569282', '7569240', '7569245', '7569258', '7569236', '7569254', '7569232', '7569256', '7569261'] 2024-02-21T18:40:40.849 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2024-02-21T18:40:40.850 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/iozone}}', '2-workunit/suites/pjd}}', 'cluster/{1-node}', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/32bits/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/ignorelist_health', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'session_timeout}', 'subvol_versions/create_subvol_version_v1', 'supported-random-distros$/{centos_latest}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/fsstress', 'tasks/fstop}', 'tasks/mgr-osd-full}', 'tasks/snap-schedule}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/lockdep}'] 2024-02-21T18:40:40.850 INFO:scrape: 2024-02-21T18:40:40.850 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-02-21T18:40:40.850 INFO:scrape:1 jobs: ['7569229'] 2024-02-21T18:40:40.850 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/pool-perm}'] 2024-02-21T18:40:40.850 INFO:scrape: 2024-02-21T18:40:40.850 INFO:scrape:Dead: None 2024-02-21T18:40:40.850 INFO:scrape:3 jobs: ['7569270', '7569225', '7569295'] 2024-02-21T18:40:40.850 INFO:scrape: 2024-02-21T18:40:40.850 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-02-21T18:40:40.850 INFO:scrape:2 jobs: ['7569265', '7569224'] 2024-02-21T18:40:40.851 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-21T18:40:40.851 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-21T18:40:40.851 INFO:scrape: 2024-02-21T18:40:40.851 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-02-21T18:40:40.851 INFO:scrape:1 jobs: ['7569292'] 2024-02-21T18:40:40.851 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-21T18:40:40.851 INFO:scrape: 2024-02-21T18:40:40.851 INFO:scrape:Failure: "2024-02-21T07:21:21.878250+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-02-21T18:40:40.851 INFO:scrape:2 jobs: ['7569272', '7569264'] 2024-02-21T18:40:40.852 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2024-02-21T18:40:40.852 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/blogbench}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/1', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-02-21T18:40:40.852 INFO:scrape: 2024-02-21T18:40:40.852 INFO:scrape:Crash: Test failure: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-02-21T18:40:40.852 INFO:scrape:ceph version 19.0.0-1514-g25296a8f (25296a8f176256aea5ed8b5850b25167dce0ae33) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0144242520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x189) [0x7f0144b81280] 6: ceph-mds(+0x4b73b4) [0x5639466fc3b4] 7: (InoTable::replay_release_ids(interval_set&)+0x60) [0x563946603a90] 8: (EPurged::replay(MDSRank*)+0x13b) [0x5639463e843b] 9: (MDLog::_replay_thread()+0x789) [0x563946645139] 10: ceph-mds(+0x3f5081) [0x56394663a081] 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0144294b43] 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0144326a00] 2024-02-21T18:40:40.852 INFO:scrape:1 jobs: ['7569288'] 2024-02-21T18:40:40.852 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/forward-scrub}'] 2024-02-21T18:40:40.852 INFO:scrape: 2024-02-21T18:40:40.852 INFO:scrape:Failure: Test failure: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-02-21T18:40:40.852 INFO:scrape:1 jobs: ['7569249'] 2024-02-21T18:40:40.853 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/strays}'] 2024-02-21T18:40:40.853 INFO:scrape: 2024-02-21T18:40:40.853 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-02-21T18:40:40.853 INFO:scrape:1 jobs: ['7569211'] 2024-02-21T18:40:40.853 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-02-21T18:40:40.853 INFO:scrape: 2024-02-21T18:40:40.853 INFO:scrape:Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) 2024-02-21T18:40:40.853 INFO:scrape:1 jobs: ['7569243'] 2024-02-21T18:40:40.853 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2024-02-21T18:40:40.853 INFO:scrape: 2024-02-21T18:40:40.853 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2024-02-21T18:40:40.853 INFO:scrape:1 jobs: ['7569262'] 2024-02-21T18:40:40.854 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-02-21T18:40:40.854 INFO:scrape: 2024-02-21T18:40:40.854 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-02-21T18:40:40.854 INFO:scrape:1 jobs: ['7569227'] 2024-02-21T18:40:40.854 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-02-21T18:40:40.854 INFO:scrape: 2024-02-21T18:40:40.859 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/ failed: 63 dead: 3 running: 0 waiting: 0 queued: 0 passed: 20 Fail ================================================================= [7569211] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:35 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569211/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa3599c6b8ee4595b90df3d96e26d983 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7569213] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:09:42 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569213/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569213/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54521679f715457b94b6da79633f544f Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569215] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:10:35 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569215/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99909752cf504497bc5d0e2a76a01248 Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569217] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:12:08 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569217/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=983c3027a34740dfa01e2cbae840604c Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569218] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:09:49 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569218/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569218/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c97a8dc5222440a803fcf6b195c9c52 Command failed on smithi001 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569221] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:16:23 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569221/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569221/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7904899f419140819f4d0da37f9bdeb1 Command failed on smithi018 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569224] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:54:45 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569224/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569224/ valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7569226] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:13:08 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569226/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569226/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c569ce41e68422ab30fc0040a0958a0 Command failed on smithi139 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569227] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 01:31:17 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569227/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569227/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6616e5d9192645d99683303c9832665d Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7569228] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:26 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569228/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569228/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5a876a283de48c8a4ef2a8ec29a74dc Command failed on smithi164 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569229] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/pool-perm} ----------------------------------------------------------------- time: 00:18:41 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569229/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b637121ab1254ed993716056ff34f900 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7569230] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:10:58 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569230/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569230/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52b6c490e95f47089c7c5a215871e072 Command failed on smithi089 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569231] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:21:40 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569231/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569231/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63e3f04b838a4f7988a4404f949cbcdc Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569232] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:15:18 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569232/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569232/ SELinux denials found on ubuntu@smithi114.front.sepia.ceph.com: ['type=AVC msg=audit(1708498366.197:203): avc: denied { checkpoint_restore } for pid=1190 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569233] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:16:20 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569233/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569233/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49a53f95e0af4063b76ff388a14a1173 Command failed on smithi176 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569236] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:19:17 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569236/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569236/ SELinux denials found on ubuntu@smithi067.front.sepia.ceph.com: ['type=AVC msg=audit(1708498463.368:200): avc: denied { checkpoint_restore } for pid=1209 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569238] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:10:03 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569238/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569238/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34053b9d3a0a463f95a6d3c26e4b1ae2 Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569240] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:13:37 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569240/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569240/ SELinux denials found on ubuntu@smithi144.front.sepia.ceph.com: ['type=AVC msg=audit(1708498377.027:192): avc: denied { checkpoint_restore } for pid=1210 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569241] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:18:36 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569241/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569241/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd782f71d9c443d59294f695bdca44f3 Command failed on smithi060 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569242] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:56 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569242/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569242/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2512314c95f44b99a5085eeae4fe7867 Command failed on smithi113 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569243] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:16:37 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569243/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569243/ sentry: https://sentry.ceph.com/organizations/ceph/?query=beb95c38a51941f091780371431a21c0 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) [7569245] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:26:46 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569245/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569245/ SELinux denials found on ubuntu@smithi170.front.sepia.ceph.com: ['type=AVC msg=audit(1708498467.776:203): avc: denied { checkpoint_restore } for pid=1192 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569246] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:10:15 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569246/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569246/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48cb164d3ced430884a11b4dc5523ada Command failed on smithi102 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569249] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} ----------------------------------------------------------------- time: 00:14:44 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569249/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569249/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aea228422b6442e688baeb9775c54350 Test failure: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) [7569250] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:11:13 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569250/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569250/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eab5a43ea6b94741a81d987defda71ec Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569251] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:13:14 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569251/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569251/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dfb68a0c61c34fab8eb21b77d248e18d Command failed on smithi099 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569252] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:10:35 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569252/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569252/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2086363db92547bf808048a816c087d0 Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569253] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:09:33 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569253/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569253/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62c7e09e35124897b0cbc0e584dc1881 Command failed on smithi088 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569254] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:15:39 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569254/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569254/ SELinux denials found on ubuntu@smithi132.front.sepia.ceph.com: ['type=AVC msg=audit(1708498385.446:193): avc: denied { checkpoint_restore } for pid=1189 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569255] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 11:43:00 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569255/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569255/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c26a260bc9a4491692a0498c34b602ca Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7569256] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:19:27 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569256/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569256/ SELinux denials found on ubuntu@smithi131.front.sepia.ceph.com: ['type=AVC msg=audit(1708498329.176:201): avc: denied { checkpoint_restore } for pid=1163 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569257] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:10:49 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569257/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569257/ sentry: https://sentry.ceph.com/organizations/ceph/?query=600c68a774874907a8b09ffe7259c0fa Command failed on smithi107 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569258] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:21:13 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569258/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569258/ SELinux denials found on ubuntu@smithi195.front.sepia.ceph.com: ['type=AVC msg=audit(1708498501.033:202): avc: denied { checkpoint_restore } for pid=1143 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569259] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:12:08 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569259/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569259/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cba58b53d9f4025af61671dd587bb0c Command failed on smithi049 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569260] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:46:09 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569260/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569260/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28ad8043369340be82f373c58300881d Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7569261] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:15:51 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569261/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569261/ SELinux denials found on ubuntu@smithi005.front.sepia.ceph.com: ['type=AVC msg=audit(1708498475.743:201): avc: denied { checkpoint_restore } for pid=1141 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569262] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:30:50 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569262/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569262/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06c65b54546f443580d12c66217041d2 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7569264] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:47:11 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569264/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569264/ "2024-02-21T06:56:31.514691+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7569265] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:54:17 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569265/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569265/ valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7569266] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:10:12 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569266/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569266/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7782c92483644c849c28b60892d53897 Command failed on smithi016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569267] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:16:27 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569267/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569267/ sentry: https://sentry.ceph.com/organizations/ceph/?query=990fc4fb0a834042ac9e99ee0980cb9c Command failed on smithi040 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569268] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:14:58 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569268/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569268/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a54176706a434ce894a59c31e53ce238 Command failed on smithi019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569269] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:37:13 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569269/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d41ddf2dbdd499daa63d5b9affdc4f8 Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) [7569272] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:44:20 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569272/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569272/ "2024-02-21T07:21:21.878250+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7569273] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:12:26 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569273/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569273/ sentry: https://sentry.ceph.com/organizations/ceph/?query=229bb96101db455e8e99117c15e17d6a Command failed on smithi151 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569274] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:18:30 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569274/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569274/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea87801b581c48a09756d5561183548a Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569275] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:11:39 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569275/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569275/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1362335d29984368b219e1b77ee57560 Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569276] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:11:34 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569276/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569276/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdc9cc6b27b44c28bbe3d63ca7c01c15 Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569277] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:10 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569277/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd83738a476049db81fa0bd3e2833a10 Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569281] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:11:36 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569281/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569281/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15f288f24c3042d3a2937e44a772091b Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569282] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:17:18 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569282/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569282/ SELinux denials found on ubuntu@smithi018.front.sepia.ceph.com: ['type=AVC msg=audit(1708500106.608:192): avc: denied { checkpoint_restore } for pid=1306 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7569283] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:10:04 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569283/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569283/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29e312e6f0d34f06a95cd724c4b091f8 Command failed on smithi164 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569284] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:02 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569284/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34d4903802584717a84125ba0024ccf3 Command failed on smithi089 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569285] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:11:18 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569285/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569285/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f918b001c41c4dc490bba31eb41083c0 Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569286] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:57:52 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569286/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569286/ error during scrub thrashing: rank damage found: {'backtrace'} [7569288] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:21:55 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569288/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569288/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4bf7b8a103cc47778edd1f83fc205b1f Test failure: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) [7569289] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:11:57 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569289/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569289/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a51a9c4ed01848c6acc39ab4b3bf34a1 Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569290] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:12:06 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569290/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569290/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f1d93be930e43b69f81b4da6acc28d3 Command failed on smithi162 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7569291] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:10:07 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569291/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569291/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de08865d953d4c2e8d307b13246cd3ca Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569292] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:48:58 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569292/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569292/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7569293] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:09:38 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569293/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569293/ sentry: https://sentry.ceph.com/organizations/ceph/?query=384d223d451c4aaba4b6e6925a5c09c8 Command failed on smithi088 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7569294] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:33:43 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569294/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569294/ error during scrub thrashing: rank damage found: {'backtrace'} [7569296] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:10:28 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569296/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2edc05cdcbef4e3e97a2f0b88774bc5b Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' Dead ================================================================= [7569225] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569225/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569225/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7569270] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569270/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569270/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7569295] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569295/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569295/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7569212] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 02:09:15 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569212/ [7569214] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:24:17 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569214/ [7569216] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-full} time: 00:22:56 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569216/ [7569219] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds_creation_retry} time: 00:12:06 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569219/ [7569220] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/metrics} time: 00:35:28 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569220/ [7569222] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:19 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569222/ [7569223] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:17 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569223/ [7569234] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:41:09 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569234/ [7569235] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/scrub} time: 00:32:11 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569235/ [7569237] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:50 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569237/ [7569239] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:15:23 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569239/ [7569244] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:06:42 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569244/ [7569247] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snapshots} time: 00:23:13 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569247/ [7569248] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:38:57 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569248/ [7569263] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:52 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569263/ [7569271] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:12:47 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569271/ [7569278] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 01:04:14 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569278/ [7569279] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-readahead} time: 00:14:28 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569279/ [7569280] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:53:35 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569280/ [7569287] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:10 info: https://pulpito.ceph.com/vshankar-2024-02-21_06:28:57-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7569287/