2024-03-11T04:49:29.317 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-11T04:49:29.318 INFO:teuthology.results:subset: '832/1024' 2024-03-11T04:49:29.318 INFO:teuthology.results:seed: '6045' 2024-03-11T05:49:56.636 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-11T05:49:56.640 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-11T17:49:52.456 INFO:teuthology.results:Tests finished! gathering results... 2024-03-11T17:50:00.416 INFO:scrape:Found 58 jobs 2024-03-11T17:50:15.499 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.500 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.500 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.500 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.514 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.514 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.514 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.514 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.553 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.554 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.554 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.554 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.770 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.770 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.770 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.770 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.784 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.784 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.784 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.784 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.795 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.795 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.795 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:15.795 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.008 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.008 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.008 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.009 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.021 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.022 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.022 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.022 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.033 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.033 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.033 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.033 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.258 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.259 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.259 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.259 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.273 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.273 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.273 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.273 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.286 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.286 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.286 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.286 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.496 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.497 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.497 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.497 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.509 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.510 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.510 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.510 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.521 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.521 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.521 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.521 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.791 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.791 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.791 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.791 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.805 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.806 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.806 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.806 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.819 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.819 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.819 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:16.820 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.030 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.031 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.031 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.031 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.044 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.044 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.045 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.045 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.056 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.057 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.057 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.057 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.271 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.271 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.271 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.272 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.285 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.285 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.285 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.285 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.297 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.297 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.297 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.297 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.529 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.529 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.529 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.529 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.545 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.545 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.545 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.545 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.558 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.558 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.559 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.559 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.801 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.801 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.801 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.802 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.816 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.816 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.816 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.816 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.829 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.829 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.829 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:17.829 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.046 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.047 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.047 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.047 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.060 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.060 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.060 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.060 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.071 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.072 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.072 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.072 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.325 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.326 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.326 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.326 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.339 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.339 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.339 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.339 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.350 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.350 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.350 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.350 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.579 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.580 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.580 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.581 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.596 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.596 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.596 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.596 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.607 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.607 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.607 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.607 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.819 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.820 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.820 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.820 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.833 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.833 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.833 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.833 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.844 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.844 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.844 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:18.845 WARNING:scrape:Saw end of BT but not start 2024-03-11T17:50:55.761 INFO:scrape:Found 20 distinct failure reasons 2024-03-11T17:50:55.762 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-11T17:50:55.762 INFO:scrape:1 jobs: ['7590726'] 2024-03-11T17:50:55.762 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-11T17:50:55.762 INFO:scrape: 2024-03-11T17:50:55.763 INFO:scrape:Failure: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2024-03-11T17:50:55.763 INFO:scrape:7 jobs: ['7590734', '7590731', '7590738', '7590753', '7590722', '7590716', '7590748'] 2024-03-11T17:50:55.763 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2024-03-11T17:50:55.763 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'frag', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2024-03-11T17:50:55.763 INFO:scrape: 2024-03-11T17:50:55.763 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi049 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-03-11T17:50:55.763 INFO:scrape:2 jobs: ['7590704', '7590730'] 2024-03-11T17:50:55.764 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2024-03-11T17:50:55.764 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'export-check', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'ubuntu_latest}}', 'wsync/no}'] 2024-03-11T17:50:55.764 INFO:scrape: 2024-03-11T17:50:55.764 INFO:scrape:Failure: valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions 2024-03-11T17:50:55.764 INFO:scrape:2 jobs: ['7590721', '7590736'] 2024-03-11T17:50:55.764 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2024-03-11T17:50:55.764 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-11T17:50:55.765 INFO:scrape: 2024-03-11T17:50:55.765 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi115.front.sepia.ceph.com: ['type=AVC msg=audit(1710135055.071:208): avc: denied { checkpoint_restore } for pid=1205 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-11T17:50:55.765 INFO:scrape:5 jobs: ['7590723', '7590739', '7590740', '7590703', '7590711'] 2024-03-11T17:50:55.765 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-03-11T17:50:55.765 INFO:scrape:suites union: ['1-ceph', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-logrotate}', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fail_fs/yes', 'fs/functional/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'syntax}', 'tasks/exports}', 'tasks/openfiletable}', 'tasks/quota}', 'tasks/{0-from/quincy'] 2024-03-11T17:50:55.765 INFO:scrape: 2024-03-11T17:50:55.765 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-11T17:50:55.766 INFO:scrape:5 jobs: ['7590719', '7590729', '7590735', '7590752', '7590744'] 2024-03-11T17:50:55.766 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-03-11T17:50:55.766 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-03-11T17:50:55.766 INFO:scrape: 2024-03-11T17:50:55.766 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-11T17:50:55.766 INFO:scrape:1 jobs: ['7590756'] 2024-03-11T17:50:55.766 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/pool-perm}'] 2024-03-11T17:50:55.766 INFO:scrape: 2024-03-11T17:50:55.767 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi006 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-11T17:50:55.767 INFO:scrape:2 jobs: ['7590737', '7590718'] 2024-03-11T17:50:55.767 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/3', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-11T17:50:55.767 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-03-11T17:50:55.767 INFO:scrape: 2024-03-11T17:50:55.767 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) 2024-03-11T17:50:55.767 INFO:scrape:1 jobs: ['7590749'] 2024-03-11T17:50:55.767 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-03-11T17:50:55.768 INFO:scrape: 2024-03-11T17:50:55.768 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2024-03-11T17:50:55.768 INFO:scrape:2 jobs: ['7590727', '7590720'] 2024-03-11T17:50:55.768 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'frag', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2024-03-11T17:50:55.768 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '2-workunit/cfuse_workunit_suites_pjd}}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'wsync/no}'] 2024-03-11T17:50:55.768 INFO:scrape: 2024-03-11T17:50:55.768 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-03-11T17:50:55.768 INFO:scrape:1 jobs: ['7590747'] 2024-03-11T17:50:55.769 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-11T17:50:55.769 INFO:scrape: 2024-03-11T17:50:55.769 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-11T17:50:55.769 INFO:scrape:1 jobs: ['7590754'] 2024-03-11T17:50:55.769 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-03-11T17:50:55.769 INFO:scrape: 2024-03-11T17:50:55.769 INFO:scrape:Dead: b'2024-03-11T17:26:20.368 DEBUG:teuthology.exit:Finished running handlers' 2024-03-11T17:50:55.769 INFO:scrape:2 jobs: ['7590724', '7590745'] 2024-03-11T17:50:55.770 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-03-11T17:50:55.770 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd-asserts}', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/xfstests-dev}', 'ubuntu_latest}}'] 2024-03-11T17:50:55.770 INFO:scrape: 2024-03-11T17:50:55.770 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-03-11T17:50:55.770 INFO:scrape:1 jobs: ['7590750'] 2024-03-11T17:50:55.770 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-11T17:50:55.770 INFO:scrape: 2024-03-11T17:50:55.770 INFO:scrape:Failure: "1710138000.0001762 mon.smithi073 (mon.0) 777 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log 2024-03-11T17:50:55.770 INFO:scrape:1 jobs: ['7590757'] 2024-03-11T17:50:55.771 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-11T17:50:55.771 INFO:scrape: 2024-03-11T17:50:55.771 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none pjd' 2024-03-11T17:50:55.771 INFO:scrape:1 jobs: ['7590741'] 2024-03-11T17:50:55.771 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-pjd}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-03-11T17:50:55.771 INFO:scrape: 2024-03-11T17:50:55.771 INFO:scrape:Failure: "2024-03-11T06:10:22.628223+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-03-11T17:50:55.772 INFO:scrape:3 jobs: ['7590755', '7590725', '7590707'] 2024-03-11T17:50:55.772 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2024-03-11T17:50:55.772 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', '5-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-03-11T17:50:55.772 INFO:scrape: 2024-03-11T17:50:55.772 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-11T17:50:55.772 INFO:scrape:1 jobs: ['7590715'] 2024-03-11T17:50:55.772 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-03-11T17:50:55.772 INFO:scrape: 2024-03-11T17:50:55.773 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2024-03-11T17:50:55.773 INFO:scrape:1 jobs: ['7590728'] 2024-03-11T17:50:55.773 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/acls}'] 2024-03-11T17:50:55.773 INFO:scrape: 2024-03-11T17:50:55.773 INFO:scrape:Failure: Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) 2024-03-11T17:50:55.773 INFO:scrape:1 jobs: ['7590742'] 2024-03-11T17:50:55.773 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/snapshot}}'] 2024-03-11T17:50:55.773 INFO:scrape: 2024-03-11T17:50:55.781 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/ failed: 39 dead: 2 running: 0 waiting: 0 queued: 0 passed: 17 Fail ================================================================= [7590703] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/openfiletable} ----------------------------------------------------------------- time: 00:14:06 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590703/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590703/ SELinux denials found on ubuntu@smithi173.front.sepia.ceph.com: ['type=AVC msg=audit(1710133558.698:197): avc: denied { checkpoint_restore } for pid=1148 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7590704] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:38:33 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590704/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f25b8f6865d48a2b568e1e9482d3e08 Command failed (workunit test kernel_untar_build.sh) on smithi049 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7590707] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:40:29 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590707/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590707/ "2024-03-11T05:36:47.156176+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7590711] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quota} ----------------------------------------------------------------- time: 00:15:59 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590711/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590711/ SELinux denials found on ubuntu@smithi033.front.sepia.ceph.com: ['type=AVC msg=audit(1710134766.177:199): avc: denied { checkpoint_restore } for pid=1208 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7590715] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:42:14 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590715/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=651c3a2fcd254d74b29790cc7194db9d Command failed (workunit test suites/ffsb.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7590716] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:24:18 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590716/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590716/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aabf1e7b75ed4d0eae0e1f09bc0b4900 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7590718] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:23:50 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590718/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590718/ sentry: https://sentry.ceph.com/organizations/ceph/?query=730d8bbad17645a09215a91317cb26ff Command failed (workunit test suites/fsx.sh) on smithi062 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7590719] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:48:14 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590719/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590719/ reached maximum tries (51) after waiting for 300 seconds [7590720] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:17:18 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590720/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590720/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65d8b04d6fc5425498b3bd8a9b3d7d47 Command failed (workunit test suites/pjd.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/pjd.sh' [7590721] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:53:14 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590721/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590721/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7590722] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:37:52 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590722/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590722/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a3ffc3f61c941b9acf95456211227b4 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7590723] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:43:05 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590723/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590723/ SELinux denials found on ubuntu@smithi115.front.sepia.ceph.com: ['type=AVC msg=audit(1710135055.071:208): avc: denied { checkpoint_restore } for pid=1205 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7590725] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:25:19 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590725/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590725/ "2024-03-11T05:40:08.649688+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7590726] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:12 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590726/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590726/ sentry: https://sentry.ceph.com/organizations/ceph/?query=edc9f904cdb947eab68cd554e33666d4 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7590727] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:15 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590727/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590727/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdc27f85c7f8460c99234c739eb435f6 Command failed (workunit test suites/pjd.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7590728] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/acls} ----------------------------------------------------------------- time: 00:15:57 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590728/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590728/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46052b52e647471ea619cc095cbc9ecb Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [7590729] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:45:08 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590729/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590729/ reached maximum tries (51) after waiting for 300 seconds [7590730] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:23:53 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590730/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590730/ sentry: https://sentry.ceph.com/organizations/ceph/?query=125f49359d674d0185b1127bbbcf6aff Command failed (workunit test kernel_untar_build.sh) on smithi120 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7590731] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:33:34 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590731/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590731/ sentry: https://sentry.ceph.com/organizations/ceph/?query=366c9b8c09034a74802647bce879b4a4 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7590734] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:37:11 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590734/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590734/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01ee5d3edbc545f1bfee4c4a41309f5f Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7590735] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:45:52 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590735/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590735/ reached maximum tries (51) after waiting for 300 seconds [7590736] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:52:59 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590736/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590736/ valgrind error: Leak_StillReachable malloc malloc strdup [7590737] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:41:14 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590737/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590737/ sentry: https://sentry.ceph.com/organizations/ceph/?query=259df56d5f184476a32d5ddd6e1e15c1 Command failed (workunit test suites/fsx.sh) on smithi006 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7590738] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:30:55 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590738/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07ff84351e6443ad89701bb2b2da1ce8 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi090 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7590739] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/exports} ----------------------------------------------------------------- time: 00:55:18 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590739/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590739/ SELinux denials found on ubuntu@smithi105.front.sepia.ceph.com: ['type=AVC msg=audit(1710135902.834:199): avc: denied { checkpoint_restore } for pid=1206 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7590740] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:40:09 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590740/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590740/ SELinux denials found on ubuntu@smithi144.front.sepia.ceph.com: ['type=AVC msg=audit(1710135935.829:210): avc: denied { checkpoint_restore } for pid=1263 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7590741] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} ----------------------------------------------------------------- time: 00:13:41 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590741/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc9060f822ba4a5db974a4ca4db14aae Command failed (workunit test fs/fscrypt.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none pjd' [7590742] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:12:58 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590742/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d95cb4fa733459682489fd2ee765f90 Test failure: test_clean_stale_subvolume_snapshot_metadata (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) [7590744] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:45:54 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590744/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590744/ reached maximum tries (51) after waiting for 300 seconds [7590747] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:45:15 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590747/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3051029be384e788bdb59a3e3a1410a Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7590748] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:30:30 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590748/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dbcadd3ef5224f7d9be93df5ae088bbd Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7590749] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:33:16 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590749/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7855bdcb91874560a4fd8159c1baa1d7 Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) [7590750] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:07:14 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590750/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590750/ valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7590752] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:38 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590752/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590752/ reached maximum tries (51) after waiting for 300 seconds [7590753] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:33:52 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590753/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65aff0f8f62847918ca0ea9e6a62f63d Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi170 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7590754] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:59:28 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590754/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590754/ error during scrub thrashing: rank damage found: {'backtrace'} [7590755] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:05:09 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590755/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590755/ "2024-03-11T06:10:22.628223+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7590756] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/pool-perm} ----------------------------------------------------------------- time: 00:15:03 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590756/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54573809d7964c1c94b7169c138945f8 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7590757] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:56 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590757/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590757/ "1710138000.0001762 mon.smithi073 (mon.0) 777 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log Dead ================================================================= [7590724] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590724/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590724/ hit max job timeout [7590745] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590745/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590745/ hit max job timeout Pass ================================================================= [7590700] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:20:59 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590700/ [7590701] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:40 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590701/ [7590702] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:42:31 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590702/ [7590705] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:39:07 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590705/ [7590706] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:37:57 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590706/ [7590708] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:19:11 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590708/ [7590709] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:13:50 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590709/ [7590710] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:16 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590710/ [7590712] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:14:12 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590712/ [7590713] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:04:22 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590713/ [7590714] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 02:00:48 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590714/ [7590717] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap_schedule_snapdir} time: 00:11:17 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590717/ [7590732] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:15:40 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590732/ [7590733] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:35 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590733/ [7590743] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/fragment} time: 00:16:33 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590743/ [7590746] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:45 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590746/ [7590751] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:55:36 info: https://pulpito.ceph.com/vshankar-2024-03-11_04:49:03-fs-main-testing-default-smithi/7590751/