2024-02-19T06:19:31.185 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-19T06:19:31.185 INFO:teuthology.results:subset: '669/1024' 2024-02-19T06:19:31.185 INFO:teuthology.results:seed: '8780' 2024-02-19T07:13:17.879 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-19T07:13:17.907 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-19T18:33:08.357 INFO:teuthology.results:Tests finished! gathering results... 2024-02-19T18:33:15.398 INFO:scrape:Found 86 jobs 2024-02-19T18:33:17.105 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566127/teuthology.log 2024-02-19T18:33:44.468 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.468 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.468 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.468 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.513 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.513 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.513 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.513 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.517 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.517 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.518 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.518 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.623 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.623 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.623 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.623 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.635 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.635 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.635 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.635 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.640 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.640 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.640 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.640 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.743 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.743 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.743 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.743 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.754 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.754 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.754 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.754 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.759 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.759 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.759 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.759 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.857 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.857 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.857 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.857 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.869 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.869 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.869 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.869 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.873 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.873 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.873 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.873 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.971 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.971 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.972 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.972 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.982 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.982 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.983 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.983 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.987 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.987 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.987 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:44.987 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.085 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.085 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.085 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.085 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.097 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.097 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.097 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.097 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.101 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.101 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.101 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.101 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.202 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.202 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.202 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.202 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.214 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.214 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.214 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.214 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.219 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.219 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.219 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.219 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.324 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.324 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.324 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.324 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.335 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.336 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.336 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.336 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.340 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.340 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.340 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.340 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.441 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.441 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.441 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.441 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.452 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.453 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.453 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.453 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.457 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.457 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.457 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.457 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.557 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.557 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.558 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.558 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.569 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.569 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.569 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.570 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.574 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.574 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.574 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.574 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.672 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.672 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.672 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.673 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.684 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.684 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.684 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.684 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.688 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.688 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.688 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.688 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.796 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.796 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.796 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.796 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.808 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.808 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.808 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.808 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.813 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.813 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.813 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.813 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.913 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.913 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.913 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.913 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.924 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.924 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.924 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.924 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.929 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.929 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.929 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:45.929 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.028 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.029 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.029 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.029 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.040 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.040 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.040 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.040 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.044 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.045 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.045 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.045 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.142 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.142 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.143 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.143 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.154 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.154 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.154 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.154 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.158 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.158 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.159 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.159 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.262 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.263 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.263 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.263 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.276 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.277 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.277 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.277 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.281 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.281 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.281 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.282 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.381 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.381 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.381 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.382 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.392 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.393 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.393 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.393 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.397 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.397 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.397 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.397 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.494 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.494 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.494 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.494 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.506 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.506 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.506 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.506 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.510 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.510 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.510 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.510 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.612 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.612 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.613 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.613 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.624 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.624 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.624 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.625 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.629 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.629 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.629 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.629 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.732 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.733 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.733 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.733 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.744 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.744 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.744 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.744 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.749 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.749 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.749 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.749 WARNING:scrape:Saw end of BT but not start 2024-02-19T18:33:46.803 INFO:scrape:Found 20 distinct failure reasons 2024-02-19T18:33:46.803 INFO:scrape:Failure: Command failed on smithi017 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' 2024-02-19T18:33:46.803 INFO:scrape:35 jobs 2024-02-19T18:33:46.804 INFO:scrape:suites intersection: [] 2024-02-19T18:33:46.804 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '1-ranks/1', '1-ranks/2', '1-rename', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-workload/dbench', '3-inline/no', '3-inline/yes', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host', 'distro/single-container-host}', 'export-check', 'fail_fs/no', 'fail_fs/yes', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/ignorelist_health', 'overrides/ignorelist_health}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-02-19T18:33:46.805 INFO:scrape: 2024-02-19T18:33:46.805 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi175.front.sepia.ceph.com: ['type=AVC msg=audit(1708325027.656:203): avc: denied { checkpoint_restore } for pid=1156 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-02-19T18:33:46.805 INFO:scrape:4 jobs: ['7566151', '7566135', '7566110', '7566172'] 2024-02-19T18:33:46.805 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2024-02-19T18:33:46.805 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'session_timeout}', 'subvol_versions/create_subvol_version_v1', 'supported-random-distros$/{centos_latest}', 'tasks/fsstress', 'tasks/fstop}', 'tasks/metrics}', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'validater/lockdep}'] 2024-02-19T18:33:46.805 INFO:scrape: 2024-02-19T18:33:46.805 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-02-19T18:33:46.805 INFO:scrape:1 jobs: ['7566117'] 2024-02-19T18:33:46.806 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-02-19T18:33:46.806 INFO:scrape: 2024-02-19T18:33:46.806 INFO:scrape:Failure: SSH connection to smithi022 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc4_g8d59ab0bd93c-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/8d59ab0bd93c806231018a6729893bf309624262/centos/9/flavors/default/x86_64/ --input-file=-' 2024-02-19T18:33:46.806 INFO:scrape:1 jobs: ['7566121'] 2024-02-19T18:33:46.806 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/fsync-tester}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-02-19T18:33:46.806 INFO:scrape: 2024-02-19T18:33:46.806 INFO:scrape:Dead: None 2024-02-19T18:33:46.806 INFO:scrape:2 jobs: ['7566127', '7566176'] 2024-02-19T18:33:46.806 INFO:scrape: 2024-02-19T18:33:46.807 INFO:scrape:Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) 2024-02-19T18:33:46.807 INFO:scrape:1 jobs: ['7566133'] 2024-02-19T18:33:46.807 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2024-02-19T18:33:46.807 INFO:scrape: 2024-02-19T18:33:46.807 INFO:scrape:Failure: Command failed on smithi022 with status 1: 'sudo yum upgrade -y linux-firmware' 2024-02-19T18:33:46.807 INFO:scrape:1 jobs: ['7566120'] 2024-02-19T18:33:46.807 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2024-02-19T18:33:46.807 INFO:scrape: 2024-02-19T18:33:46.807 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-02-19T18:33:46.807 INFO:scrape:1 jobs: ['7566119'] 2024-02-19T18:33:46.807 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/pool-perm}'] 2024-02-19T18:33:46.808 INFO:scrape: 2024-02-19T18:33:46.808 INFO:scrape:Dead: b'2024-02-19T18:30:31.734 DEBUG:teuthology.exit:Finished running handlers' 2024-02-19T18:33:46.808 INFO:scrape:1 jobs: ['7566145'] 2024-02-19T18:33:46.808 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-02-19T18:33:46.808 INFO:scrape: 2024-02-19T18:33:46.808 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-02-19T18:33:46.808 INFO:scrape:2 jobs: ['7566182', '7566155'] 2024-02-19T18:33:46.808 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-19T18:33:46.808 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-19T18:33:46.808 INFO:scrape: 2024-02-19T18:33:46.809 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-02-19T18:33:46.809 INFO:scrape:1 jobs: ['7566150'] 2024-02-19T18:33:46.809 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-02-19T18:33:46.809 INFO:scrape: 2024-02-19T18:33:46.809 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-02-19T18:33:46.809 INFO:scrape:1 jobs: ['7566101'] 2024-02-19T18:33:46.809 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-02-19T18:33:46.809 INFO:scrape: 2024-02-19T18:33:46.809 INFO:scrape:Failure: Command failed on smithi172 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2024-02-19T18:33:46.809 INFO:scrape:1 jobs: ['7566128'] 2024-02-19T18:33:46.810 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-02-19T18:33:46.810 INFO:scrape: 2024-02-19T18:33:46.810 INFO:scrape:Failure: Test failure: test_snap_schedule_subvol_and_group_arguments_05 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-02-19T18:33:46.810 INFO:scrape:1 jobs: ['7566130'] 2024-02-19T18:33:46.810 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/snap-schedule}'] 2024-02-19T18:33:46.810 INFO:scrape: 2024-02-19T18:33:46.810 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2024-02-19T18:33:46.810 INFO:scrape:1 jobs: ['7566152'] 2024-02-19T18:33:46.810 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-02-19T18:33:46.810 INFO:scrape: 2024-02-19T18:33:46.810 INFO:scrape:Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) 2024-02-19T18:33:46.811 INFO:scrape:1 jobs: ['7566159'] 2024-02-19T18:33:46.811 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-02-19T18:33:46.811 INFO:scrape: 2024-02-19T18:33:46.811 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-02-19T18:33:46.811 INFO:scrape:1 jobs: ['7566115'] 2024-02-19T18:33:46.811 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2024-02-19T18:33:46.811 INFO:scrape: 2024-02-19T18:33:46.811 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-02-19T18:33:46.811 INFO:scrape:1 jobs: ['7566114'] 2024-02-19T18:33:46.811 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-19T18:33:46.811 INFO:scrape: 2024-02-19T18:33:46.811 INFO:scrape:Failure: ['/dev/vg_nvme: already exists in filesystem', "Creating volume group 'vg_nvme' failed", "Run `vgcreate --help' for more information."] 2024-02-19T18:33:46.812 INFO:scrape:1 jobs: ['7566160'] 2024-02-19T18:33:46.812 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/fs/norstats}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-02-19T18:33:46.812 INFO:scrape: 2024-02-19T18:33:46.812 INFO:scrape:Failure: "2024-02-19T06:44:11.322525+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-02-19T18:33:46.812 INFO:scrape:1 jobs: ['7566154'] 2024-02-19T18:33:46.812 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-02-19T18:33:46.812 INFO:scrape: 2024-02-19T18:33:46.817 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/ failed: 55 dead: 4 running: 0 waiting: 0 queued: 0 passed: 27 Fail ================================================================= [7566101] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:18 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566101/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c8be0030f40434296aeafbaa567dd68 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7566103] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:09:39 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566103/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566103/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6d917eb852c43cbbb9191ccba3fa14f Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566105] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:09:25 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566105/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3ae8c917bb94fe78ca3d1c9200168b4 Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566107] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:42 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566107/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566107/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65443294bb524912a16a5f4bbcf3f21f Command failed on smithi149 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566108] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:13:44 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566108/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566108/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6222f42eb46547238eb04f1c76800f09 Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566110] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/metrics} ----------------------------------------------------------------- time: 00:32:21 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566110/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566110/ SELinux denials found on ubuntu@smithi136.front.sepia.ceph.com: ['type=AVC msg=audit(1708324947.665:203): avc: denied { checkpoint_restore } for pid=1191 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7566111] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:12:16 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566111/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566111/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a41a92cb6414cd0a7d43c5c4a50b53c Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566114] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:58:53 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566114/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566114/ valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7566115] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:32:58 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566115/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566115/ error during scrub thrashing: rank damage found: {'backtrace'} [7566116] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:08 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566116/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566116/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c021e37ff7642b5922231a95eb21d18 Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566117] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 01:33:41 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566117/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566117/ sentry: https://sentry.ceph.com/organizations/ceph/?query=992d05e706044f13b2092761bbfeda04 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7566118] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:12:32 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566118/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566118/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdb15710f1b7444b8ea8d54a072ea2a6 Command failed on smithi053 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566119] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/pool-perm} ----------------------------------------------------------------- time: 00:16:37 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566119/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566119/ sentry: https://sentry.ceph.com/organizations/ceph/?query=caf93771feb44fc59fcd05f6f14b01bd Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7566120] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566120/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566120/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cfed346f3ce14c00be93e860fd6eafb7 Command failed on smithi022 with status 1: 'sudo yum upgrade -y linux- firmware' [7566123] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:35 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566123/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566123/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06ff0142b44d470abec14ae590dcb031 Command failed on smithi023 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566128] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:08:07 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566128/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566128/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8125f8172f144519d25b4192eb855a0 Command failed on smithi172 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7566130] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 01:57:24 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566130/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566130/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bdccdcce269a4182bbde292f71580a4d Test failure: test_snap_schedule_subvol_and_group_arguments_05 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) [7566131] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:13:27 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566131/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566131/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3888a7a9c61e4c6cb3d36596a190189d Command failed on smithi114 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566132] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:07 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566132/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566132/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a5a7eda319b4c6396e0b55cbeed589a Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566133] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:18:06 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566133/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566133/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4fcb602930d4980b3e2b0013294a9b5 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) [7566135] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:18:44 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566135/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566135/ SELinux denials found on ubuntu@smithi086.front.sepia.ceph.com: ['type=AVC msg=audit(1708325003.506:202): avc: denied { checkpoint_restore } for pid=1190 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7566136] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:12:45 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566136/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566136/ sentry: https://sentry.ceph.com/organizations/ceph/?query=158ac514432f45d6a81f07e73aaaa53f Command failed on smithi045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566140] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:10:10 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566140/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566140/ sentry: https://sentry.ceph.com/organizations/ceph/?query=864b32c595cc4d4388f93e7a0a6abec9 Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566141] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:17 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566141/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566141/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18f8b3099b6b47e0be65af21aeed181f Command failed on smithi102 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566142] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:13:39 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566142/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566142/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d867674241b4670bbabac6769b528b5 Command failed on smithi110 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566143] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:13:47 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566143/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566143/ sentry: https://sentry.ceph.com/organizations/ceph/?query=103c3de3959f4f588018abb848d3e202 Command failed on smithi047 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566147] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:13:51 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566147/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566147/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b15a34e75f44d248d488ce5c747ac52 Command failed on smithi099 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566149] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:10:32 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566149/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566149/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26b43ff01eab4718b070dae07eb09fe0 Command failed on smithi179 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566150] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:47:53 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566150/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8f89ce3a77346e19aa2939fc2b4a806 Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7566151] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:15:45 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566151/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566151/ SELinux denials found on ubuntu@smithi175.front.sepia.ceph.com: ['type=AVC msg=audit(1708325027.656:203): avc: denied { checkpoint_restore } for pid=1156 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7566152] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:33:11 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566152/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e9b4ed1314548f0ac1e2bb4fe82a862 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7566154] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:43:34 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566154/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566154/ "2024-02-19T06:44:11.322525+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7566155] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:53:45 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566155/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566155/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7566156] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:13:39 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566156/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566156/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04b302464a0e45f2bc02e4bec42dcefc Command failed on smithi080 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566157] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:29 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566157/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566157/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4357e7a8726f4e8db137b011691fdfc2 Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566158] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:10:23 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566158/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566158/ sentry: https://sentry.ceph.com/organizations/ceph/?query=618f87f16c594e42998ebf0e588ca26e Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566159] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:38:23 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566159/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566159/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ed2eb427e244d76917ebf00658c5284 Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) [7566160] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:06:47 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566160/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566160/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb5980ee93874beab7b39815ac326ed5 ['/dev/vg_nvme: already exists in filesystem', "Creating volume group 'vg_nvme' failed", "Run `vgcreate --help' for more information."] [7566163] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:09:27 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566163/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566163/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5203e060670049c487d73c8c47f12be6 Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566164] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:08:58 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566164/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566164/ sentry: https://sentry.ceph.com/organizations/ceph/?query=318129b823dd4782a66bf592250397fc Command failed on smithi158 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566165] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:09:57 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566165/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566165/ sentry: https://sentry.ceph.com/organizations/ceph/?query=826b5ab05dd1458eaab853e61b265eda Command failed on smithi038 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566166] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:08:58 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566166/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566166/ sentry: https://sentry.ceph.com/organizations/ceph/?query=74c4f28a33b846049fc2fe59611e207c Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566167] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:09:58 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566167/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566167/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5dc88400f7c0484d8b0e3930d0229a37 Command failed on smithi060 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566171] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:09:49 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566171/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566171/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51f185e1886d4fe1a3c62d4623f0350c Command failed on smithi017 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566172] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:15:15 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566172/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566172/ SELinux denials found on ubuntu@smithi090.front.sepia.ceph.com: ['type=AVC msg=audit(1708326773.261:191): avc: denied { checkpoint_restore } for pid=1175 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7566173] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:33 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566173/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566173/ sentry: https://sentry.ceph.com/organizations/ceph/?query=779cf394cc0a414ebb17a132add2cf2b Command failed on smithi073 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566174] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:29 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566174/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566174/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0392ddf5f844f0985101358f745a351 Command failed on smithi019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566175] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:09:09 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566175/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566175/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e395d2ed7b414b1ba11b16663935b9cb Command failed on smithi103 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566179] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:08:58 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566179/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=86f5e00efeb1446ab1d56a2021189fdc Command failed on smithi072 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566180] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:09:38 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566180/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e6ffa39e5bd49de8922339d9b36157d Command failed on smithi077 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566181] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:09:28 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566181/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566181/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68a823e43a714f50b0d40f7960475bc4 Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566182] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:52:37 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566182/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566182/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7566183] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:09:08 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566183/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566183/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d88f750f52b4def98074c528727dc69 Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' [7566185] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:09:55 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566185/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566185/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0310389d9b64e7c98e09f29409de84e Command failed on smithi018 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7566186] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:09:27 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566186/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43a0cfee935b4425b3650b8bde738f52 Command failed on smithi114 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:25296a8f176256aea5ed8b5850b25167dce0ae33 pull' Dead ================================================================= [7566121] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566121/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=550cfdf7e0b74f3b876a7c5ff16eb9ac SSH connection to smithi022 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc4_g8d59ab0bd93c-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/8d 59ab0bd93c806231018a6729893bf309624262/centos/9/flavors/default/x86_64/ --input-file=-' [7566127] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566127/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566127/ Error reimaging machines: Expected smithi172's OS to be centos 8 but found centos 9 [7566145] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566145/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566145/ hit max job timeout [7566176] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566176/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566176/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7566102] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 02:06:25 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566102/ [7566104] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:24:26 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566104/ [7566106] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-full} time: 00:19:20 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566106/ [7566109] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds_creation_retry} time: 00:11:23 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566109/ [7566112] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:18 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566112/ [7566113] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:13 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566113/ [7566122] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:58 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566122/ [7566124] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:33:54 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566124/ [7566125] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/scrub} time: 00:29:37 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566125/ [7566126] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:21:15 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566126/ [7566129] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:16:09 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566129/ [7566134] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:02:19 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566134/ [7566137] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snapshots} time: 00:23:07 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566137/ [7566138] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:32:24 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566138/ [7566139] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} time: 00:32:23 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566139/ [7566144] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:15:05 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566144/ [7566146] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:29 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566146/ [7566148] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:18:49 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566148/ [7566153] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:18 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566153/ [7566161] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:15:18 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566161/ [7566162] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} time: 00:37:59 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566162/ [7566168] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:54:16 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566168/ [7566169] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-readahead} time: 00:12:28 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566169/ [7566170] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:22 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566170/ [7566177] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:56 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566177/ [7566178] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:23:36 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566178/ [7566184] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:36:20 info: https://pulpito.ceph.com/vshankar-2024-02-19_06:18:55-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7566184/