2024-03-24T21:24:58.123 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-24T21:24:58.124 INFO:teuthology.results:subset: '179/512' 2024-03-24T21:24:58.124 INFO:teuthology.results:seed: '288' 2024-03-25T08:26:25.144 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-25T08:26:25.177 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-25T19:09:24.640 INFO:teuthology.results:Tests finished! gathering results... 2024-03-25T19:09:45.950 INFO:scrape:Found 244 jobs 2024-03-25T19:09:45.950 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620157/teuthology.log 2024-03-25T19:09:57.846 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:57.846 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:57.847 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:57.847 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.698 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.699 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.699 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.699 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.865 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.865 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.865 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.865 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.888 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.888 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.888 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:09:59.888 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:04.503 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:04.503 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:04.503 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:04.503 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.931 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.932 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.932 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.932 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.945 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.946 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.946 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.946 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.993 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.993 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.993 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:25.993 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.460 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.461 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.461 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.461 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.485 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.486 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.486 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:28.486 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.564 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.564 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.564 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.564 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.568 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.568 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.568 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.568 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.627 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.627 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.627 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.628 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.632 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.632 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.632 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.632 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.636 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.636 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.636 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.636 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.640 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.641 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.641 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.641 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.712 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.712 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.712 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.713 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.716 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.716 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.716 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.716 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.720 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.720 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.720 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.720 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.724 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.724 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.724 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.724 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.728 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.728 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.728 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.729 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.733 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.733 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.733 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.733 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.801 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.801 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.801 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.802 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.805 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.805 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.805 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.805 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.809 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.809 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.809 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.810 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.813 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.813 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.814 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.814 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.817 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.818 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.818 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.818 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.822 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.822 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.822 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.822 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.890 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.891 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.891 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.891 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.894 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.895 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.895 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.895 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.899 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.899 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.899 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.899 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.903 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.903 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.903 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.903 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.907 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.907 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.907 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.908 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.911 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.912 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.912 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.912 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.983 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.983 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.983 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.983 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.987 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.987 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.987 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.987 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.991 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.992 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.992 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.992 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.996 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.996 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.996 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:31.996 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.000 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.000 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.000 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.000 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.004 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.005 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.005 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.005 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.073 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.073 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.074 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.074 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.078 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.078 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.078 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.078 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.082 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.082 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.082 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.082 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.086 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.086 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.086 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.086 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.090 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.090 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.090 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.091 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.095 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.095 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.095 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.095 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.161 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.161 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.161 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.161 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.165 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.165 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.165 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.165 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.169 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.169 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.169 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.169 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.173 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.173 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.173 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.173 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.177 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.177 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.177 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.177 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.181 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.181 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.181 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.181 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.248 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.248 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.248 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.248 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.252 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.252 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.252 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.252 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.256 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.256 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.256 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.257 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.260 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.260 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.261 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.261 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.264 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.265 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.265 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.265 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.269 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.269 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.269 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.269 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.337 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.337 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.337 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.337 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.341 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.341 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.341 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.341 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.345 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.345 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.345 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.345 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.349 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.349 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.349 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.349 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.353 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.353 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.353 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.353 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.357 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.357 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.357 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.357 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.426 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.426 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.426 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.426 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.430 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.430 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.430 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.430 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.434 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.434 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.435 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.435 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.439 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.439 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.439 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.439 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.443 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.443 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.443 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.443 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.447 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.447 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.447 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.447 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.514 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.515 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.515 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.515 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.518 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.518 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.518 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.518 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.522 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.522 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.522 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.522 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.526 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.526 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.526 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.527 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.530 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.530 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.530 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.531 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.534 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.534 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.534 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.535 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.602 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.602 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.602 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.602 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.606 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.606 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.606 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.606 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.610 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.610 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.610 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.610 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.614 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.614 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.614 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.615 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.618 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.618 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.619 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.619 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.623 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.623 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.623 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.623 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.691 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.691 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.691 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.691 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.695 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.695 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.695 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.696 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.699 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.699 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.700 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.700 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.703 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.704 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.704 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.704 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.708 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.708 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.708 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.708 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.712 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.712 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.712 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.712 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.779 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.779 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.780 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.780 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.783 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.783 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.783 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.783 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.787 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.787 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.787 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.788 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.791 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.792 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.792 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.792 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.796 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.796 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.796 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.796 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.800 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.800 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.800 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.800 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.867 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.867 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.867 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.867 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.871 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.871 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.871 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.871 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.875 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.875 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.875 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.875 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.879 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.879 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.879 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.879 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.883 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.883 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.883 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.883 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.887 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.887 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.887 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.887 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.955 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.956 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.956 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.956 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.960 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.960 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.960 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.960 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.964 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.964 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.964 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.964 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.968 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.968 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.968 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.969 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.973 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.973 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.973 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.973 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.977 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.977 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.978 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:32.978 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.046 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.047 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.047 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.047 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.050 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.051 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.051 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.051 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.055 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.055 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.055 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.055 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.059 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.059 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.059 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.059 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.063 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.063 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.063 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.063 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.067 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.067 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.067 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.067 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.134 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.134 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.134 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.134 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.138 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.138 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.138 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.138 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.142 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.142 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.142 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.142 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.146 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.146 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.146 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.146 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.150 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.150 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.150 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.150 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.154 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.154 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.154 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:10:33.154 WARNING:scrape:Saw end of BT but not start 2024-03-25T19:11:08.541 INFO:scrape:Found 27 distinct failure reasons 2024-03-25T19:11:08.542 INFO:scrape:Dead: None 2024-03-25T19:11:08.542 INFO:scrape:8 jobs: ['7620157', '7620156', '7620160', '7620173', '7620056', '7620112', '7620057', '7620172'] 2024-03-25T19:11:08.542 INFO:scrape: 2024-03-25T19:11:08.542 INFO:scrape:Failure: "2024-03-25T06:46:19.370948+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-03-25T19:11:08.542 INFO:scrape:44 jobs 2024-03-25T19:11:08.543 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-25T19:11:08.544 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-25T19:11:08.544 INFO:scrape: 2024-03-25T19:11:08.544 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-25T19:11:08.544 INFO:scrape:4 jobs: ['7619986', '7620132', '7620003', '7619960'] 2024-03-25T19:11:08.544 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-25T19:11:08.544 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-25T19:11:08.544 INFO:scrape: 2024-03-25T19:11:08.544 INFO:scrape:Failure: Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) 2024-03-25T19:11:08.544 INFO:scrape:1 jobs: ['7619946'] 2024-03-25T19:11:08.545 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/misc}}'] 2024-03-25T19:11:08.545 INFO:scrape: 2024-03-25T19:11:08.545 INFO:scrape:Failure: "1711319336.5355825 mon.smithi067 (mon.0) 687 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log 2024-03-25T19:11:08.545 INFO:scrape:4 jobs: ['7619987', '7619938', '7620068', '7620026'] 2024-03-25T19:11:08.545 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-25T19:11:08.545 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-25T19:11:08.545 INFO:scrape: 2024-03-25T19:11:08.545 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-25T19:11:08.545 INFO:scrape:1 jobs: ['7619970'] 2024-03-25T19:11:08.546 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-25T19:11:08.546 INFO:scrape: 2024-03-25T19:11:08.546 INFO:scrape:Failure: SSH connection to smithi189 was lost: 'set -ex\ndd if=/proc/self/mounts of=/dev/stdout' 2024-03-25T19:11:08.546 INFO:scrape:2 jobs: ['7620118', '7620174'] 2024-03-25T19:11:08.546 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-03-25T19:11:08.546 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'kclient-overrides/{distro/stock/{centos_9.stream', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/multifs-auth}'] 2024-03-25T19:11:08.546 INFO:scrape: 2024-03-25T19:11:08.546 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-25T19:11:08.546 INFO:scrape:4 jobs: ['7620102', '7620051', '7620006', '7619959'] 2024-03-25T19:11:08.547 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-25T19:11:08.547 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-25T19:11:08.547 INFO:scrape: 2024-03-25T19:11:08.547 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi083 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-25T19:11:08.547 INFO:scrape:5 jobs: ['7619990', '7620105', '7619935', '7620154', '7620043'] 2024-03-25T19:11:08.547 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-25T19:11:08.547 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-25T19:11:08.547 INFO:scrape: 2024-03-25T19:11:08.547 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi107 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-25T19:11:08.548 INFO:scrape:1 jobs: ['7620015'] 2024-03-25T19:11:08.548 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2024-03-25T19:11:08.548 INFO:scrape: 2024-03-25T19:11:08.548 INFO:scrape:Failure: Command failed on smithi178 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.14 deep-scrub' 2024-03-25T19:11:08.548 INFO:scrape:4 jobs: ['7620022', '7620076', '7620146', '7619954'] 2024-03-25T19:11:08.548 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-03-25T19:11:08.548 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-03-25T19:11:08.548 INFO:scrape: 2024-03-25T19:11:08.548 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1711344520.043:199): avc: denied { checkpoint_restore } for pid=1202 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-25T19:11:08.548 INFO:scrape:5 jobs: ['7620035', '7619995', '7620023', '7619944', '7620103'] 2024-03-25T19:11:08.549 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2024-03-25T19:11:08.549 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'session_timeout}', 'subvol_versions/create_subvol_version_v1', 'tasks/client-limits}', 'tasks/dbench', 'tasks/snap-schedule}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/lockdep}'] 2024-03-25T19:11:08.549 INFO:scrape: 2024-03-25T19:11:08.549 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-25T19:11:08.549 INFO:scrape:1 jobs: ['7619961'] 2024-03-25T19:11:08.549 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/pool-perm}'] 2024-03-25T19:11:08.549 INFO:scrape: 2024-03-25T19:11:08.549 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-25T19:11:08.549 INFO:scrape:2 jobs: ['7620081', '7620145'] 2024-03-25T19:11:08.549 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-25T19:11:08.550 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-25T19:11:08.550 INFO:scrape: 2024-03-25T19:11:08.550 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-03-25T19:11:08.550 INFO:scrape:1 jobs: ['7620039'] 2024-03-25T19:11:08.550 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-03-25T19:11:08.550 INFO:scrape: 2024-03-25T19:11:08.550 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} 2024-03-25T19:11:08.550 INFO:scrape:1 jobs: ['7620121'] 2024-03-25T19:11:08.550 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-03-25T19:11:08.550 INFO:scrape: 2024-03-25T19:11:08.551 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh' 2024-03-25T19:11:08.551 INFO:scrape:1 jobs: ['7620084'] 2024-03-25T19:11:08.551 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2024-03-25T19:11:08.551 INFO:scrape: 2024-03-25T19:11:08.551 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-25T19:11:08.551 INFO:scrape:1 jobs: ['7620027'] 2024-03-25T19:11:08.551 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-03-25T19:11:08.551 INFO:scrape: 2024-03-25T19:11:08.551 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-03-25T19:11:08.551 INFO:scrape:1 jobs: ['7620097'] 2024-03-25T19:11:08.551 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-25T19:11:08.551 INFO:scrape: 2024-03-25T19:11:08.552 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-03-25T19:11:08.552 INFO:scrape:1 jobs: ['7620007'] 2024-03-25T19:11:08.552 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-03-25T19:11:08.552 INFO:scrape: 2024-03-25T19:11:08.552 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) 2024-03-25T19:11:08.552 INFO:scrape:2 jobs: ['7620100', '7620091'] 2024-03-25T19:11:08.552 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse'] 2024-03-25T19:11:08.552 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}'] 2024-03-25T19:11:08.552 INFO:scrape: 2024-03-25T19:11:08.552 INFO:scrape:Failure: Stale jobs detected, aborting. 2024-03-25T19:11:08.552 INFO:scrape:1 jobs: ['7620168'] 2024-03-25T19:11:08.552 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-25T19:11:08.553 INFO:scrape: 2024-03-25T19:11:08.553 INFO:scrape:Failure: Test failure: test_async_subvolume_rm (tasks.cephfs.test_volumes.TestSubvolumes) 2024-03-25T19:11:08.553 INFO:scrape:1 jobs: ['7620011'] 2024-03-25T19:11:08.553 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}'] 2024-03-25T19:11:08.553 INFO:scrape: 2024-03-25T19:11:08.553 INFO:scrape:Failure: Command failed on smithi028 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23f8b496-ea81-11ee-95cd-87774f69a715 -e sha1=f73f1dfd96849300c4322bd7259a3ab37f94c200 -- bash -c 'ceph orch ps'" 2024-03-25T19:11:08.553 INFO:scrape:1 jobs: ['7620150'] 2024-03-25T19:11:08.553 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-25T19:11:08.553 INFO:scrape: 2024-03-25T19:11:08.553 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:11:08.553 INFO:scrape:2 jobs: ['7619988', '7620079'] 2024-03-25T19:11:08.553 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2024-03-25T19:11:08.554 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-25T19:11:08.554 INFO:scrape: 2024-03-25T19:11:08.554 INFO:scrape:Failure: ['Data could not be sent to remote host "smithi183.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi183.front.sepia.ceph.com port 22: No route to host', 'ubuntu@smithi183.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).'] 2024-03-25T19:11:08.554 INFO:scrape:1 jobs: ['7620161'] 2024-03-25T19:11:08.554 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/fs/test_o_trunc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-03-25T19:11:08.554 INFO:scrape: 2024-03-25T19:11:08.554 INFO:scrape:Failure: Command failed (workunit test suites/iogen.sh) on smithi006 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh' 2024-03-25T19:11:08.554 INFO:scrape:1 jobs: ['7620053'] 2024-03-25T19:11:08.554 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-03-25T19:11:08.554 INFO:scrape: 2024-03-25T19:11:08.567 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/ failed: 92 dead: 9 running: 0 waiting: 0 queued: 0 passed: 143 Fail ================================================================= [7619935] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:28:07 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619935/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619935/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3e716f25bcb4b5bb6a43c1f9a10f77f Command failed (workunit test suites/fsx.sh) on smithi031 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7619938] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619938/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619938/ "1711317270.0127258 mon.smithi090 (mon.0) 717 : cluster [WRN] Replacing daemon mds.cephfs.smithi090.pzplnv as rank 0 with standby daemon mds.cephfs.smithi090.ytefeh" in cluster log [7619944] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:25:34 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619944/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619944/ SELinux denials found on ubuntu@smithi092.front.sepia.ceph.com: ['type=AVC msg=audit(1711316543.463:199): avc: denied { checkpoint_restore } for pid=1168 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7619945] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:36:38 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619945/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619945/ "2024-03-24T21:55:26.297763+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619946] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:21:29 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619946/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619946/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec00dd0ffba74c03b39bdebcf834a141 Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc) [7619947] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:53:34 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619947/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619947/ "2024-03-24T21:53:18.080369+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619950] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:28:32 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619950/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619950/ "2024-03-24T21:54:45.069188+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619951] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:32:08 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619951/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619951/ "2024-03-24T21:54:09.891674+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619954] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:50 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619954/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619954/ Command failed on smithi107 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.39 deep-scrub' [7619955] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:27:14 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619955/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619955/ "2024-03-24T21:57:45.369224+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619957] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:03:10 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619957/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619957/ "2024-03-24T21:57:31.250850+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619959] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:38:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619959/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619959/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab507a5b7fa64461ac062c832c30b8da Command failed (workunit test suites/fsstress.sh) on smithi122 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7619960] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:52:37 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619960/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619960/ error during scrub thrashing: rank damage found: {'backtrace'} [7619961] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/pool-perm} ----------------------------------------------------------------- time: 00:10:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619961/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619961/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ca541ac408b42118e940da422aaf692 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7619970] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:12:36 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619970/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619970/ sentry: https://sentry.ceph.com/organizations/ceph/?query=baf0b0d0a9f249ca8b4d05b2431c2d51 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7619972] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:42:35 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619972/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619972/ "2024-03-24T22:21:25.582381+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619975] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:23:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619975/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619975/ "2024-03-24T22:18:54.396081+0000 mon.a (mon.0) 187 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619980] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:36:45 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619980/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619980/ "2024-03-24T22:25:51.807787+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619986] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:31:27 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619986/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619986/ error during scrub thrashing: rank damage found: {'backtrace'} [7619987] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:08 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619987/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619987/ "1711319336.5355825 mon.smithi067 (mon.0) 687 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7619988] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:26:03 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619988/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2617032f900e4a9fa771be03844287f0 valgrind error: Leak_PossiblyLost calloc __trans_list_add [7619990] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:32:20 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619990/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bca7bd8ed024579ab91e1c1baf1b0f8 Command failed (workunit test suites/fsx.sh) on smithi083 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7619992] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:29:15 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619992/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619992/ "2024-03-24T22:37:33.837430+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7619995] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:11:30 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619995/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619995/ SELinux denials found on ubuntu@smithi184.front.sepia.ceph.com: ['type=AVC msg=audit(1711319096.931:199): avc: denied { checkpoint_restore } for pid=1186 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7619997] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:38:40 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619997/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619997/ "2024-03-24T22:41:00.595440+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620002] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:33:15 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620002/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620002/ "2024-03-25T05:09:45.860677+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620003] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:28:22 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620003/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620003/ error during scrub thrashing: rank damage found: {'backtrace'} [7620006] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:40:34 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620006/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620006/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa5385cdae574900aee5e61202b6eedb Command failed (workunit test suites/fsstress.sh) on smithi096 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7620007] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 01:41:42 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620007/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620007/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fbee11c99778482283395f2184e97795 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7620011] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 01:08:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620011/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620011/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ded71829fe7c4ac49c2efe6a5f03bed9 Test failure: test_async_subvolume_rm (tasks.cephfs.test_volumes.TestSubvolumes) [7620013] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:12:46 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620013/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620013/ "2024-03-25T05:25:09.312258+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620015] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:27:10 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620015/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620015/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56d1af95c4b0429cb7842d9f14670f43 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi107 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7620017] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:29:29 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620017/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620017/ "2024-03-25T05:31:32.168172+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620019] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:41:57 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620019/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620019/ "2024-03-25T05:32:40.151424+0000 mon.a (mon.0) 194 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620022] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:20:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620022/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620022/ Command failed on smithi178 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.14 deep-scrub' [7620023] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:13:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620023/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620023/ SELinux denials found on ubuntu@smithi117.front.sepia.ceph.com: ['type=AVC msg=audit(1711344228.585:200): avc: denied { checkpoint_restore } for pid=1213 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7620026] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:20 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620026/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620026/ "1711345162.4547343 mon.smithi118 (mon.0) 686 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7620027] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:38:58 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620027/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620027/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15975cfd09cb44b6aab10cadd62e6b98 Command failed (workunit test suites/dbench.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7620032] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:08:25 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620032/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620032/ "2024-03-25T05:45:08.914049+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620035] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:45:22 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620035/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620035/ SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1711344520.043:199): avc: denied { checkpoint_restore } for pid=1202 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7620037] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:36:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620037/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620037/ "2024-03-25T05:56:01.662523+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620039] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 08:10:21 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620039/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620039/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3f20a254fca4eaba6875a6a2eb58aca Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7620043] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:35:53 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620043/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620043/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88e7219d816a41a5a9080bf5fe7ce600 Command failed (workunit test suites/fsx.sh) on smithi105 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7620045] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:28:02 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620045/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620045/ "2024-03-25T06:27:04.360624+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620047] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:37:31 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620047/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620047/ "2024-03-25T06:28:31.308380+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620051] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:38:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620051/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620051/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27c2fdf95a5649f2a2a0fdf866aeb4a4 Command failed (workunit test suites/fsstress.sh) on smithi042 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7620053] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 01:03:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620053/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620053/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04279fd4c86b41489e4467f46e9a3efd Command failed (workunit test suites/iogen.sh) on smithi006 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh' [7620055] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:36:55 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620055/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620055/ "2024-03-25T06:43:36.980951+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620058] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:43 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620058/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620058/ "2024-03-25T06:40:01.677790+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620063] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:34:54 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620063/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620063/ "2024-03-25T06:47:28.702817+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620066] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:01:17 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620066/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620066/ "2024-03-25T06:46:19.370948+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620068] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:04 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620068/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620068/ "1711349806.1696603 mon.smithi033 (mon.0) 711 : cluster [WRN] Replacing daemon mds.cephfs.smithi033.bxpqge as rank 0 with standby daemon mds.cephfs.smithi033.eibqqy" in cluster log [7620070] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:27:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620070/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620070/ "2024-03-25T06:54:31.554446+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620072] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:03:11 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620072/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620072/ "2024-03-25T06:54:51.519533+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620075] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:59:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620075/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620075/ "2024-03-25T06:58:54.989192+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620076] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:34 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620076/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620076/ Command failed on smithi105 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.3a deep-scrub' [7620079] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:40:21 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620079/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620079/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7620080] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:38:38 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620080/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620080/ "2024-03-25T06:59:44.822268+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620081] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:38:02 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620081/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620081/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20b483defc574a8180cf327be9624d73 Command failed (workunit test suites/ffsb.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7620084] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:08:33 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620084/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620084/ sentry: https://sentry.ceph.com/organizations/ceph/?query=30525b9545b0470d8eafdedd9a9d9a89 Command failed (workunit test fs/snaps/snaptest-realm-split.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm- split.sh' [7620085] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:25:45 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620085/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620085/ "2024-03-25T07:08:54.339774+0000 mon.a (mon.0) 187 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620091] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} ----------------------------------------------------------------- time: 00:24:17 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620091/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d057dad60e14e43aee3876de638e3d4 Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) [7620097] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:35:52 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620097/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620097/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff6a97d409c54c61b8381ead4d32a16f Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7620100] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:27:25 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620100/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=32b52824394d449c9e788517f859b52c Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) [7620101] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:36:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620101/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620101/ "2024-03-25T07:42:14.706013+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620102] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:17 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620102/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620102/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2c0268573024a68b04ac5bb091afb32 Command failed (workunit test suites/fsstress.sh) on smithi118 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7620103] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-limits} ----------------------------------------------------------------- time: 00:22:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620103/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620103/ SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1711351838.534:199): avc: denied { checkpoint_restore } for pid=1206 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7620105] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:25:53 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620105/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=feebecadd877403b98d899ee5d9db703 Command failed (workunit test suites/fsx.sh) on smithi033 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7620107] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:35:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620107/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620107/ "2024-03-25T07:49:46.988632+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620110] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:35:39 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620110/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620110/ "2024-03-25T07:53:13.184585+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620115] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:36:05 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620115/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620115/ "2024-03-25T07:56:01.018481+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620117] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:25:03 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620117/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620117/ "2024-03-25T07:55:06.375818+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620118] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 11:13:47 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620118/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620118/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00a6fb1881a3430aa6e76ef088959d23 SSH connection to smithi189 was lost: 'set -ex\ndd if=/proc/self/mounts of=/dev/stdout' [7620121] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620121/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620121/ The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} [7620124] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:33:34 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620124/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620124/ "2024-03-25T08:01:49.070097+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620128] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:52:21 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620128/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620128/ "2024-03-25T08:01:39.904018+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620132] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:29:11 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620132/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620132/ error during scrub thrashing: rank damage found: {'backtrace'} [7620133] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:46:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620133/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620133/ "2024-03-25T08:16:08.630075+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620140] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:39:28 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620140/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620140/ "2024-03-25T08:24:48.469020+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620142] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:25:07 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620142/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620142/ "2024-03-25T08:24:10.789259+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620145] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:55:36 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620145/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620145/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8c58a255b444111958a9e7173bf44c1 Command failed (workunit test suites/ffsb.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7620146] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:05 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620146/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620146/ Command failed on smithi005 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.3a deep-scrub' [7620148] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:28:47 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620148/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620148/ "2024-03-25T08:31:22.440477+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620150] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:51:27 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620150/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36991ddefcae49f186f431a463ec0182 Command failed on smithi028 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23f8b496-ea81-11ee-95cd-87774f69a715 -e sha1=f73f1dfd96849300c4322bd7259a3ab37f94c200 -- bash -c 'ceph orch ps'" [7620151] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:33:42 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620151/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620151/ "2024-03-25T08:28:38.483800+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620154] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:27:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620154/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620154/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea3ffca93c4f45e2a39405a1025af828 Command failed (workunit test suites/fsx.sh) on smithi001 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5b534581aae05db8101ae105c576bb56c54e9c12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7620158] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:32:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620158/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620158/ "2024-03-25T08:31:18.121053+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620161] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:21:01 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620161/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620161/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f99e22db6df2497485fe9f9a30753442 ['Data could not be sent to remote host "smithi183.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi183.front.sepia.ceph.com port 22: No route to host', 'ubuntu@smithi183.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).'] [7620165] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:53:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620165/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620165/ "2024-03-25T08:38:44.632875+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620168] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620168/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620168/ sentry: https://sentry.ceph.com/organizations/ceph/?query=baa7594133934e098e2027e69e0ee7b8 Stale jobs detected, aborting. [7620169] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:28:58 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620169/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620169/ "2024-03-25T08:47:29.679547+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7620171] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:32:22 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620171/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620171/ "2024-03-25T08:43:54.257396+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log Dead ================================================================= [7620056] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620056/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620056/ Error reimaging machines: Failed to power on smithi094 [7620057] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620057/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620057/ Error reimaging machines: Failed to power on smithi183 [7620112] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-recovery} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620112/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620112/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7620156] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620156/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620156/ Error reimaging machines: Expected smithi183's OS to be ubuntu 22.04 but found centos 9 [7620157] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620157/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620157/ Error reimaging machines: list index out of range [7620160] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620160/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620160/ Error reimaging machines: Expected smithi183's OS to be ubuntu 22.04 but found centos 9 [7620172] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620172/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620172/ Error reimaging machines: Expected smithi192's OS to be centos 9 but found ubuntu 22.04 [7620173] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620173/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620173/ Error reimaging machines: list index out of range [7620174] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620174/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620174/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f8f6186393074daa8a866e39ef509eb2 SSH connection to smithi186 was lost: 'sudo dpkg -i /tmp/linux-image.deb' Pass ================================================================= [7619931] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:38:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619931/ [7619932] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:48:11 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619932/ [7619933] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:34:06 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619933/ [7619934] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:43 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619934/ [7619936] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:13:55 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619936/ [7619937] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-full} time: 00:19:28 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619937/ [7619939] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:51:40 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619939/ [7619940] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:31:41 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619940/ [7619941] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:23 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619941/ [7619942] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:16 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619942/ [7619943] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds_creation_retry} time: 00:13:12 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619943/ [7619948] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/metrics} time: 00:26:16 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619948/ [7619949] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:31:33 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619949/ [7619952] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/multimds_misc} time: 00:14:40 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619952/ [7619953] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:54:44 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619953/ [7619956] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/openfiletable} time: 00:13:40 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619956/ [7619958] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:12:10 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619958/ [7619962] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:55:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619962/ [7619963] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:16:10 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619963/ [7619964] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:43 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619964/ [7619965] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:19:14 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619965/ [7619966] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:27:04 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619966/ [7619967] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:35 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619967/ [7619968] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:17:01 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619968/ [7619969] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619969/ [7619971] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:20:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619971/ [7619973] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quota} time: 00:12:06 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619973/ [7619974] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:32 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619974/ [7619976] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:18 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619976/ [7619977] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:33 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619977/ [7619978] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:45:54 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619978/ [7619979] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:31:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619979/ [7619981] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} time: 00:12:14 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619981/ [7619982] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:11:35 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619982/ [7619983] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:33:02 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619983/ [7619984] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:38:30 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619984/ [7619985] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/scrub} time: 00:28:27 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619985/ [7619989] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:12:39 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619989/ [7619991] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/sessionmap} time: 00:11:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619991/ [7619993] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:43 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619993/ [7619994] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:33:14 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619994/ [7619996] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:48 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619996/ [7619998] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619998/ [7619999] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:13:32 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7619999/ [7620000] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:11:58 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620000/ [7620001] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap_schedule_snapdir} time: 00:13:16 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620001/ [7620004] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:13:36 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620004/ [7620005] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snapshots} time: 00:21:56 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620005/ [7620008] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:55 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620008/ [7620009] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:22:17 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620009/ [7620010] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:37:31 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620010/ [7620012] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:23:42 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620012/ [7620014] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} time: 00:32:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620014/ [7620016] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/dbench}} time: 00:50:24 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620016/ [7620018] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/test_journal_migration} time: 00:20:36 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620018/ [7620020] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:37:23 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620020/ [7620021] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:18:13 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620021/ [7620024] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:37:08 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620024/ [7620025] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/truncate_delay} time: 00:13:58 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620025/ [7620028] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:44:24 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620028/ [7620029] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:13:37 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620029/ [7620030] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:13:16 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620030/ [7620031] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620031/ [7620033] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/workunit/dir-max-entries} time: 00:15:43 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620033/ [7620034] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} time: 00:17:45 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620034/ [7620036] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:23:45 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620036/ [7620038] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:26:13 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620038/ [7620040] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:16 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620040/ [7620041] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:34:06 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620041/ [7620042] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:55:51 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620042/ [7620044] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/workunit/quota} time: 00:10:52 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620044/ [7620046] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:14:46 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620046/ [7620048] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:15:33 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620048/ [7620049] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:17:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620049/ [7620050] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:00:45 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620050/ [7620052] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/acls} time: 00:11:36 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620052/ [7620054] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:11:57 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620054/ [7620059] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:03 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620059/ [7620060] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:31 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620060/ [7620061] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:12:31 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620061/ [7620062] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:09:32 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620062/ [7620064] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:34:47 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620064/ [7620065] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:02 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620065/ [7620067] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:38:17 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620067/ [7620069] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/asok_dump_tree} time: 00:13:46 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620069/ [7620071] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:09:11 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620071/ [7620073] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:12:30 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620073/ [7620074] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/auto-repair} time: 00:14:57 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620074/ [7620077] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/backtrace} time: 00:10:01 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620077/ [7620078] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:17:34 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620078/ [7620082] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:22:48 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620082/ [7620083] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/cap-flush} time: 00:08:56 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620083/ [7620086] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:38 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620086/ [7620087] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:09:31 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620087/ [7620088] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:45:01 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620088/ [7620089] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:17:59 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620089/ [7620090] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:09:47 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620090/ [7620092] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:22:06 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620092/ [7620093] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:38 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620093/ [7620094] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:38:31 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620094/ [7620095] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} time: 00:44:33 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620095/ [7620096] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:55 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620096/ [7620098] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:59:10 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620098/ [7620099] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:14:57 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620099/ [7620104] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:48:02 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620104/ [7620106] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:05 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620106/ [7620108] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-readahead} time: 00:10:00 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620108/ [7620109] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:12:43 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620109/ [7620111] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:11:40 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620111/ [7620113] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}} time: 02:34:37 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620113/ [7620114] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:21:58 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620114/ [7620116] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:36 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620116/ [7620119] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:48:24 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620119/ [7620120] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:25:02 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620120/ [7620122] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:10:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620122/ [7620123] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:28 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620123/ [7620125] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 01:18:18 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620125/ [7620126] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:47 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620126/ [7620127] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:04 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620127/ [7620129] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:13:24 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620129/ [7620130] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:17 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620130/ [7620131] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/exports} time: 00:43:13 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620131/ [7620134] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:25:10 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620134/ [7620135] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:47:13 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620135/ [7620136] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} time: 00:23:33 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620136/ [7620137] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:24 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620137/ [7620138] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:13:36 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620138/ [7620139] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:47:24 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620139/ [7620141] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:15:05 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620141/ [7620143] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/fragment} time: 00:17:45 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620143/ [7620144] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:28:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620144/ [7620147] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:13:49 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620147/ [7620149] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:16:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620149/ [7620152] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:13:16 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620152/ [7620153] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds-flush} time: 00:12:14 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620153/ [7620155] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:12:50 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620155/ [7620159] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-full} time: 00:19:18 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620159/ [7620162] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:34:27 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620162/ [7620163] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:50:15 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620163/ [7620164] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:24:26 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620164/ [7620166] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds_creation_retry} time: 00:11:09 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620166/ [7620167] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:24:08 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620167/ [7620170] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/metrics} time: 00:33:38 info: https://pulpito.ceph.com/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620170/