2024-03-29T07:17:09.471 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-29T07:17:09.471 INFO:teuthology.results:subset: '990/1024' 2024-03-29T07:17:09.471 INFO:teuthology.results:seed: '7327' 2024-03-29T10:17:31.378 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-29T10:17:31.395 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-29T12:07:41.402 INFO:teuthology.results:Tests finished! gathering results... 2024-03-29T12:07:59.562 INFO:scrape:Found 215 jobs 2024-03-29T12:08:01.294 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628248/teuthology.log 2024-03-29T12:08:01.474 WARNING:scrape:Misunderstood line: 2024-03-29T12:08:19.533 WARNING:scrape:Misunderstood line: 2024-03-29T12:08:30.109 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.109 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.110 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.110 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.170 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.171 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.171 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.171 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.359 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.359 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.360 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.360 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.362 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.362 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.362 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.363 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.367 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.367 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.368 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.369 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.524 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.525 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.525 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.525 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.586 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.587 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.587 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.587 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.599 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.599 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.600 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.600 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.602 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.602 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.602 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.602 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.604 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.604 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.604 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.604 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.609 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.609 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.609 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.609 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.801 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.801 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.802 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.802 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.860 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.861 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.861 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.861 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.873 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.874 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.874 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.874 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.876 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.876 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.876 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.876 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.878 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.878 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.878 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.878 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.883 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.883 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.883 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:30.883 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.044 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.044 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.045 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.045 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.105 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.105 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.105 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.106 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.118 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.118 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.118 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.119 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.121 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.121 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.121 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.121 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.123 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.123 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.123 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.123 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.128 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.128 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.128 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.128 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.283 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.284 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.284 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.284 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.343 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.343 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.343 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.343 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.355 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.355 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.355 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.355 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.357 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.359 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.359 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.360 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.360 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.364 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.364 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.364 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.364 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.515 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.515 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.515 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.515 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.573 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.574 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.574 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.574 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.586 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.586 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.586 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.586 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.588 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.589 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.590 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.590 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.592 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.592 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.592 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.592 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.596 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.596 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.596 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.597 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.756 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.756 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.756 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.756 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.817 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.818 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.818 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.818 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.829 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.830 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.830 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.830 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.832 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.832 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.832 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.832 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.834 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.834 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.834 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.835 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.839 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.839 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.839 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.839 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.992 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.992 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.992 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:31.993 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.052 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.053 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.053 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.053 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.065 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.066 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.066 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.066 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.068 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.068 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.068 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.068 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.070 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.070 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.070 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.071 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.075 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.075 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.075 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.075 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.230 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.230 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.230 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.230 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.289 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.289 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.289 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.289 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.301 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.301 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.301 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.301 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.303 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.303 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.303 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.303 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.305 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.306 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.306 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.306 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.310 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.310 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.310 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.310 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.473 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.474 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.474 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.474 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.532 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.532 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.532 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.532 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.544 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.545 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.545 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.545 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.547 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.547 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.547 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.547 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.549 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.549 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.549 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.549 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.553 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.553 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.553 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.554 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.731 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.732 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.732 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.732 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.789 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.789 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.789 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.789 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.801 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.801 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.802 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.802 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.804 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.804 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.804 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.804 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.806 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.806 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.806 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.806 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.810 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.811 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.811 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.811 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.965 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.965 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.965 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:32.965 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.025 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.026 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.026 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.026 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.037 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.038 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.038 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.038 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.040 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.040 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.040 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.040 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.042 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.042 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.042 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.042 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.046 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.046 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.047 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.047 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.195 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.195 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.195 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.196 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.253 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.253 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.253 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.253 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.265 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.265 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.265 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.266 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.267 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.267 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.268 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.268 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.270 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.270 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.270 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.270 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.274 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.274 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.274 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.274 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.423 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.423 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.423 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.424 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.482 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.482 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.482 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.482 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.494 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.494 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.494 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.494 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.496 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.496 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.496 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.496 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.498 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.499 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.499 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.499 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.503 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.503 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.503 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.503 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.654 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.655 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.655 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.655 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.712 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.712 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.712 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.713 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.724 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.725 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.725 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.725 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.727 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.727 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.727 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.727 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.729 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.729 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.729 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.729 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.733 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.733 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.734 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.734 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.888 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.889 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.889 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.889 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.946 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.947 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.947 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.947 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.958 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.958 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.959 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.959 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.960 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.961 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.961 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.961 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.963 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.963 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.963 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.963 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.967 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.967 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.967 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:33.967 WARNING:scrape:Saw end of BT but not start 2024-03-29T12:08:42.497 INFO:scrape:Found 17 distinct failure reasons 2024-03-29T12:08:42.498 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0e392d9aee635f2b65768c460c7a151ead7a5e2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-29T12:08:42.498 INFO:scrape:1 jobs: ['7628228'] 2024-03-29T12:08:42.498 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-03-29T12:08:42.498 INFO:scrape: 2024-03-29T12:08:42.499 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-29T12:08:42.499 INFO:scrape:5 jobs: ['7628209', '7628157', '7628132', '7628061', '7628112'] 2024-03-29T12:08:42.499 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-29T12:08:42.499 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-29T12:08:42.499 INFO:scrape: 2024-03-29T12:08:42.499 INFO:scrape:Dead: None 2024-03-29T12:08:42.499 INFO:scrape:6 jobs: ['7628248', '7628247', '7628184', '7628229', '7628249', '7628096'] 2024-03-29T12:08:42.500 INFO:scrape: 2024-03-29T12:08:42.500 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2024-03-29T12:08:42.500 INFO:scrape:1 jobs: ['7628160'] 2024-03-29T12:08:42.500 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2024-03-29T12:08:42.500 INFO:scrape: 2024-03-29T12:08:42.500 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-29T12:08:42.500 INFO:scrape:3 jobs: ['7628239', '7628213', '7628087'] 2024-03-29T12:08:42.500 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-03-29T12:08:42.501 INFO:scrape:suites union: ['1-ceph', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/fuse', '2-logrotate', '3-inline/no', '3-inline/yes', '3-modules}', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/3', 'roles', 'session_timeout}', 'syntax}', 'tasks/dbench', 'tasks/{0-from/quincy', 'tasks/{0-from/v16.2.4', 'validater/valgrind}'] 2024-03-29T12:08:42.501 INFO:scrape: 2024-03-29T12:08:42.501 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-03-29T12:08:42.501 INFO:scrape:1 jobs: ['7628142'] 2024-03-29T12:08:42.501 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2024-03-29T12:08:42.501 INFO:scrape: 2024-03-29T12:08:42.501 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi171.front.sepia.ceph.com: ['type=AVC msg=audit(1711701836.320:210): avc: denied { checkpoint_restore } for pid=1636 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-29T12:08:42.501 INFO:scrape:7 jobs: ['7628161', '7628091', '7628152', '7628176', '7628143', '7628179', '7628193'] 2024-03-29T12:08:42.502 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}'] 2024-03-29T12:08:42.502 INFO:scrape:suites union: ['1-ceph', '1-upgrade}}', '2-logrotate', '2-workunit/suites/iozone}}', '3-modules}', 'bluestore-bitmap', 'centos_8', 'clusters/1-mds-1-client', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/32bits/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/upgrade/nofs/{bluestore-bitmap', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no-mds-cluster', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/fscrypt-dbench}', 'tasks/mgr-osd-full}', 'tasks/workunit/snaps}', 'tasks/{0-pacific', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'traceless/50pc}'] 2024-03-29T12:08:42.502 INFO:scrape: 2024-03-29T12:08:42.502 INFO:scrape:Failure: 2024-03-29T12:08:42.502 INFO:scrape:1 jobs: ['7628123'] 2024-03-29T12:08:42.502 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-03-29T12:08:42.503 INFO:scrape: 2024-03-29T12:08:42.503 INFO:scrape:Failure: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-29T12:08:42.503 INFO:scrape:1 jobs: ['7628195'] 2024-03-29T12:08:42.503 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/snap-schedule}'] 2024-03-29T12:08:42.503 INFO:scrape: 2024-03-29T12:08:42.503 INFO:scrape:Failure: "2024-03-29T09:00:04.730740+0000 mon.a (mon.0) 2150 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2024-03-29T12:08:42.504 INFO:scrape:1 jobs: ['7628145'] 2024-03-29T12:08:42.504 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror/{begin/{0-install', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/{ignorelist_health}', 'supported-random-distros$/{centos_latest}', 'tasks/mirror}'] 2024-03-29T12:08:42.504 INFO:scrape: 2024-03-29T12:08:42.504 INFO:scrape:Failure: Command failed on smithi123 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid babc8cd0-eda7-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json' 2024-03-29T12:08:42.504 INFO:scrape:1 jobs: ['7628137'] 2024-03-29T12:08:42.504 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2024-03-29T12:08:42.504 INFO:scrape: 2024-03-29T12:08:42.505 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-29T12:08:42.505 INFO:scrape:1 jobs: ['7628169'] 2024-03-29T12:08:42.505 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/pool-perm}'] 2024-03-29T12:08:42.505 INFO:scrape: 2024-03-29T12:08:42.505 INFO:scrape:Failure: Command failed on smithi066 with status 5: 'sudo systemctl stop ceph-6c429176-edac-11ee-b647-cb9ed24678a4@mon.b' 2024-03-29T12:08:42.505 INFO:scrape:1 jobs: ['7628168'] 2024-03-29T12:08:42.505 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/yes', '5-workunit/suites/fsync-tester}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}'] 2024-03-29T12:08:42.505 INFO:scrape: 2024-03-29T12:08:42.506 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) 2024-03-29T12:08:42.506 INFO:scrape:1 jobs: ['7628154'] 2024-03-29T12:08:42.506 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-03-29T12:08:42.506 INFO:scrape: 2024-03-29T12:08:42.506 INFO:scrape:Failure: "2024-03-29T08:25:09.366379+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log 2024-03-29T12:08:42.507 INFO:scrape:1 jobs: ['7628113'] 2024-03-29T12:08:42.507 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/forward-scrub}'] 2024-03-29T12:08:42.507 INFO:scrape: 2024-03-29T12:08:42.507 INFO:scrape:Failure: Command failed on smithi187 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e392d9aee635f2b65768c460c7a151ead7a5e2 shell --fsid 26776ca8-edb0-11ee-b647-cb9ed24678a4 -- ceph daemon mds.f perf dump' 2024-03-29T12:08:42.507 INFO:scrape:1 jobs: ['7628205'] 2024-03-29T12:08:42.507 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-03-29T12:08:42.507 INFO:scrape: 2024-03-29T12:08:42.507 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_clone_with_no_wait_enabled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2024-03-29T12:08:42.507 INFO:scrape:1 jobs: ['7628180'] 2024-03-29T12:08:42.508 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}'] 2024-03-29T12:08:42.508 INFO:scrape: 2024-03-29T12:08:42.520 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/ failed: 28 dead: 6 running: 0 waiting: 0 queued: 0 passed: 181 Fail ================================================================= [7628061] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:43:06 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628061/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628061/ error during scrub thrashing: rank damage found: {'backtrace'} [7628087] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:16:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628087/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628087/ reached maximum tries (51) after waiting for 300 seconds [7628091] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:15:07 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628091/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628091/ SELinux denials found on ubuntu@smithi084.front.sepia.ceph.com: ['type=AVC msg=audit(1711698254.303:210): avc: denied { checkpoint_restore } for pid=1228 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7628112] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:36:26 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628112/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628112/ error during scrub thrashing: rank damage found: {'backtrace'} [7628113] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:24:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628113/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628113/ "2024-03-29T08:25:09.366379+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log [7628123] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:26:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628123/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628123/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3df227e6705d442bb3314cdc9cb2489f [7628132] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:02:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628132/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628132/ error during scrub thrashing: rank damage found: {'backtrace'} [7628137] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:06 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628137/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628137/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46549f06ef8743e7b7fb7b8e12fcefcd Command failed on smithi123 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid babc8cd0-eda7-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json' [7628142] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 01:38:18 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628142/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628142/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28336713f2c04bc49e230d662e4b8459 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7628143] fs/full/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:21:47 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628143/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628143/ SELinux denials found on ubuntu@smithi184.front.sepia.ceph.com: ['type=AVC msg=audit(1711700734.127:210): avc: denied { checkpoint_restore } for pid=1199 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7628145] fs/mirror/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{centos_latest} tasks/mirror} ----------------------------------------------------------------- time: 02:16:54 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628145/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628145/ "2024-03-29T09:00:04.730740+0000 mon.a (mon.0) 2150 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7628152] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:53:25 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628152/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628152/ SELinux denials found on ubuntu@smithi174.front.sepia.ceph.com: ['type=AVC msg=audit(1711701341.797:208): avc: denied { checkpoint_restore } for pid=1204 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7628154] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:35:31 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628154/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628154/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f5d781507024b36b348d426a3865881 Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) [7628157] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:33:31 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628157/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628157/ error during scrub thrashing: rank damage found: {'backtrace'} [7628160] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:29:03 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628160/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628160/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7546ef1b62184e38a653ef2a0c3ea0b9 saw valgrind issues [7628161] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:25:26 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628161/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628161/ SELinux denials found on ubuntu@smithi171.front.sepia.ceph.com: ['type=AVC msg=audit(1711701836.320:210): avc: denied { checkpoint_restore } for pid=1636 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7628168] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:18:24 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628168/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628168/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdaf4adb09554ef98d37e9ed5e2e7def Command failed on smithi066 with status 5: 'sudo systemctl stop ceph-6c429176-edac-11ee-b647-cb9ed24678a4@mon.b' [7628169] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} ----------------------------------------------------------------- time: 00:19:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628169/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ecf4333faa9447eeafbed6cde0cc1976 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7628176] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:18:05 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628176/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628176/ SELinux denials found on ubuntu@smithi143.front.sepia.ceph.com: ['type=AVC msg=audit(1711702418.188:200): avc: denied { checkpoint_restore } for pid=1171 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7628179] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:20:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628179/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628179/ SELinux denials found on ubuntu@smithi113.front.sepia.ceph.com: ['type=AVC msg=audit(1711702536.559:210): avc: denied { checkpoint_restore } for pid=1235 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7628180] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:55:51 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628180/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73354c66a1604a93b42ad4fb26395fc2 Test failure: test_subvolume_snapshot_clone_with_no_wait_enabled (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7628193] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 01:09:43 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628193/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628193/ SELinux denials found on ubuntu@smithi119.front.sepia.ceph.com: ['type=AVC msg=audit(1711703631.810:198): avc: denied { checkpoint_restore } for pid=1150 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7628195] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:10:30 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0b9ea7dcb494d7293b452e8c2de435f Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) [7628205] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:31:01 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628205/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628205/ Command failed on smithi187 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:d0e392d9aee635f2b65768c460c7a151ead7a5e2 shell --fsid 26776ca8-edb0-11ee-b647-cb9ed24678a4 -- ceph daemon mds.f perf dump' [7628209] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:45:55 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628209/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628209/ error during scrub thrashing: rank damage found: {'backtrace'} [7628213] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:43:30 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628213/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628213/ reached maximum tries (51) after waiting for 300 seconds [7628228] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:58:03 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628228/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628228/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9fda8a7b2ad748878082085de60bd7d5 Command failed (workunit test suites/dbench.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0e392d9aee635f2b65768c460c7a151ead7a5e2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7628239] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 01:03:19 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628239/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628239/ reached maximum tries (51) after waiting for 300 seconds Dead ================================================================= [7628096] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628096/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628096/ Error reimaging machines: Failed to power on smithi062 [7628184] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628184/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628184/ Error reimaging machines: Failed to power on smithi018 [7628229] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628229/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628229/ Error reimaging machines: Failed to power on smithi038 [7628247] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628247/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628247/ Error reimaging machines: Failed to power on smithi018 [7628248] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628248/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628248/ Error reimaging machines: Failed to power on smithi184 [7628249] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628249/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628249/ Error reimaging machines: Failed to power on smithi118 Pass ================================================================= [7628059] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:20:46 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628059/ [7628060] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:18:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628060/ [7628062] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:19:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628062/ [7628063] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/auto-repair} time: 00:15:27 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628063/ [7628064] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:18:17 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628064/ [7628065] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:50 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628065/ [7628066] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:34 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628066/ [7628067] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:23:56 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628067/ [7628068] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:06 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628068/ [7628069] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:58:45 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628069/ [7628070] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:34:19 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628070/ [7628071] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/backtrace} time: 00:16:28 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628071/ [7628072] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:38:22 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628072/ [7628073] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/cap-flush} time: 00:19:12 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628073/ [7628074] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:39:21 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628074/ [7628075] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:26 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628075/ [7628076] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:17 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628076/ [7628077] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:40:26 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628077/ [7628078] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:55 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628078/ [7628079] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:34:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628079/ [7628080] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-limits} time: 00:33:25 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628080/ [7628081] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:40:04 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628081/ [7628082] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:39 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628082/ [7628083] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-readahead} time: 00:16:45 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628083/ [7628084] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:20 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628084/ [7628085] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:33:46 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628085/ [7628086] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:24:17 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628086/ [7628088] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-recovery} time: 00:54:11 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628088/ [7628089] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:22:12 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628089/ [7628090] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:34:29 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628090/ [7628092] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:18:59 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628092/ [7628093] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:41 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628093/ [7628094] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:46:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628094/ [7628095] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:53:13 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628095/ [7628097] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:26:39 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628097/ [7628098] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:56 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628098/ [7628099] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:52 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628099/ [7628100] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:30:17 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628100/ [7628101] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:45:55 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628101/ [7628102] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:18:09 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628102/ [7628103] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:19:23 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628103/ [7628104] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:14:31 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628104/ [7628105] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/exports} time: 00:51:34 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628105/ [7628106] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:03 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628106/ [7628107] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:28:57 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628107/ [7628108] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:22:39 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628108/ [7628109] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:28:57 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628109/ [7628110] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:36:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628110/ [7628111] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:57:48 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628111/ [7628114] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:20:45 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628114/ [7628115] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:09 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628115/ [7628116] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:24:35 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628116/ [7628117] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:34 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628117/ [7628118] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:50:56 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628118/ [7628119] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:23:41 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628119/ [7628120] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:23:57 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628120/ [7628121] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:20:07 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628121/ [7628122] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:26:44 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628122/ [7628124] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:24:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628124/ [7628125] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-flush} time: 00:15:19 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628125/ [7628126] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} time: 01:11:27 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628126/ [7628127] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:17:53 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628127/ [7628128] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:30 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628128/ [7628129] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:39:08 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628129/ [7628130] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds-full} time: 00:26:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628130/ [7628131] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:25 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628131/ [7628133] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:13:30 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628133/ [7628134] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:15:12 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628134/ [7628135] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:37:58 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628135/ [7628136] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:36:50 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628136/ [7628138] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:34:04 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628138/ [7628139] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:17 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628139/ [7628140] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:12:18 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628140/ [7628141] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:55:19 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628141/ [7628144] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:14:13 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628144/ [7628146] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} time: 00:31:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628146/ [7628147] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:09 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628147/ [7628148] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:39:19 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628148/ [7628149] fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/nfs} time: 00:33:52 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628149/ [7628150] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:18:39 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628150/ [7628151] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:48:53 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628151/ [7628153] fs/top/{begin/{0-install 1-ceph 2-logrotate 3-modules} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:16:13 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628153/ [7628155] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:09:22 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628155/ [7628156] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:39:20 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628156/ [7628158] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:25:35 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628158/ [7628159] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:22:54 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628159/ [7628162] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:21:19 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628162/ [7628163] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:35:28 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628163/ [7628164] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/openfiletable} time: 00:19:44 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628164/ [7628165] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:50:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628165/ [7628166] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628166/ [7628167] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:34:36 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628167/ [7628170] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:18:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628170/ [7628171] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota} time: 00:19:12 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628171/ [7628172] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:36:23 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628172/ [7628173] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:07:41 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628173/ [7628174] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628174/ [7628175] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:26 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628175/ [7628177] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/recovery-fs} time: 00:12:02 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628177/ [7628178] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:43:25 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628178/ [7628181] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:46:11 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628181/ [7628182] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/scrub} time: 00:36:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628182/ [7628183] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:57 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628183/ [7628185] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:34:52 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628185/ [7628186] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628186/ [7628187] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:42 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628187/ [7628188] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:53:21 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628188/ [7628189] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/sessionmap} time: 00:19:59 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628189/ [7628190] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:48:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628190/ [7628191] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:23:19 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628191/ [7628192] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:32:07 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628192/ [7628194] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:43:33 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628194/ [7628196] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:28:18 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628196/ [7628197] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:23:43 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628197/ [7628198] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:26:39 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628198/ [7628199] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap_schedule_snapdir} time: 00:30:33 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628199/ [7628200] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:00:31 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628200/ [7628201] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:55:21 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628201/ [7628202] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:12 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628202/ [7628203] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:13:29 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628203/ [7628204] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:36:12 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628204/ [7628206] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snapshots} time: 00:28:33 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628206/ [7628207] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:27:56 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628207/ [7628208] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:41 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628208/ [7628210] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays} time: 00:33:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628210/ [7628211] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:34:31 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628211/ [7628212] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:41:04 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628212/ [7628214] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/test_journal_migration} time: 00:23:02 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628214/ [7628215] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 00:59:21 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628215/ [7628216] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:39:32 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628216/ [7628217] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:26:48 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628217/ [7628218] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 02:24:52 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628218/ [7628219] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:57:11 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628219/ [7628220] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:24:15 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628220/ [7628221] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628221/ [7628222] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:42:40 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628222/ [7628223] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/truncate_delay} time: 00:16:29 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628223/ [7628224] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:41:28 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628224/ [7628225] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:22:39 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628225/ [7628226] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:24:38 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628226/ [7628227] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/workunit/dir-max-entries} time: 00:17:36 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628227/ [7628230] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:42 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628230/ [7628231] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:27 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628231/ [7628232] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/workunit/quota} time: 00:16:38 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628232/ [7628233] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:12 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628233/ [7628234] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 01:54:59 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628234/ [7628235] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:27:48 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628235/ [7628236] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:38:17 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628236/ [7628237] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:26:16 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628237/ [7628238] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/acls} time: 00:17:11 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628238/ [7628240] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:23:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628240/ [7628241] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:40:57 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628241/ [7628242] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:36 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628242/ [7628243] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} time: 01:20:33 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628243/ [7628244] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:58:23 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628244/ [7628245] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628245/ [7628246] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:28:32 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628246/ [7628250] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} time: 00:41:22 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628250/ [7628251] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/asok_dump_tree} time: 00:25:55 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628251/ [7628252] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:00 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628252/ [7628253] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:24:17 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628253/ [7628254] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:26:44 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628254/ [7628255] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:57 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628255/ [7628256] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:53:10 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628256/ [7628257] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:07 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628257/ [7628258] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:17:41 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628258/ [7628259] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:50:49 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628259/ [7628260] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:55:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628260/ [7628261] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:37 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628261/ [7628262] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/auto-repair} time: 00:15:22 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628262/ [7628263] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:05 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628263/ [7628264] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:27:54 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628264/ [7628265] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:35:42 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628265/ [7628266] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:27:52 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628266/ [7628267] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:13:23 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628267/ [7628268] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/backtrace} time: 00:18:56 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628268/ [7628269] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:40:21 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628269/ [7628270] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:33:09 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628270/ [7628271] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/cap-flush} time: 00:15:30 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628271/ [7628272] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:47:44 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628272/ [7628273] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:14 info: https://pulpito.ceph.com/vshankar-2024-03-28_11:21:59-fs-wip-vshankar-testing-20240326.105515-reef-testing-default-smithi/7628273/