2024-04-12T18:45:42.762 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-12T18:45:42.763 INFO:teuthology.results:subset: '463/1024' 2024-04-12T18:45:42.763 INFO:teuthology.results:seed: '7105' 2024-04-13T02:45:56.576 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-13T02:45:56.608 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-13T14:46:02.307 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2024-04-13T14:46:23.971 INFO:scrape:Found 205 jobs 2024-04-13T14:46:26.090 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.091 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.091 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.091 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.173 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.174 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.174 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.174 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.260 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.260 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.260 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.260 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.340 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.340 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.340 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.340 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.422 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.422 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.422 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:26.422 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.256 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.257 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.257 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.257 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.777 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.777 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.777 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:28.777 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:29.109 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:29.109 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:29.110 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:29.110 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:31.965 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:31.965 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:31.965 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:31.965 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.055 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.055 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.055 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.055 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.056 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.056 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.056 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.056 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.056 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.058 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.058 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.058 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.058 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.059 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.059 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.059 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.059 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.059 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.059 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.060 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.060 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.060 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.060 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.060 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.061 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.061 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.061 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.061 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.061 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.062 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.062 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.062 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.062 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.063 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.063 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.063 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.063 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.063 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.063 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.063 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.064 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.064 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.064 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.064 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.064 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.065 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.065 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.065 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.065 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.065 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.065 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.066 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.066 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.066 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.066 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.066 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.067 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.074 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.074 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.074 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.074 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.075 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.075 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.075 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.075 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.075 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.076 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.076 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.076 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.076 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.076 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.076 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.077 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.077 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.077 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.077 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.077 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.078 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.078 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.078 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:32.078 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.128 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.128 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.128 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.128 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.135 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.135 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.135 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.135 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.136 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.136 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.136 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.136 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.136 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.136 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.136 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.137 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.137 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.137 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.137 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.137 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.138 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.138 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.138 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.138 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.138 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.139 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.139 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.139 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.139 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.139 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.139 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.139 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.140 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.140 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.140 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.140 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.140 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.140 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.141 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.141 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.141 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.141 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.141 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.141 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.142 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.142 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.142 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.142 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.143 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.143 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.143 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.143 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.143 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.143 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.144 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.144 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.144 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.144 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.144 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.144 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.145 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.145 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.145 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.145 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.145 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.146 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.146 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.146 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.146 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.146 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.146 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.146 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.147 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.147 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.147 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.147 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.147 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.148 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.148 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.148 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.149 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.149 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.149 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.150 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.150 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.150 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.151 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.151 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.151 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.151 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.152 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.152 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.152 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.152 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.152 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.153 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.153 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.153 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.153 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.153 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.154 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.154 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.154 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.154 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.155 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.155 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.155 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.155 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.155 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.155 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.156 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.156 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.156 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.156 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.156 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.156 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.157 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.157 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.157 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.157 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.157 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.157 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.158 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.158 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.158 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.158 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.158 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.159 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.159 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.159 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.159 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.159 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.160 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.160 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.160 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.160 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.160 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.160 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.160 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.161 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.161 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.161 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.161 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.161 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.161 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.162 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.162 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.162 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.162 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.162 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.162 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.163 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.163 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.163 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.163 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.163 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.164 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.164 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.164 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.164 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.164 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.165 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.165 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.165 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.165 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.165 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.165 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.165 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.166 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.166 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.166 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.166 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.167 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.167 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.167 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.167 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.167 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.167 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.167 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.168 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.168 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.168 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.168 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.168 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.169 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.169 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.169 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.169 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.169 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.169 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.169 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.170 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.170 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.170 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.170 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:46:37.170 WARNING:scrape:Saw end of BT but not start 2024-04-13T14:47:40.028 INFO:scrape:Found 25 distinct failure reasons 2024-04-13T14:47:40.029 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi131 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-04-13T14:47:40.029 INFO:scrape:ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fd5b3e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fd5b3ea154c] 3: raise() 4: abort() 5: ceph-mds(+0x14db08) [0x564124103b08] 6: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2460) [0x56412417a9b0] 7: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5641243f8b62] 8: (MDLog::trim()+0xc8a) [0x5641243fc81a] 9: (MDLog::log_trim_upkeep()+0x398) [0x5641243fccc8] 10: /lib64/libstdc++.so.6(+0xdb924) [0x7fd5b42db924] 11: /lib64/libc.so.6(+0x9f802) [0x7fd5b3e9f802] 12: /lib64/libc.so.6(+0x3f450) [0x7fd5b3e3f450] 2024-04-13T14:47:40.029 INFO:scrape:5 jobs: ['7653524', '7653638', '7653662', '7653586', '7653667'] 2024-04-13T14:47:40.029 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg_health}'] 2024-04-13T14:47:40.029 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/backtrace}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/workunit/snaps}'] 2024-04-13T14:47:40.029 INFO:scrape: 2024-04-13T14:47:40.029 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun 2024-04-13T14:47:40.030 INFO:scrape:1 jobs: ['7653547'] 2024-04-13T14:47:40.030 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-13T14:47:40.030 INFO:scrape: 2024-04-13T14:47:40.030 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported'} 2024-04-13T14:47:40.030 INFO:scrape:2 jobs: ['7653503', '7653482'] 2024-04-13T14:47:40.030 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2024-04-13T14:47:40.030 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-04-13T14:47:40.030 INFO:scrape: 2024-04-13T14:47:40.031 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-04-13T14:47:40.031 INFO:scrape:2 jobs: ['7653636', '7653672'] 2024-04-13T14:47:40.031 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'session_timeout}', 'validater/valgrind}'] 2024-04-13T14:47:40.031 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-13T14:47:40.031 INFO:scrape: 2024-04-13T14:47:40.031 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T14:47:40.031 INFO:scrape:6 jobs: ['7653491', '7653686', '7653613', '7653578', '7653544', '7653504'] 2024-04-13T14:47:40.031 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}'] 2024-04-13T14:47:40.031 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}'] 2024-04-13T14:47:40.032 INFO:scrape: 2024-04-13T14:47:40.032 INFO:scrape:Failure: "2024-04-13T02:40:00.000153+0000 mon.a (mon.0) 1017 : cluster [WRN] Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled" in cluster log 2024-04-13T14:47:40.032 INFO:scrape:4 jobs: ['7653659', '7653626', '7653664', '7653650'] 2024-04-13T14:47:40.032 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mgr', 'mon', 'osd}'] 2024-04-13T14:47:40.032 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '2-logrotate', '2-scrub/yes', '3-inline/no', '3-inline/yes', '3-modules}', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-verify}', '5-upgrade-with-workload', '5-workunit/suites/iozone}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'frag', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'ranks/1', 'roles', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{centos_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.1}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-04-13T14:47:40.032 INFO:scrape: 2024-04-13T14:47:40.032 INFO:scrape:Dead: b'2024-04-13T14:46:30.996 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 13 14:46:30 smithi092 ceph-mon[36649]: pgmap v21565: 129 pgs: 129 active+clean; 3.5 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s' 2024-04-13T14:47:40.032 INFO:scrape:1 jobs: ['7653680'] 2024-04-13T14:47:40.033 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-04-13T14:47:40.033 INFO:scrape: 2024-04-13T14:47:40.033 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) 2024-04-13T14:47:40.033 INFO:scrape:1 jobs: ['7653609'] 2024-04-13T14:47:40.033 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/test_journal_migration}'] 2024-04-13T14:47:40.033 INFO:scrape: 2024-04-13T14:47:40.033 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-13T14:47:40.033 INFO:scrape:1 jobs: ['7653508'] 2024-04-13T14:47:40.033 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-04-13T14:47:40.033 INFO:scrape: 2024-04-13T14:47:40.034 INFO:scrape:Dead: b'2024-04-13T07:14:18.286 DEBUG:teuthology.exit:Finished running handlers' 2024-04-13T14:47:40.034 INFO:scrape:7 jobs: ['7653515', '7653602', '7653494', '7653553', '7653601', '7653534', '7653588'] 2024-04-13T14:47:40.034 INFO:scrape:suites intersection: ['2-logrotate', '3-modules}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}'] 2024-04-13T14:47:40.034 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '2-workunit/fs/snaps}}', '3-modules}', '3-snaps/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'export-check', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/none', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'wsync/no}'] 2024-04-13T14:47:40.034 INFO:scrape: 2024-04-13T14:47:40.034 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2024-04-13T14:47:40.034 INFO:scrape:1 jobs: ['7653644'] 2024-04-13T14:47:40.035 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/acls}'] 2024-04-13T14:47:40.035 INFO:scrape: 2024-04-13T14:47:40.035 INFO:scrape:Failure: ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8. 2024-04-13T14:47:40.035 INFO:scrape:1 jobs: ['7653536'] 2024-04-13T14:47:40.035 INFO:scrape:suites: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'fs/upgrade/nofs/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-pacific'] 2024-04-13T14:47:40.035 INFO:scrape: 2024-04-13T14:47:40.035 INFO:scrape:Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) 2024-04-13T14:47:40.035 INFO:scrape:1 jobs: ['7653558'] 2024-04-13T14:47:40.035 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/openfiletable}'] 2024-04-13T14:47:40.035 INFO:scrape: 2024-04-13T14:47:40.035 INFO:scrape:Crash: Command failed on smithi087 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' 2024-04-13T14:47:40.036 INFO:scrape:ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f076fc42520] 2: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x16f1) [0x5583068272b1] 3: (MDLog::try_expire(LogSegment*, int)+0x62) [0x558306a9bec2] 4: (MDLog::trim()+0xc19) [0x558306aa1619] 5: (MDLog::log_trim_upkeep()+0x410) [0x558306aa1b30] 6: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc2b3) [0x7f07700dc2b3] 7: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f076fc94b43] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f076fd26a00] 2024-04-13T14:47:40.036 INFO:scrape:3 jobs: ['7653590', '7653679', '7653506'] 2024-04-13T14:47:40.036 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'conf/{client', 'distro/{ubuntu_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}'] 2024-04-13T14:47:40.036 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/fs/snaps}}', '3-modules}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'no_client_pidfile', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'subvol_versions/create_subvol_version_v2', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/data-scan}', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'traceless/50pc}'] 2024-04-13T14:47:40.036 INFO:scrape: 2024-04-13T14:47:40.036 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-04-13T14:47:40.036 INFO:scrape:1 jobs: ['7653527'] 2024-04-13T14:47:40.036 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-04-13T14:47:40.036 INFO:scrape: 2024-04-13T14:47:40.037 INFO:scrape:Failure: Test failure: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize) 2024-04-13T14:47:40.037 INFO:scrape:1 jobs: ['7653647'] 2024-04-13T14:47:40.037 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-04-13T14:47:40.037 INFO:scrape: 2024-04-13T14:47:40.037 INFO:scrape:Failure: Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2024-04-13T14:47:40.037 INFO:scrape:2 jobs: ['7653631', '7653622'] 2024-04-13T14:47:40.037 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'mount/fuse', 'overrides/{ignorelist_health', 'pg_health}'] 2024-04-13T14:47:40.037 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/{ignorelist_health', 'pg_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}'] 2024-04-13T14:47:40.037 INFO:scrape: 2024-04-13T14:47:40.037 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-04-13T14:47:40.037 INFO:scrape:1 jobs: ['7653635'] 2024-04-13T14:47:40.038 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-04-13T14:47:40.038 INFO:scrape: 2024-04-13T14:47:40.038 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi100.front.sepia.ceph.com: ['type=AVC msg=audit(1712952410.210:210): avc: denied { checkpoint_restore } for pid=1643 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-04-13T14:47:40.038 INFO:scrape:1 jobs: ['7653559'] 2024-04-13T14:47:40.038 INFO:scrape:suites: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', '3-workload/stress_tests/fsstress}}', 'branch/nautilus', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-install'] 2024-04-13T14:47:40.038 INFO:scrape: 2024-04-13T14:47:40.038 INFO:scrape:Failure: Test failure: test_quiesce_authpin_wait (tasks.cephfs.test_quiesce.TestQuiesceMultiRank) 2024-04-13T14:47:40.038 INFO:scrape:1 jobs: ['7653566'] 2024-04-13T14:47:40.038 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/quiesce}'] 2024-04-13T14:47:40.038 INFO:scrape: 2024-04-13T14:47:40.039 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi100 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-04-13T14:47:40.039 INFO:scrape:1 jobs: ['7653594'] 2024-04-13T14:47:40.039 INFO:scrape:suites: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_snaptests}}', '3-modules}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2024-04-13T14:47:40.039 INFO:scrape: 2024-04-13T14:47:40.039 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) 2024-04-13T14:47:40.039 INFO:scrape:1 jobs: ['7653533'] 2024-04-13T14:47:40.039 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/mds-flush}'] 2024-04-13T14:47:40.039 INFO:scrape: 2024-04-13T14:47:40.039 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi038 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-04-13T14:47:40.039 INFO:scrape:1 jobs: ['7653500'] 2024-04-13T14:47:40.040 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-04-13T14:47:40.040 INFO:scrape: 2024-04-13T14:47:40.040 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-13T14:47:40.040 INFO:scrape:1 jobs: ['7653523'] 2024-04-13T14:47:40.040 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs-shell}'] 2024-04-13T14:47:40.040 INFO:scrape: 2024-04-13T14:47:40.040 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-13T14:47:40.040 INFO:scrape:1 jobs: ['7653607'] 2024-04-13T14:47:40.040 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-04-13T14:47:40.040 INFO:scrape: 2024-04-13T14:47:40.054 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/ failed: 40 dead: 7 running: 1 waiting: 0 queued: 0 passed: 157 Fail ================================================================= [7653482] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:42 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653482/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653482/ The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} [7653491] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:48:02 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653491/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653491/ reached maximum tries (51) after waiting for 300 seconds [7653500] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:36:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653500/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1bd7bb53e04b45abac77d7e7028e111e Command failed (workunit test kernel_untar_build.sh) on smithi038 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7653503] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:58:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653503/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653503/ The following counters failed to be set on mds daemons: {'mds.exported'} [7653504] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653504/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653504/ reached maximum tries (51) after waiting for 300 seconds [7653506] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/data-scan} ----------------------------------------------------------------- time: 00:14:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653506/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a412503ef6364d6f83208521e081c60b Test failure: test_extra_data_pool_rebuild_few_files (tasks.cephfs.test_data_scan.TestDataScan) [7653508] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 01:03:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653508/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653508/ error during scrub thrashing: rank damage found: {'backtrace'} [7653523] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:55 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653523/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653523/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d84863db5014b03a8a053c7d1a0c9c9 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7653524] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:43:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653524/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653524/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f8068191fff4719bfadc19ce4d58bd0 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi131 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7653527] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:07:38 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653527/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b973e0fdd09e496fb5620656fea8e8c9 Command failed (workunit test suites/ffsb.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7653533] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds-flush} ----------------------------------------------------------------- time: 00:12:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653533/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653533/ sentry: https://sentry.ceph.com/organizations/ceph/?query=755c47263f0843efbcd7c4e919486483 Test failure: test_flush (tasks.cephfs.test_flush.TestFlush) [7653536] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:11:30 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653536/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653536/ sentry: https://sentry.ceph.com/organizations/ceph/?query=329db6e3bcc243bfa5f0ef7b1db82dfc ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8. [7653544] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:15 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653544/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653544/ reached maximum tries (51) after waiting for 300 seconds [7653547] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:51:31 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653547/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653547/ valgrind error: Leak_StillReachable operator new[](unsigned long) rocksdb::ObjectLibrary::AddFactoryEntry(char const*, std::unique_ptr >&&) UnknownInlinedFun [7653558] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable} ----------------------------------------------------------------- time: 00:10:48 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653558/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653558/ sentry: https://sentry.ceph.com/organizations/ceph/?query=72799149ed0e458f99022fb0facee999 Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) [7653559] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} ----------------------------------------------------------------- time: 00:20:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653559/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653559/ SELinux denials found on ubuntu@smithi100.front.sepia.ceph.com: ['type=AVC msg=audit(1712952410.210:210): avc: denied { checkpoint_restore } for pid=1643 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7653566] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce} ----------------------------------------------------------------- time: 00:37:06 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653566/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f803945722946ff8e383e96fd67c9f7 Test failure: test_quiesce_authpin_wait (tasks.cephfs.test_quiesce.TestQuiesceMultiRank) [7653578] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:31:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653578/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653578/ reached maximum tries (51) after waiting for 300 seconds [7653586] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:26:13 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653586/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653586/ sentry: https://sentry.ceph.com/organizations/ceph/?query=677b51c6f8e5421da9d2db4c72722e29 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi123 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7653590] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:24:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653590/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653590/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9bb0b5dc4444d239b2444f26f155570 Command failed on smithi087 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [7653594] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:26:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653594/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653594/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a284b63a7e0a42d2ab30c6366c975e76 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi100 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7653607] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:16:49 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653607/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653607/ sentry: https://sentry.ceph.com/organizations/ceph/?query=288174e2beab4e8baa695f443d3d5b65 Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7653609] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:21:20 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653609/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653609/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8763d0a599714676812da38c8a2a74bb Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) [7653613] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:30:49 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653613/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653613/ reached maximum tries (51) after waiting for 300 seconds [7653622] fs/mirror/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} ----------------------------------------------------------------- time: 00:21:45 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653622/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c1bbd5732e84c2181f28b974e1f2631 Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7653626] fs/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:45:02 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653626/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653626/ "2024-04-12T21:37:43.542533+0000 mon.a (mon.0) 320 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log [7653631] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/{ignorelist_health pg_health} tasks/mirror}} ----------------------------------------------------------------- time: 00:29:22 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653631/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653631/ sentry: https://sentry.ceph.com/organizations/ceph/?query=982f155971b44eaa84cf0c78a936689b Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7653635] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:15:41 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653635/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653635/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07f5dc88efed42e294fb383f0f024d72 Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7653636] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:51:45 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653636/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653636/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7653638] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:37:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653638/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653638/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba450fbe567140fbbf9601b850f746e7 Command failed (workunit test suites/dbench.sh) on smithi187 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7653644] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/acls} ----------------------------------------------------------------- time: 00:12:54 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653644/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653644/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7050cf38123c45b49d55f0cfc288045c Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [7653647] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:35:39 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653647/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653647/ sentry: https://sentry.ceph.com/organizations/ceph/?query=076b6bc51fe241b896f64ca3bde803e2 Test failure: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize) [7653650] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:43:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653650/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653650/ "2024-04-13T02:50:00.000400+0000 mon.smithi072 (mon.0) 472 : cluster [WRN] pg 1.0 is active+undersized+degraded, acting [0,1]" in cluster log [7653659] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:32:26 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653659/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653659/ "2024-04-13T02:40:00.000153+0000 mon.a (mon.0) 1017 : cluster [WRN] Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled" in cluster log [7653662] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:26:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653662/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653662/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8a704f667f0d4994a0f43d93196b98e7 SSH connection to smithi031 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [7653664] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:41:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653664/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653664/ "2024-04-13T03:10:00.000131+0000 mon.smithi176 (mon.0) 685 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem is degraded" in cluster log [7653667] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/backtrace} ----------------------------------------------------------------- time: 00:09:23 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653667/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653667/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9282ce33cfe4d2abe15fed063f5aef7 Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) [7653672] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:32:41 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653672/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653672/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7653679] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:32:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653679/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653679/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e163e9f556f446afa42e99692ac0e1e7 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7653686] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653686/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653686/ reached maximum tries (51) after waiting for 300 seconds Dead ================================================================= [7653494] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653494/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653494/ hit max job timeout [7653515] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653515/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653515/ hit max job timeout [7653534] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653534/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653534/ hit max job timeout [7653553] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653553/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653553/ hit max job timeout [7653588] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653588/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653588/ hit max job timeout [7653601] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653601/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653601/ hit max job timeout [7653602] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602/ hit max job timeout Running ================================================================= [7653680] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653680/ Pass ================================================================= [7653483] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/cap-flush} time: 00:09:55 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653483/ [7653484] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} time: 00:16:05 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653484/ [7653485] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:16:07 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653485/ [7653486] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:33:49 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653486/ [7653487] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:14:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653487/ [7653488] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}} time: 01:43:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653488/ [7653489] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:32:54 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653489/ [7653490] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/client-limits} time: 00:22:28 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653490/ [7653492] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-readahead} time: 00:12:27 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653492/ [7653493] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:42:19 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653493/ [7653495] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653495/ [7653496] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653496/ [7653497] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:39 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653497/ [7653498] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/client-recovery} time: 00:46:56 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653498/ [7653499] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:14:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653499/ [7653501] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:25:19 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653501/ [7653502] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:12:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653502/ [7653505] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:18:53 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653505/ [7653507] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:41 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653507/ [7653509] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/exports} time: 00:45:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653509/ [7653510] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:27:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653510/ [7653511] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 01:04:54 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653511/ [7653512] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:24 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653512/ [7653513] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:37:28 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653513/ [7653514] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:26:20 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653514/ [7653516] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:22:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653516/ [7653517] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653517/ [7653518] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/{ignorelist_health pg_health}} time: 00:24:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653518/ [7653519] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs/{frag test}} time: 00:23:56 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653519/ [7653520] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 01:08:07 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653520/ [7653521] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:40:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653521/ [7653522] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:15 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653522/ [7653525] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:22:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653525/ [7653526] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:12:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653526/ [7653528] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:42:15 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653528/ [7653529] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:17:27 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653529/ [7653530] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:21:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653530/ [7653531] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:30:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653531/ [7653532] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:54 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653532/ [7653535] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:33:07 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653535/ [7653537] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/mds-full} time: 00:25:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653537/ [7653538] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:13:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653538/ [7653539] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653539/ [7653540] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:48:42 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653540/ [7653541] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:11:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653541/ [7653542] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:18:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653542/ [7653543] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:18:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653543/ [7653545] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:32:37 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653545/ [7653546] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:11:29 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653546/ [7653548] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:36:28 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653548/ [7653549] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:20:20 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653549/ [7653550] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/dbench}} time: 00:51:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653550/ [7653551] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:14:35 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653551/ [7653552] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:39:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653552/ [7653554] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:21:51 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653554/ [7653555] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:14:27 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653555/ [7653556] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:08:25 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653556/ [7653557] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:31:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653557/ [7653560] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653560/ [7653561] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:35:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653561/ [7653562] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} time: 00:14:51 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653562/ [7653563] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:40 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653563/ [7653564] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:17:35 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653564/ [7653565] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:34:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653565/ [7653567] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} time: 00:17:25 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653567/ [7653568] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653568/ [7653569] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 01:10:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653569/ [7653570] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:32:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653570/ [7653571] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:19 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653571/ [7653572] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs_python} time: 00:14:55 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653572/ [7653573] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653573/ [7653574] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota} time: 00:12:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653574/ [7653575] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} time: 00:14:12 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653575/ [7653576] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:19:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653576/ [7653577] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:29:12 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653577/ [7653579] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/scrub} time: 00:35:20 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653579/ [7653580] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:13:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653580/ [7653581] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:15:40 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653581/ [7653582] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/sessionmap} time: 00:13:11 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653582/ [7653583] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:19:13 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653583/ [7653584] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} time: 00:29:28 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653584/ [7653585] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:28:05 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653585/ [7653587] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:14:41 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653587/ [7653589] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653589/ [7653591] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} time: 02:11:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653591/ [7653592] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653592/ [7653593] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-pjd}} time: 00:19:30 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653593/ [7653595] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/snapshot}} time: 00:36:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653595/ [7653596] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:42:05 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653596/ [7653597] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap_schedule_snapdir} time: 00:14:26 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653597/ [7653598] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:20:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653598/ [7653599] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:17:38 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653599/ [7653600] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/new_ops/{newops}}} time: 00:12:15 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653600/ [7653603] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snapshots} time: 00:28:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653603/ [7653604] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:21 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653604/ [7653605] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/strays} time: 00:33:11 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653605/ [7653606] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653606/ [7653608] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:24:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653608/ [7653610] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:49:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653610/ [7653611] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 01:38:11 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653611/ [7653612] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/truncate_delay} time: 00:11:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653612/ [7653614] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:25 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653614/ [7653615] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:33:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653615/ [7653616] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/dir-max-entries} time: 00:15:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653616/ [7653617] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:19 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653617/ [7653618] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest clusters/small-cluster conf/{client mds mgr mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/trim-i24137} time: 00:11:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653618/ [7653619] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host overrides/{ignorelist_health pg_health}} time: 00:50:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653619/ [7653620] fs/full/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-node-1-mds-1-osd conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/mgr-osd-full} time: 00:20:41 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653620/ [7653621] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client} time: 00:12:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653621/ [7653623] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:24:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653623/ [7653624] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 01:07:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653624/ [7653625] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:42:37 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653625/ [7653627] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:17:39 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653627/ [7653628] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} time: 00:47:56 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653628/ [7653629] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 01:42:02 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653629/ [7653630] fs/top/{begin/{0-install 1-ceph 2-logrotate 3-modules} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:15:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653630/ [7653632] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}} time: 01:58:44 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653632/ [7653633] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:15:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653633/ [7653634] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:32:02 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653634/ [7653637] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:37:44 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653637/ [7653639] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:47:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653639/ [7653640] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:12:21 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653640/ [7653641] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:32:38 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653641/ [7653642] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/quota} time: 00:11:27 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653642/ [7653643] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} time: 00:21:21 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653643/ [7653645] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:13:25 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653645/ [7653646] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:33:13 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653646/ [7653648] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:24:38 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653648/ [7653649] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:33:38 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653649/ [7653651] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:10:07 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653651/ [7653652] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:30 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653652/ [7653653] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653653/ [7653654] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653654/ [7653655] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 00:58:11 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653655/ [7653656] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:48:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653656/ [7653657] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/asok_dump_tree} time: 00:18:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653657/ [7653658] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:19:13 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653658/ [7653660] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:14:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653660/ [7653661] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:07:46 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653661/ [7653663] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair} time: 00:15:35 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653663/ [7653665] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:12:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653665/ [7653666] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:33:41 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653666/ [7653668] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/cap-flush} time: 00:13:40 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653668/ [7653669] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653669/ [7653670] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:13:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653670/ [7653671] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:13:32 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653671/ [7653673] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:23:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653673/ [7653674] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:35 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653674/ [7653675] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-limits} time: 00:29:19 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653675/ [7653676] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:31 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653676/ [7653677] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/ino_release_cb} time: 00:13:49 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653677/ [7653678] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:18:42 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653678/ [7653681] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-ffsb}} time: 02:04:32 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653681/ [7653682] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:13:24 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653682/ [7653683] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/client-readahead} time: 00:09:24 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653683/ [7653684] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:09:54 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653684/ [7653685] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 00:45:49 info: https://pulpito.ceph.com/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653685/