2024-02-07T16:42:34.328 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-07T16:42:34.328 INFO:teuthology.results:subset: '111/120000' 2024-02-07T16:42:34.328 INFO:teuthology.results:seed: '6149' 2024-02-08T07:43:15.257 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-08T07:43:15.288 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-08T19:36:25.275 INFO:teuthology.results:Tests finished! gathering results... 2024-02-08T19:36:50.409 INFO:scrape:Found 325 jobs 2024-02-08T19:36:58.306 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550228/teuthology.log 2024-02-08T19:37:04.780 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.780 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.780 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.780 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.781 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.782 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.782 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.782 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.783 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.783 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.784 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.784 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.785 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.785 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.785 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.786 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.786 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.786 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.786 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:04.786 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.087 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.088 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.088 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.088 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.089 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.089 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.090 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.090 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.091 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.091 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.091 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.091 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.092 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.092 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.092 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.093 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.094 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.094 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.094 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.094 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.283 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.283 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.283 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.284 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.284 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.284 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.284 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.285 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.285 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.285 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.286 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.287 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.287 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.287 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.287 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.480 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.480 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.480 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.481 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.482 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.482 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.482 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.482 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.483 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.484 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.484 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.484 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.485 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.485 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.485 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.486 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.486 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.486 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.486 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.487 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.695 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.696 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.696 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.696 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.698 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.699 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.699 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.699 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.702 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.702 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.703 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.703 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.704 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.705 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.705 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.707 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.707 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.708 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.708 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.708 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.923 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.923 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.923 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.923 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.925 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.925 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.925 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.925 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.926 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.926 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.927 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.927 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.927 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.928 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.928 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:05.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.132 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.132 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.133 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.133 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.134 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.134 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.134 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.134 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.136 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.136 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.136 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.136 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.137 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.137 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.137 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.138 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.138 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.138 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.139 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.139 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.329 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.329 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.329 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.329 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.330 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.331 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.331 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.331 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.332 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.332 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.333 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.333 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.333 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.334 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.334 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.335 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.335 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.335 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.335 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.335 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.530 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.530 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.531 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.531 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.532 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.532 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.532 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.532 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.534 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.534 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.534 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.534 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.535 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.535 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.535 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.536 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.537 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.537 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.537 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.537 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.732 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.733 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.733 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.733 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.734 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.734 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.735 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.735 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.736 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.736 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.736 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.737 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.737 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.738 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.738 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.739 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.739 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.739 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.739 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.739 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.925 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.926 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.926 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.926 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.927 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.927 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.927 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.927 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.929 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.930 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.930 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.930 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.931 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.931 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.931 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.932 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:06.932 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.116 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.117 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.117 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.117 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.118 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.118 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.118 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.119 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.120 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.120 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.120 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.120 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.121 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.121 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.121 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.122 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.122 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.123 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.123 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:07.123 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.277 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.278 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.278 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.279 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.279 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.279 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.280 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.281 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.282 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.283 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.283 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.367 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.367 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.368 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.369 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.369 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.369 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.369 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.370 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.370 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.370 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.370 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.371 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.371 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.371 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.371 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.371 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.371 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.372 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.372 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.372 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.372 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.372 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.373 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.373 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.458 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.458 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.459 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.460 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.460 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.460 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.460 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.461 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.461 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.462 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.462 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.462 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.462 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.462 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.462 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.462 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.463 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.463 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.463 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.463 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.463 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.463 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.464 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.464 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.549 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.549 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.549 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.551 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.551 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.551 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.551 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.552 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.552 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.552 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.552 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.552 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.553 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.553 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.553 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.553 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.553 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.553 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.553 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.554 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.554 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.554 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.554 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.554 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.636 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.637 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.637 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.638 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.638 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.639 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.639 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.640 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.640 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.640 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.640 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.640 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.640 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.640 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.641 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.642 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.642 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.723 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.724 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.724 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.725 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.725 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.726 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.726 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.727 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.727 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.727 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.727 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.727 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.727 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.727 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.728 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.728 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.728 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.728 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.728 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.728 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.729 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.729 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.729 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.729 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.810 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.811 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.811 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.812 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.812 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.812 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.813 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.813 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.814 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.814 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.814 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.814 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.814 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.814 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.815 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.816 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.816 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.896 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.896 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.896 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.897 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.897 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.898 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.898 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.899 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.899 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.899 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.899 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.899 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.899 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.899 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.900 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.901 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.901 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:09.986 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.987 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.987 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.988 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.988 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.988 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.988 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.989 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.990 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.990 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.990 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.990 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.990 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.990 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.990 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.991 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.991 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.991 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.991 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.991 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.991 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.991 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:09.992 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:09.992 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:10.072 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.072 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.072 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.074 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.074 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.074 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.074 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.075 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.075 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.075 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.075 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.075 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.075 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.076 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.076 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.076 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.076 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.076 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.076 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.076 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.077 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.077 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.077 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:10.077 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:10.159 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.160 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.160 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.161 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.162 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.162 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.162 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.163 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.163 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.163 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.163 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.163 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.163 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.164 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.164 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.164 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.164 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.164 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.165 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.165 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.165 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.165 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.165 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:10.165 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:10.248 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.249 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.249 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.250 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.250 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.250 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.250 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.251 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.252 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.252 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.252 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.252 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.252 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.252 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.252 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.253 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.253 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.253 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.253 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.253 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.253 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.253 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.254 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:10.254 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:10.346 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.347 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.347 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.348 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.348 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.348 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.349 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.349 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.350 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.350 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.350 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.350 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.350 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.350 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.351 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.352 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:10.352 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:10.433 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.434 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.434 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.435 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.435 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.435 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.436 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.437 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.437 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.437 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.437 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.437 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.437 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.437 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.438 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.439 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:10.439 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:10.518 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.518 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.518 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.519 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.520 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.520 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.520 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.521 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.521 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.521 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.521 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.521 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.521 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.521 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.522 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.522 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.522 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.522 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.522 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.522 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.522 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.523 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.523 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:10.523 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:10.612 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.613 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.613 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.614 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.614 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.614 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.614 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.615 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.615 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.616 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.616 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.616 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.616 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.616 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.616 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.616 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.617 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.617 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.617 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.617 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.617 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.617 WARNING:scrape:Saw end of BT but not start 2024-02-08T19:37:10.618 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.2.1-504-g308f19ea (308f19ea703b2ecb7fc19d5af4edca54d277028f) reef (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f7d26161763] , 2: ceph_test_rados(+0x2a011) [0x55ba0f899011] 2024-02-08T19:37:10.618 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-02-08T19:37:14.430 INFO:scrape:Found 20 distinct failure reasons 2024-02-08T19:37:14.430 INFO:scrape:Dead: b'2024-02-08T05:14:08.836 DEBUG:teuthology.exit:Finished running handlers' 2024-02-08T19:37:14.430 INFO:scrape:41 jobs 2024-02-08T19:37:14.431 INFO:scrape:suites intersection: ['rados'] 2024-02-08T19:37:14.431 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/{fixed-2', 'clusters/{fixed-4', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/on', 'd-balancer/read', 'd-balancer/upmap-read', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/write_fadvise_dontneed}'] 2024-02-08T19:37:14.431 INFO:scrape: 2024-02-08T19:37:14.431 INFO:scrape:Failure: "2024-02-07T21:07:24.780279+0000 mon.smithi094 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log 2024-02-08T19:37:14.431 INFO:scrape:36 jobs 2024-02-08T19:37:14.432 INFO:scrape:suites intersection: [] 2024-02-08T19:37:14.432 INFO:scrape:suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '1-install/nautilus', '1-install/nautilus-v1only', '1-install/nautilus-v2only', '1-install/octopus', '1-install/pacific', '1-install/quincy', '1-start', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', 'agent/off', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/cephadm/osds/{0-distro/centos_9.stream', 'rados/cephadm/osds/{0-distro/centos_9.stream_runc', 'rados/cephadm/osds/{0-distro/ubuntu_22.04', 'rados/cephadm/smoke/{0-distro/centos_9.stream', 'rados/cephadm/smoke/{0-distro/centos_9.stream_runc', 'rados/cephadm/smoke/{0-distro/ubuntu_22.04', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'start}', 'task/test_host_drain}', 'task/test_monitoring_stack_basic}', 'task/test_orch_cli_mon}', 'task/test_rgw_multisite}', 'task/test_set_mon_crush_locations}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-snaps}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2024-02-08T19:37:14.432 INFO:scrape: 2024-02-08T19:37:14.432 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-02-08T19:37:14.432 INFO:scrape:1 jobs: ['7550205'] 2024-02-08T19:37:14.432 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/scrub}'] 2024-02-08T19:37:14.433 INFO:scrape: 2024-02-08T19:37:14.433 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2024-02-08T19:37:14.433 INFO:scrape:2 jobs: ['7550207', '7550538'] 2024-02-08T19:37:14.433 INFO:scrape:suites intersection: ['rados'] 2024-02-08T19:37:14.433 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton/{all/backfill-toofull', 'rados/singleton/{all/ec-lost-unfound', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-02-08T19:37:14.433 INFO:scrape: 2024-02-08T19:37:14.433 INFO:scrape:Crash: Command failed on smithi027 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats' 2024-02-08T19:37:14.433 INFO:scrape:ceph version 19.0.0-1312-gf4e4dcde (f4e4dcdef27eab6190bcda03dba829ee096072fb) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0cab8a4520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x565010bd76d3] 6: (coll_t::calc_str()+0x5f) [0x565010f5e8cf] 7: ceph-osd(+0x11e048b) [0x5650119c148b] 8: ceph-osd(+0x507da0) [0x565010ce8da0] 9: (ECBackend::RecoveryBackend::handle_recovery_push(PushOp const&, RecoveryMessages*, bool)+0x68d) [0x5650110b0b5d] 10: (ECBackend::handle_recovery_push(PushOp const&, RecoveryMessages*, bool)+0x11a) [0x5650110b147a] 11: (ECBackend::_handle_message(boost::intrusive_ptr)+0x1d5) [0x5650110b7015] 12: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x565010eb3426] 13: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x565010e0b43d] 14: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x565010d5f807] 15: (ceph::osd::scheduler::PGRecoveryMsg::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x198) [0x565010fa7ca8] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x565010d6a483] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x565011262e93] 18: ceph-osd(+0xa823f4) [0x5650112633f4] 19: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f0cab8f6b43] 20: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f0cab988a00] 2024-02-08T19:37:14.433 INFO:scrape:4 jobs: ['7550196', '7550485', '7550348', '7550312'] 2024-02-08T19:37:14.434 INFO:scrape:suites intersection: ['rados'] 2024-02-08T19:37:14.434 INFO:scrape:suites union: ['clusters/{fixed-2', 'clusters/{fixed-4', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/singleton/{all/ec-inconsistent-hinfo', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{default}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/repair_test}', 'thrashers/careful', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] 2024-02-08T19:37:14.434 INFO:scrape: 2024-02-08T19:37:14.434 INFO:scrape:Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-02-08T19:37:14.434 INFO:scrape:5 jobs: ['7550345', '7550576', '7550426', '7550275', '7550346'] 2024-02-08T19:37:14.434 INFO:scrape:suites intersection: [] 2024-02-08T19:37:14.434 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-02-08T19:37:14.434 INFO:scrape: 2024-02-08T19:37:14.434 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2024-02-08T19:37:14.435 INFO:scrape:1 jobs: ['7550380'] 2024-02-08T19:37:14.435 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] 2024-02-08T19:37:14.435 INFO:scrape: 2024-02-08T19:37:14.435 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 2024-02-08T19:37:14.435 INFO:scrape:2 jobs: ['7550332', '7550582'] 2024-02-08T19:37:14.435 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output'] 2024-02-08T19:37:14.435 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2024-02-08T19:37:14.435 INFO:scrape: 2024-02-08T19:37:14.435 INFO:scrape:Dead: None 2024-02-08T19:37:14.435 INFO:scrape:2 jobs: ['7550228', '7550330'] 2024-02-08T19:37:14.435 INFO:scrape: 2024-02-08T19:37:14.436 INFO:scrape:Failure: reached maximum tries (801) after waiting for 4800 seconds 2024-02-08T19:37:14.436 INFO:scrape:1 jobs: ['7550336'] 2024-02-08T19:37:14.436 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-radosbench}'] 2024-02-08T19:37:14.436 INFO:scrape: 2024-02-08T19:37:14.436 INFO:scrape:Failure: Command failed (workunit test cephadm/test_repos.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 2024-02-08T19:37:14.436 INFO:scrape:1 jobs: ['7550259'] 2024-02-08T19:37:14.436 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] 2024-02-08T19:37:14.436 INFO:scrape: 2024-02-08T19:37:14.436 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-02-08T19:37:14.436 INFO:scrape:2 jobs: ['7550209', '7550386'] 2024-02-08T19:37:14.436 INFO:scrape:suites intersection: ['2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-02-08T19:37:14.437 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/read', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] 2024-02-08T19:37:14.437 INFO:scrape: 2024-02-08T19:37:14.437 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2024-02-08T19:37:14.437 INFO:scrape:1 jobs: ['7550199'] 2024-02-08T19:37:14.437 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}'] 2024-02-08T19:37:14.437 INFO:scrape: 2024-02-08T19:37:14.437 INFO:scrape:Timeout 3h running clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-02-08T19:37:14.437 INFO:scrape:1 jobs: ['7550588'] 2024-02-08T19:37:14.437 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd}'] 2024-02-08T19:37:14.438 INFO:scrape: 2024-02-08T19:37:14.438 INFO:scrape:Assertion: 2024-02-07T17:53:27.598 INFO:journalctl@ceph.osd.0.smithi052.stdout:Feb 07 17:53:27 smithi052 bash[80502]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x565187c1298f] 2024-02-08T19:37:14.438 INFO:scrape:1 jobs: ['7550239'] 2024-02-08T19:37:14.438 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-02-08T19:37:14.438 INFO:scrape: 2024-02-08T19:37:14.438 INFO:scrape:Failure: Command failed (workunit test erasure-code/test-erasure-code.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh' 2024-02-08T19:37:14.438 INFO:scrape:1 jobs: ['7550303'] 2024-02-08T19:37:14.438 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/erasure-code}'] 2024-02-08T19:37:14.438 INFO:scrape: 2024-02-08T19:37:14.438 INFO:scrape:Failure: ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6486 (apt-get)'] 2024-02-08T19:37:14.439 INFO:scrape:1 jobs: ['7550248'] 2024-02-08T19:37:14.439 INFO:scrape:suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/balancer', 'supported-random-distro$/{ubuntu_latest}}'] 2024-02-08T19:37:14.439 INFO:scrape: 2024-02-08T19:37:14.439 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' 2024-02-08T19:37:14.439 INFO:scrape:1 jobs: ['7550420'] 2024-02-08T19:37:14.439 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_cephadm_timeout}'] 2024-02-08T19:37:14.439 INFO:scrape: 2024-02-08T19:37:14.439 INFO:scrape:Assertion: 2024-02-07T19:46:25.700 INFO:journalctl@ceph.osd.0.smithi139.stdout:Feb 07 19:46:25 smithi139 ceph-9c736ef4-c5ef-11ee-95b6-87774f69a715-osd-0[102324]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x5607a409398f] 2024-02-08T19:37:14.439 INFO:scrape:1 jobs: ['7550403'] 2024-02-08T19:37:14.439 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-02-08T19:37:14.439 INFO:scrape: 2024-02-08T19:37:14.439 INFO:scrape:Failure: Command failed (workunit test osd-backfill/osd-backfill-space.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh' 2024-02-08T19:37:14.440 INFO:scrape:1 jobs: ['7550520'] 2024-02-08T19:37:14.440 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd-backfill}'] 2024-02-08T19:37:14.440 INFO:scrape: 2024-02-08T19:37:14.454 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/ failed: 62 dead: 44 running: 0 waiting: 0 queued: 0 passed: 219 Fail ================================================================= [7550180] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:16 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550180/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550180/ "2024-02-07T16:59:37.072554+0000 mon.smithi042 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550186] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:38:16 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550186/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550186/ "2024-02-07T17:08:26.992577+0000 mon.a (mon.0) 345 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,e" in cluster log [7550191] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:13:29 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550191/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550191/ "2024-02-07T17:02:26.073415+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550192] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 01:00:23 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550192/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550192/ "2024-02-07T17:58:39.441420+0000 mon.a (mon.0) 187 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550196] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:17:45 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550196/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f50a46a54774d978fd2c85f555d4cd8 Command failed on smithi027 with status 6: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats' [7550197] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:12:57 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550197/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550197/ "2024-02-07T17:14:19.297960+0000 mon.smithi091 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550199] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 03:19:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550199/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=230ccc4e28c94f98b68c0c1aca918185 Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi104 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' [7550205] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} ----------------------------------------------------------------- time: 00:38:57 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550205/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550205/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea432307835c490782364f4452bf68cf Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- repair.sh' [7550207] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:19:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550207/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550207/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bdc3c7072acd4369b3a3672ea7b2353e Command failed on smithi133 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7550209] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:16:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550209/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cab96f2b91a346029b3f714bf548bf09 Command failed (workunit test rados/test.sh) on smithi097 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7550210] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:25:05 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550210/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550210/ "2024-02-07T17:20:57.640922+0000 mon.a (mon.0) 183 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7550219] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:21:26 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550219/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550219/ "2024-02-07T17:27:25.658962+0000 mon.smithi143 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550239] rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 02:02:24 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550239/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550239/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c0ec48c13ca41c187a9c67646bc4be0 Command failed (workunit test rbd/test_librbd_python.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' [7550243] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:12:21 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550243/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550243/ "2024-02-07T17:33:20.402918+0000 mon.smithi007 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550247] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:14:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550247/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550247/ "2024-02-07T17:33:57.310322+0000 mon.a (mon.0) 175 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7550259] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} ----------------------------------------------------------------- time: 00:08:08 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550259/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550259/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4629f93319b54b68b00ef65a1a148c62 Command failed (workunit test cephadm/test_repos.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' [7550269] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:14:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550269/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550269/ "2024-02-07T17:48:11.942825+0000 mon.smithi019 (mon.0) 268 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550274] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:44:23 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550274/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550274/ "2024-02-07T18:30:00.000244+0000 mon.a (mon.0) 1212 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7550275] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:00:58 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550275/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550275/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7550282] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} ----------------------------------------------------------------- time: 00:24:21 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550282/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550282/ "2024-02-07T18:08:27.348713+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550286] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:15:18 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550286/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550286/ "2024-02-07T18:01:44.878272+0000 mon.a (mon.0) 213 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7550292] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:19:50 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550292/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550292/ "2024-02-07T18:05:26.677605+0000 mon.smithi121 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550303] rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} ----------------------------------------------------------------- time: 00:24:03 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550303/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550303/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2dabc1f97f8343fd96f3c175210786b7 Command failed (workunit test erasure-code/test-erasure-code.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test- erasure-code.sh' [7550305] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:18:29 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550305/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550305/ "2024-02-07T18:18:16.496640+0000 mon.a (mon.0) 575 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7550311] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:19 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550311/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550311/ "2024-02-07T18:13:19.684989+0000 mon.smithi018 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550312] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:20:28 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550312/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550312/ Command failed on smithi027 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7550313] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:25:49 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550313/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550313/ "2024-02-07T18:30:00.000119+0000 mon.a (mon.0) 842 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7550332] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:08:16 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550332/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fd695d2a7d24a248a86938040b42184 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7550333] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:23:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550333/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550333/ "2024-02-07T18:37:42.187537+0000 mon.a (mon.0) 271 : cluster 3 [WRN] MON_DOWN: 1/5 mons down, quorum a,e,c,d" in cluster log [7550336] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 01:36:40 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550336/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f226cfe686ef4272b67b1bdabaa779cb reached maximum tries (801) after waiting for 4800 seconds [7550340] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:32 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550340/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550340/ "2024-02-07T18:40:39.296619+0000 mon.smithi110 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550345] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:19:49 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550345/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550345/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7550346] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:25:50 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550346/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550346/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7550348] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:19:20 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550348/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550348/ Command failed on smithi086 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7550352] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:22:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550352/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550352/ "2024-02-07T18:54:20.884812+0000 mon.a (mon.0) 196 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550354] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:36:07 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550354/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550354/ "2024-02-07T18:53:07.712729+0000 mon.a (mon.0) 198 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550355] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:18:25 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550355/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550355/ "2024-02-07T18:52:01.659514+0000 mon.smithi161 (mon.0) 254 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550375] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:12:59 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550375/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550375/ "2024-02-07T19:06:33.616186+0000 mon.smithi098 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550380] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 06:14:25 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550380/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcd44da936a64207a4cc143ecc6d8a3e Command failed (workunit test rados/test.sh) on smithi152 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7550386] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 03:14:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550386/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f91f06dcf2734af5b70cbaa36ddf0184 Command failed (workunit test rados/test.sh) on smithi023 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7550390] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:13:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550390/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550390/ "2024-02-07T19:19:17.066588+0000 mon.smithi086 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550394] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:23:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550394/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550394/ "2024-02-07T19:24:20.022231+0000 mon.a (mon.0) 195 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550403] rados/upgrade/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:21:26 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550403/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9556159a13d24220a785375f6d236889 Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [7550420] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_timeout} ----------------------------------------------------------------- time: 00:13:32 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550420/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550420/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b77539126e42467cb081910c37d712a2 Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_cephadm_timeout.py' [7550426] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 01:47:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550426/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550426/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7550432] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:19:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550432/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550432/ "2024-02-07T20:25:05.514367+0000 mon.smithi053 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550454] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} ----------------------------------------------------------------- time: 00:15:30 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550454/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550454/ "2024-02-07T20:47:21.239716+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550463] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:35:27 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550463/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550463/ "2024-02-07T21:10:00.000090+0000 mon.a (mon.0) 1794 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log [7550471] rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:14:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550471/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550471/ "2024-02-07T21:07:24.780279+0000 mon.smithi094 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550485] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/repair_test} ----------------------------------------------------------------- time: 00:26:48 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550485/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550485/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7955b824898c4ea092fc9cc74d678511 Command failed on smithi032 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7550508] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} ----------------------------------------------------------------- time: 00:25:07 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550508/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550508/ "2024-02-07T22:22:08.470048+0000 mon.a (mon.0) 270 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550514] rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:12:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550514/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550514/ "2024-02-07T22:56:07.268328+0000 mon.smithi102 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550520] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} ----------------------------------------------------------------- time: 01:42:15 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550520/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550520/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a374bc64de9b4fdda68630d844661709 Command failed (workunit test osd-backfill/osd-backfill-space.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd- backfill-space.sh' [7550538] rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:24:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550538/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550538/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db7962c0043a40deb6149b5fbbbe9c46 Command failed on smithi150 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7550544] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_orch_cli_mon} ----------------------------------------------------------------- time: 00:24:34 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550544/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550544/ "2024-02-08T01:26:17.669744+0000 mon.a (mon.0) 387 : cluster 3 [WRN] MON_DOWN: 1/5 mons down, quorum a,e,c,d" in cluster log [7550546] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:27:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550546/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550546/ "2024-02-08T01:34:40.310672+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550554] rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:19:42 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550554/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550554/ "2024-02-08T02:05:25.695705+0000 mon.smithi132 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log [7550576] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 00:56:55 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550576/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550576/ valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun [7550582] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:09:58 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550582/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e74b0e42bf8241968f3860d2398a5a27 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' [7550588] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} ----------------------------------------------------------------- time: 03:27:18 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550588/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550588/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a31cd66f675f4a75af374e6c8b7d5867 Command failed (workunit test osd/divergent-priors.sh) on smithi005 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4e4dcdef27eab6190bcda03dba829ee096072fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh' [7550590] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_rgw_multisite} ----------------------------------------------------------------- time: 00:20:03 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550590/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550590/ "2024-02-08T07:55:00.706026+0000 mon.a (mon.0) 196 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7550600] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_set_mon_crush_locations} ----------------------------------------------------------------- time: 00:24:44 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550600/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550600/ "2024-02-08T08:10:00.176445+0000 mon.a (mon.0) 509 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log Dead ================================================================= [7550184] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550184/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550184/ hit max job timeout [7550190] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550190/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550190/ hit max job timeout [7550195] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550195/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550195/ hit max job timeout [7550204] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550204/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550204/ hit max job timeout [7550208] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550208/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550208/ hit max job timeout [7550217] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550217/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550217/ hit max job timeout [7550225] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550225/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550225/ hit max job timeout [7550228] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550228/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550228/ Error reimaging machines: Failed to power on smithi064 [7550229] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550229/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550229/ hit max job timeout [7550237] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550237/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550237/ hit max job timeout [7550248] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:03:13 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550248/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550248/ sentry: https://sentry.ceph.com/organizations/ceph/?query=554583aaa36f40598b4bd0b1312802ad ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6486 (apt-get)'] [7550256] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550256/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550256/ hit max job timeout [7550258] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550258/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550258/ hit max job timeout [7550264] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550264/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550264/ hit max job timeout [7550267] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550267/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550267/ hit max job timeout [7550279] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550279/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550279/ hit max job timeout [7550290] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550290/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550290/ hit max job timeout [7550299] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550299/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550299/ hit max job timeout [7550301] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550301/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550301/ hit max job timeout [7550302] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550302/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550302/ hit max job timeout [7550304] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550304/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550304/ hit max job timeout [7550316] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550316/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550316/ hit max job timeout [7550327] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550327/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550327/ hit max job timeout [7550330] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550330/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550330/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7550347] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550347/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550347/ hit max job timeout [7550363] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550363/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550363/ hit max job timeout [7550371] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550371/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550371/ hit max job timeout [7550372] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550372/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550372/ hit max job timeout [7550391] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550391/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550391/ hit max job timeout [7550398] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550398/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550398/ hit max job timeout [7550401] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550401/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550401/ hit max job timeout [7550410] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550410/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550410/ hit max job timeout [7550412] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550412/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550412/ hit max job timeout [7550424] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550424/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550424/ hit max job timeout [7550461] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550461/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550461/ hit max job timeout [7550469] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550469/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550469/ hit max job timeout [7550476] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550476/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550476/ hit max job timeout [7550500] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550500/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550500/ hit max job timeout [7550534] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550534/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550534/ hit max job timeout [7550556] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550556/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550556/ hit max job timeout [7550558] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550558/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550558/ hit max job timeout [7550564] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550564/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550564/ hit max job timeout [7550574] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550574/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550574/ hit max job timeout [7550580] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550580/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550580/ hit max job timeout Pass ================================================================= [7550181] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:12:44 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550181/ [7550182] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} time: 00:18:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550182/ [7550183] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:13:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550183/ [7550185] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:28 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550185/ [7550187] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed} time: 00:10:18 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550187/ [7550188] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:09:27 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550188/ [7550189] rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} time: 00:07:10 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550189/ [7550193] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550193/ [7550194] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} time: 00:14:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550194/ [7550198] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:15:26 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550198/ [7550200] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:27 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550200/ [7550201] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:23:27 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550201/ [7550202] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:17:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550202/ [7550203] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:15:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550203/ [7550206] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:50 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550206/ [7550211] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550211/ [7550212] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:31:05 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550212/ [7550213] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:09:40 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550213/ [7550214] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_adoption} time: 00:12:02 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550214/ [7550215] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:38:56 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550215/ [7550216] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:09:59 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550216/ [7550218] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:17:10 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550218/ [7550220] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:29:49 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550220/ [7550221] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:31:40 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550221/ [7550222] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:13:22 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550222/ [7550223] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:28:31 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550223/ [7550224] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:11:04 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550224/ [7550226] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:55 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550226/ [7550227] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:34:16 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550227/ [7550230] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:30:15 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550230/ [7550231] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:51:07 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550231/ [7550232] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} time: 00:14:31 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550232/ [7550233] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/dashboard} time: 01:09:23 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550233/ [7550234] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:12:30 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550234/ [7550235] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:40:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550235/ [7550236] rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} time: 00:09:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550236/ [7550238] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} time: 00:15:08 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550238/ [7550240] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:20:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550240/ [7550241] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:11:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550241/ [7550242] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:18:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550242/ [7550244] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:08 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550244/ [7550245] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:07:57 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550245/ [7550246] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:10:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550246/ [7550249] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:13:48 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550249/ [7550250] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} time: 00:32:25 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550250/ [7550251] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:13:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550251/ [7550252] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:21:32 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550252/ [7550253] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:59 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550253/ [7550254] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:13:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550254/ [7550255] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:54 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550255/ [7550257] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:32 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550257/ [7550260] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:14:45 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550260/ [7550261] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:19:09 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550261/ [7550262] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550262/ [7550263] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_timeout} time: 00:13:47 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550263/ [7550265] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:20:20 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550265/ [7550266] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550266/ [7550268] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/failover} time: 00:13:10 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550268/ [7550270] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} time: 00:19:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550270/ [7550271] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:13:42 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550271/ [7550272] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550272/ [7550273] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:21:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550273/ [7550276] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} time: 00:13:48 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550276/ [7550277] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:17:13 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550277/ [7550278] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:10:05 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550278/ [7550280] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:11 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550280/ [7550281] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:27:02 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550281/ [7550283] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:14:07 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550283/ [7550284] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:09:42 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550284/ [7550285] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:17:21 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550285/ [7550287] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:05 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550287/ [7550288] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:56 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550288/ [7550289] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:22:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550289/ [7550291] rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:16:10 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550291/ [7550293] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:14:30 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550293/ [7550294] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:27:58 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550294/ [7550295] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:19 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550295/ [7550296] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:09 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550296/ [7550297] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:13:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550297/ [7550298] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} time: 00:14:37 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550298/ [7550300] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:14:37 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550300/ [7550306] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:22 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550306/ [7550307] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:33:57 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550307/ [7550308] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:18:58 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550308/ [7550309] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:22:59 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550309/ [7550310] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:16:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550310/ [7550314] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550314/ [7550315] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:14:22 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550315/ [7550317] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:20:03 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550317/ [7550318] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:02 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550318/ [7550319] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-stupid} tasks/e2e} time: 00:22:48 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550319/ [7550320] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:24 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550320/ [7550321] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:26:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550321/ [7550322] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:18:49 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550322/ [7550323] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:42 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550323/ [7550324] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_striper} time: 00:07:48 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550324/ [7550325] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}} time: 00:08:49 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550325/ [7550326] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:09:49 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550326/ [7550328] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} time: 00:20:25 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550328/ [7550329] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_latest} tasks/module_selftest} time: 00:23:19 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550329/ [7550331] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} time: 00:12:15 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550331/ [7550334] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} time: 00:14:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550334/ [7550335] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:15:38 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550335/ [7550337] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:19:01 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550337/ [7550338] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:11:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550338/ [7550339] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:40 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550339/ [7550341] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:35 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550341/ [7550342] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}} time: 00:15:48 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550342/ [7550343] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:23:07 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550343/ [7550344] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} time: 00:13:26 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550344/ [7550349] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:20:12 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550349/ [7550350] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:33:19 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550350/ [7550351] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:23:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550351/ [7550353] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:10:18 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550353/ [7550356] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:20:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550356/ [7550357] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} time: 00:28:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550357/ [7550358] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} time: 00:16:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550358/ [7550359] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/per_module_finisher_stats} time: 00:08:44 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550359/ [7550360] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550360/ [7550361] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:19:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550361/ [7550362] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} time: 00:07:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550362/ [7550364] rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_adoption} time: 00:08:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550364/ [7550365] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} time: 00:12:47 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550365/ [7550366] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:17:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550366/ [7550367] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 01:13:50 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550367/ [7550368] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:21 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550368/ [7550369] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}} time: 00:32:13 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550369/ [7550370] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/dedup-io-snaps} time: 00:11:34 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550370/ [7550373] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:13 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550373/ [7550374] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix} time: 00:18:54 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550374/ [7550376] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}} time: 00:23:18 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550376/ [7550377] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:15:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550377/ [7550378] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:07 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550378/ [7550379] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} time: 00:11:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550379/ [7550381] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550381/ [7550382] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:24:42 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550382/ [7550383] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:14:15 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550383/ [7550384] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:12:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550384/ [7550385] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} time: 00:19:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550385/ [7550387] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:33:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550387/ [7550388] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_latest} tasks/progress} time: 00:20:14 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550388/ [7550389] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:11 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550389/ [7550392] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:49:43 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550392/ [7550393] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:53:32 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550393/ [7550395] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:59 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550395/ [7550396] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} time: 00:05:47 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550396/ [7550397] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery} time: 00:14:11 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550397/ [7550399] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:24:08 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550399/ [7550400] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-hybrid} tasks/dashboard} time: 01:08:55 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550400/ [7550402] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} time: 00:26:32 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550402/ [7550404] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench} time: 00:59:16 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550404/ [7550405] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}} time: 00:10:02 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550405/ [7550406] rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:14:47 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550406/ [7550407] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:12:50 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550407/ [7550408] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:15:59 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550408/ [7550409] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:09:06 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550409/ [7550414] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} time: 00:18:47 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550414/ [7550416] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:11:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550416/ [7550418] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:10:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550418/ [7550422] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:37 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550422/ [7550428] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_latest} tasks/prometheus} time: 00:10:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550428/ [7550430] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 02:04:04 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550430/ [7550434] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:14:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550434/ [7550436] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_latest}} time: 02:10:49 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550436/ [7550438] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:14:54 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550438/ [7550440] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:08:43 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550440/ [7550442] rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} time: 00:18:50 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550442/ [7550444] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:14:34 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550444/ [7550446] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:17:48 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550446/ [7550448] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:47:08 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550448/ [7550451] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} time: 00:30:40 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550451/ [7550453] rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} time: 01:00:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550453/ [7550456] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:07:15 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550456/ [7550458] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:10:09 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550458/ [7550465] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:24:53 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550465/ [7550467] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:22:23 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550467/ [7550473] rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:20 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550473/ [7550475] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:21:43 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550475/ [7550479] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:20:01 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550479/ [7550481] rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} time: 00:13:02 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550481/ [7550483] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} time: 00:18:25 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550483/ [7550488] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:19:18 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550488/ [7550491] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:14:23 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550491/ [7550494] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:11:20 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550494/ [7550496] rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:39 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550496/ [7550498] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:08:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550498/ [7550502] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:17:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550502/ [7550504] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:12:18 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550504/ [7550506] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}} time: 00:07:57 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550506/ [7550510] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:23:19 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550510/ [7550512] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} time: 00:07:25 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550512/ [7550516] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:09:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550516/ [7550518] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:31:32 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550518/ [7550522] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:32:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550522/ [7550524] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} time: 00:20:40 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550524/ [7550526] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:13:27 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550526/ [7550528] rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_orch_cli} time: 00:15:10 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550528/ [7550530] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} time: 00:08:40 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550530/ [7550532] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:09:37 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550532/ [7550536] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:27:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550536/ [7550540] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:13:06 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550540/ [7550542] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:13:33 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550542/ [7550548] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:52 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550548/ [7550550] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:08 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550550/ [7550552] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:12:25 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550552/ [7550560] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} tasks/e2e} time: 00:30:56 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550560/ [7550562] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_latest} tasks/crash} time: 00:10:13 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550562/ [7550566] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:21:06 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550566/ [7550568] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:45 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550568/ [7550570] rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:15:29 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550570/ [7550572] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} time: 00:10:09 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550572/ [7550578] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} time: 00:13:03 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550578/ [7550584] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:13:00 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550584/ [7550586] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}} time: 00:09:51 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550586/ [7550592] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} time: 00:13:03 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550592/ [7550594] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:13:46 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550594/ [7550596] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925} time: 00:12:47 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550596/ [7550598] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} time: 00:10:09 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550598/ [7550602] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:11:42 info: https://pulpito.ceph.com/yuriw-2024-02-07_16:39:46-rados-wip-yuri5-testing-2024-02-06-1819-distro-default-smithi/7550602/