Found 93 jobs Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Saw end of BT but not start Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) , 1: /lib64/libc.so.6(+0x54db0) [0x7f7f9b054db0] , 2: /lib64/libc.so.6(+0xa154c) [0x7f7f9b0a154c] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 71 distinct failure reasons Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55fdbebcd638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f67176c7520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55fdbebcd638] 6: ceph-osd(+0x3f7799) [0x55fdbebcd799] 7: ceph-osd(+0x3e8ab8) [0x55fdbebbeab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55fdbf11abd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55fdbf12f8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55fdbf190e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55fdbf19a10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55fdbf19ac88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55fdbf19f1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55fdbf1a0013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x55fdbf08dbbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55fdbf08f6c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55fdbeeabaf6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55fdbee0317d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55fdbed58247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fdbef9f609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55fdbed62c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55fdbf25e293] 23: ceph-osd(+0xa887f4) [0x55fdbf25e7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f6717719b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f67177aba00] 1 jobs: ['7600484'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/write_fadvise_dontneed}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55fd3e6cd3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8b54c54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8b54ca154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55fd3e6cd3df] 6: ceph-osd(+0x3f4540) [0x55fd3e6cd540] 7: ceph-osd(+0x3e6182) [0x55fd3e6bf182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55fd3ec34450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55fd3ec3e49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55fd3ecaea61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55fd3ecb209b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55fd3ecb2c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55fd3eca09e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55fd3eca1833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55fd3eba221d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55fd3e961492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55fd3e928f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55fd3e91f715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55fd3e85adc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd3eaa5889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55fd3e876167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55fd3ed7697a] 23: ceph-osd(+0xa9df24) [0x55fd3ed76f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f8b54c9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8b54c3f450] 1 jobs: ['7600542'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5558729af3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7faf28e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7faf28ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5558729af3df] 6: ceph-osd(+0x3f4540) [0x5558729af540] 7: ceph-osd(+0x3e6182) [0x5558729a1182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x555872f16450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x555872f2049d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x555872f90a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x555872f9409b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x555872f94c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x555872f829e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x555872f83833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x555872e8421d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x555872c43492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x555872c0af47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x555872c01715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x555872b3cdc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x555872d87889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x555872b58167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55587305897a] 23: ceph-osd(+0xa9df24) [0x555873058f24] 24: /lib64/libc.so.6(+0x9f802) [0x7faf28e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7faf28e3f450] 1 jobs: ['7600471'] suites: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/one', 'workloads/snaps-few-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5618d199d3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff42d054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7ff42d0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5618d199d3df] 6: ceph-osd(+0x3f4540) [0x5618d199d540] 7: ceph-osd(+0x3e6182) [0x5618d198f182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5618d1f04450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5618d1f0e49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5618d1f7ea61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5618d1f8209b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5618d1f82c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5618d1f709e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5618d1f71833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x5618d1e73212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5618d1e67ed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5618d1ca7ed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x5618d1bf181d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5618d1b2adc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5618d1d75889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5618d1b46167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5618d204697a] 23: ceph-osd(+0xa9df24) [0x5618d2046f24] 24: /lib64/libc.so.6(+0x9f802) [0x7ff42d09f802] 25: /lib64/libc.so.6(+0x3f450) [0x7ff42d03f450] 1 jobs: ['7600489'] suites: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/mon-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/sync', 'workloads/rados_5925}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a9a69b7638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f9e338cc520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a9a69b7638] 6: ceph-osd(+0x3f7799) [0x55a9a69b7799] 7: ceph-osd(+0x3e8ab8) [0x55a9a69a8ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55a9a6f04bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55a9a6f198fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55a9a6f7ae71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55a9a6f8410b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55a9a6f84c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55a9a6f891f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55a9a6f8a013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x55a9a6e77bbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55a9a6e796c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55a9a6c95af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55a9a6bed17d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55a9a6b42247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a9a6d89609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55a9a6b4cc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55a9a7048293] 23: ceph-osd(+0xa887f4) [0x55a9a70487f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f9e3391eb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f9e339b0a00] 1 jobs: ['7600518'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x557b83d1e3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f5425854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f54258a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x557b83d1e3df] 6: ceph-osd(+0x3f4540) [0x557b83d1e540] 7: ceph-osd(+0x3e6182) [0x557b83d10182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x557b84285450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x557b8428f49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x557b842ffa61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x557b8430309b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x557b84303c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x557b842f19e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x557b842f2833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x557b841f321d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x557b83fb2492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x557b83f79f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x557b83f70715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x557b83eabdc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x557b840f6889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x557b83ec7167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x557b843c797a] 23: ceph-osd(+0xa9df24) [0x557b843c7f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f542589f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f542583f450] 1 jobs: ['7600487'] suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Assertion: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x4fc384] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x4fc384] 2: ceph-osd(+0x3f4540) [0x4fc540] 3: ceph-osd(+0x3e6182) [0x4ee182] 4: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0xa63450] 5: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0xa6d49d] 6: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0xadda61] 7: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0xae109b] 8: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0xae1c78] 9: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0xacf9e5] 10: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0xad0833] 11: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x9d121d] 12: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x790492] 13: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x757f47] 14: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x74e715] 15: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x689dc7] 16: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8d4889] 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6a5167] 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xba597a] 19: ceph-osd(+0xa9df24) [0xba5f24] 20: /lib64/libc.so.6(+0x9f802) [0x54eb802] 2 jobs: ['7600523', '7600498'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'validater/valgrind}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'validater/valgrind}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x557c7bec9638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f69cabb8520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x557c7bec9638] 6: ceph-osd(+0x3f7799) [0x557c7bec9799] 7: ceph-osd(+0x3e8ab8) [0x557c7bebaab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x557c7c416bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x557c7c42b8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x557c7c48ce71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x557c7c49610b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x557c7c496c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x557c7c49b1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x557c7c49c013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x557c7c38da7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x557c7c142142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x557c7c11828f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x557c7c11be09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x557c7c054247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x557c7c29b609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x557c7c05ec33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x557c7c55a293] 23: ceph-osd(+0xa887f4) [0x557c7c55a7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f69cac0ab43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f69cac9ca00] 1 jobs: ['7600508'] suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56439af873df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f9a6d454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f9a6d4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56439af873df] 6: ceph-osd(+0x3f4540) [0x56439af87540] 7: ceph-osd(+0x3e6182) [0x56439af79182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56439b4ee450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56439b4f849d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56439b568a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56439b56c09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56439b56cc78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56439b55a9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56439b55b833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x56439b45d212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x56439b451ed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x56439b291ed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x56439b1db81d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56439b114dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56439b35f889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56439b130167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56439b63097a] 23: ceph-osd(+0xa9df24) [0x56439b630f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f9a6d49f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f9a6d43f450] 1 jobs: ['7600486'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/lockdep}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x565148bce3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f3f1b054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f3f1b0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x565148bce3df] 6: ceph-osd(+0x3f4540) [0x565148bce540] 7: ceph-osd(+0x3e6182) [0x565148bc0182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x565149135450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56514913f49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5651491afa61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5651491b309b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5651491b3c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5651491a19e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5651491a2833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5651490a321d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x565148e62492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x565148e29f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x565148e20715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x565148d5bdc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565148fa6889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x565148d77167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56514927797a] 23: ceph-osd(+0xa9df24) [0x565149277f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f3f1b09f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f3f1b03f450] 1 jobs: ['7600515'] suites: ['debug/mgr', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zlib}', 'tasks/dashboard}'] Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2 jobs: ['7600454', '7600497'] suites intersection: [] suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x555f4b2b0638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fe7ee523520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x555f4b2b0638] 6: ceph-osd(+0x3f7799) [0x555f4b2b0799] 7: ceph-osd(+0x3e8ab8) [0x555f4b2a1ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x555f4b7fdbd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x555f4b8128fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x555f4b873e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x555f4b87d10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x555f4b87dc88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x555f4b8821f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x555f4b883013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x555f4b770bbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x555f4b7726c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x555f4b58eaf6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x555f4b4e617d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x555f4b43b247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x555f4b682609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x555f4b445c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x555f4b941293] 23: ceph-osd(+0xa887f4) [0x555f4b9417f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fe7ee575b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fe7ee607a00] 1 jobs: ['7600485'] suites: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] Failure: Command failed on smithi059 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_snap_mapper' 6 jobs: ['7600528', '7600521', '7600476', '7600457', '7600540', '7600493'] suites intersection: [] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'agent/on', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/singleton-nomsgr/{all/ceph-snapmapper', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'task/test_ca_signed_key}', 'thrashers/none', 'thrashosds-health', 'workloads/redirect_promote_tests}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5626f73923df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f63b8854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f63b88a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5626f73923df] 6: ceph-osd(+0x3f4540) [0x5626f7392540] 7: ceph-osd(+0x3e6182) [0x5626f7384182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5626f78f9450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5626f790349d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5626f7973a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5626f797709b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5626f7977c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5626f79659e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5626f7966833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5626f786721d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5626f7626492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5626f75edf47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5626f75e4715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5626f751fdc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626f776a889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5626f753b167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5626f7a3b97a] 23: ceph-osd(+0xa9df24) [0x5626f7a3bf24] 24: /lib64/libc.so.6(+0x9f802) [0x7f63b889f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f63b883f450] 1 jobs: ['7600479'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5606cf6853df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8d27054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8d270a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5606cf6853df] 6: ceph-osd(+0x3f4540) [0x5606cf685540] 7: ceph-osd(+0x3e6182) [0x5606cf677182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5606cfbec450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5606cfbf649d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5606cfc66a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5606cfc6a09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5606cfc6ac78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5606cfc589e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5606cfc59833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x5606cfb5b212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5606cfb4fed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5606cf98fed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x5606cf8d981d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5606cf812dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5606cfa5d889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5606cf82e167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5606cfd2e97a] 23: ceph-osd(+0xa9df24) [0x5606cfd2ef24] 24: /lib64/libc.so.6(+0x9f802) [0x7f8d2709f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8d2703f450] 1 jobs: ['7600478'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/sync-many', 'workloads/pool-create-delete}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5592d7fb03df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1786054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f17860a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5592d7fb03df] 6: ceph-osd(+0x3f4540) [0x5592d7fb0540] 7: ceph-osd(+0x3e6182) [0x5592d7fa2182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5592d8517450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5592d852149d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5592d8591a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5592d859509b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5592d8595c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5592d85839e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5592d8584833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x5592d8486212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5592d847aed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5592d82baed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x5592d820481d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5592d813ddc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5592d8388889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5592d8159167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5592d865997a] 23: ceph-osd(+0xa9df24) [0x5592d8659f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f178609f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f178603f450] 1 jobs: ['7600495'] suites: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5625daf673df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7faaf6854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7faaf68a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5625daf673df] 6: ceph-osd(+0x3f4540) [0x5625daf67540] 7: ceph-osd(+0x3e6182) [0x5625daf59182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5625db4ce450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5625db4d849d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5625db548a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5625db54c09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5625db54cc78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5625db53a9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x5625db53b833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5625db43c21d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5625db1fb492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5625db1c2f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5625db1b9715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5625db0f4dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5625db33f889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x5625db110167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x5625db61097a] 23: ceph-osd(+0xa9df24) [0x5625db610f24] 24: /lib64/libc.so.6(+0x9f802) [0x7faaf689f802] 25: /lib64/libc.so.6(+0x3f450) [0x7faaf683f450] 1 jobs: ['7600531'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] Failure: "2024-03-15T00:00:00.000154+0000 mon.a (mon.0) 1177 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log 1 jobs: ['7600536'] suites: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/quincy', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/default', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564784ac43df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fbbb0454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fbbb04a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564784ac43df] 6: ceph-osd(+0x3f4540) [0x564784ac4540] 7: ceph-osd(+0x3e6182) [0x564784ab6182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56478502b450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56478503549d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x5647850a5a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5647850a909b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5647850a9c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5647850979e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x564785098833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x564784f9921d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x564784d58492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x564784d1ff47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x564784d16715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x564784c51dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x564784e9c889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x564784c6d167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56478516d97a] 23: ceph-osd(+0xa9df24) [0x56478516df24] 24: /lib64/libc.so.6(+0x9f802) [0x7fbbb049f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fbbb043f450] 1 jobs: ['7600519'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/redirect}'] Failure: Command failed (workunit test rados/test_python.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f6948ddec49f204a3bb696c60ce2bd00b4465ea1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh' 1 jobs: ['7600475'] suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_python}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f6948ddec49f204a3bb696c60ce2bd00b4465ea1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 1 jobs: ['7600461'] suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_cephadm}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x563e9e7de3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fcfa6254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fcfa62a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x563e9e7de3df] 6: ceph-osd(+0x3f4540) [0x563e9e7de540] 7: ceph-osd(+0x3e6182) [0x563e9e7d0182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x563e9ed45450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x563e9ed4f49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x563e9edbfa61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x563e9edc309b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x563e9edc3c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x563e9edb19e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x563e9edb2833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x563e9ecb321d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x563e9ea72492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x563e9ea39f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x563e9ea30715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x563e9e96bdc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563e9ebb6889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x563e9e987167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x563e9ee8797a] 23: ceph-osd(+0xa9df24) [0x563e9ee87f24] 24: /lib64/libc.so.6(+0x9f802) [0x7fcfa629f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fcfa623f450] 1 jobs: ['7600506'] suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/rados_workunit_loadgen_mix}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x563e42e46638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7b13660520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x563e42e46638] 6: ceph-osd(+0x3f7799) [0x563e42e46799] 7: ceph-osd(+0x3e8ab8) [0x563e42e37ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x563e43393bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x563e433a88fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x563e43409e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x563e4341310b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x563e43413c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x563e434181f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x563e43419013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x563e43306bbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x563e433086c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x563e43124af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x563e4307c17d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x563e42fd1247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563e43218609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563e42fdbc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x563e434d7293] 23: ceph-osd(+0xa887f4) [0x563e434d77f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f7b136b2b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f7b13744a00] 1 jobs: ['7600470'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/none', 'thrashosds-health', 'workloads/redirect_set_object}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5628296f6638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7022fb3520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5628296f6638] 6: ceph-osd(+0x3f7799) [0x5628296f6799] 7: ceph-osd(+0x3e8ab8) [0x5628296e7ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562829c43bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x562829c588fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x562829cb9e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x562829cc310b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x562829cc3c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x562829cc81f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x562829cc9013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x562829bbaa7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56282996f142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x56282994528f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x562829948e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x562829881247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562829ac8609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x56282988bc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x562829d87293] 23: ceph-osd(+0xa887f4) [0x562829d877f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f7023005b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f7023097a00] 1 jobs: ['7600533'] suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5652e3c2f638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f368dd75520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5652e3c2f638] 6: ceph-osd(+0x3f7799) [0x5652e3c2f799] 7: ceph-osd(+0x3e8ab8) [0x5652e3c20ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5652e417cbd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5652e41918fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5652e41f2e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5652e41fc10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5652e41fcc88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5652e42011f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5652e4202013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5652e40f3a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5652e3ea8142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5652e3e7e28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5652e3e81e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5652e3dba247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5652e4001609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5652e3dc4c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5652e42c0293] 23: ceph-osd(+0xa887f4) [0x5652e42c07f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f368ddc7b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f368de59a00] 1 jobs: ['7600480'] suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x555d959043df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f5169454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f51694a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x555d959043df] 6: ceph-osd(+0x3f4540) [0x555d95904540] 7: ceph-osd(+0x3e6182) [0x555d958f6182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x555d95e6b450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x555d95e7549d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x555d95ee5a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x555d95ee909b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x555d95ee9c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x555d95ed79e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x555d95ed8833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x555d95dd921d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x555d95b98492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x555d95b5ff47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x555d95b56715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x555d95a91dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x555d95cdc889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x555d95aad167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x555d95fad97a] 23: ceph-osd(+0xa9df24) [0x555d95fadf24] 24: /lib64/libc.so.6(+0x9f802) [0x7f516949f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f516943f450] 1 jobs: ['7600503'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560f3eb1a638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc956262520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x560f3eb1a638] 6: ceph-osd(+0x3f7799) [0x560f3eb1a799] 7: ceph-osd(+0x3e8ab8) [0x560f3eb0bab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x560f3f067bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x560f3f07c8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x560f3f0dde71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x560f3f0e710b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x560f3f0e7c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x560f3f0ec1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x560f3f0ed013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x560f3efdabbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x560f3efdc6c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x560f3edf8af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x560f3ed5017d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560f3eca5247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560f3eeec609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560f3ecafc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x560f3f1ab293] 23: ceph-osd(+0xa887f4) [0x560f3f1ab7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc9562b4b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc956346a00] 1 jobs: ['7600514'] suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_workunit_loadgen_mostlyread}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5627988d2638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc5a22e9520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5627988d2638] 6: ceph-osd(+0x3f7799) [0x5627988d2799] 7: ceph-osd(+0x3e8ab8) [0x5627988c3ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562798e1fbd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x562798e348fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x562798e95e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x562798e9f10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x562798e9fc88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x562798ea41f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x562798ea5013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x562798d96a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x562798b4b142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x562798b2128f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x562798b24e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x562798a5d247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562798ca4609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x562798a67c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x562798f63293] 23: ceph-osd(+0xa887f4) [0x562798f637f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc5a233bb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc5a23cda00] 1 jobs: ['7600505'] suites: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ccbdf713df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fbbf2454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fbbf24a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ccbdf713df] 6: ceph-osd(+0x3f4540) [0x55ccbdf71540] 7: ceph-osd(+0x3e6182) [0x55ccbdf63182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ccbe4d8450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ccbe4e249d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ccbe552a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ccbe55609b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ccbe556c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ccbe5449e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ccbe545833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ccbe44621d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ccbe205492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55ccbe1ccf47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55ccbe1c3715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ccbe0fedc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ccbe349889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ccbe11a167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ccbe61a97a] 23: ceph-osd(+0xa9df24) [0x55ccbe61af24] 24: /lib64/libc.so.6(+0x9f802) [0x7fbbf249f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fbbf243f450] 1 jobs: ['7600530'] suites: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/connectivity', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zstd}', 'supported-random-distro$/{centos_latest}', 'tasks/workunits}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c71b136638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fef2f3d0520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c71b136638] 6: ceph-osd(+0x3f7799) [0x55c71b136799] 7: ceph-osd(+0x3e8ab8) [0x55c71b127ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55c71b683bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55c71b6988fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55c71b6f9e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55c71b70310b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55c71b703c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55c71b7081f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55c71b709013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55c71b5faa7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55c71b3af142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55c71b38528f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55c71b388e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55c71b2c1247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c71b508609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55c71b2cbc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55c71b7c7293] 23: ceph-osd(+0xa887f4) [0x55c71b7c77f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fef2f422b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fef2f4b4a00] 1 jobs: ['7600488'] suites: ['mon_election/connectivity', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/wpq_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/radosbench_4M_seq_read}'] Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f6948ddec49f204a3bb696c60ce2bd00b4465ea1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 1 jobs: ['7600456'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/scrub}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55e6f4661638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fdbb7e40520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55e6f4661638] 6: ceph-osd(+0x3f7799) [0x55e6f4661799] 7: ceph-osd(+0x3e8ab8) [0x55e6f4652ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e6f4baebd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55e6f4bc38fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55e6f4c24e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e6f4c2e10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e6f4c2ec88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55e6f4c331f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55e6f4c34013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x55e6f4b21bbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55e6f4b236c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55e6f493faf6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55e6f489717d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e6f47ec247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e6f4a33609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e6f47f6c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e6f4cf2293] 23: ceph-osd(+0xa887f4) [0x55e6f4cf27f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fdbb7e92b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fdbb7f24a00] 1 jobs: ['7600501'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-snaps}'] Dead: b'2024-03-15T11:33:57.623 DEBUG:teuthology.exit:Finished running handlers' 4 jobs: ['7600520', '7600467', '7600516', '7600481'] suites intersection: ['rados'] suites union: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/none', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/singleton/{all/mon-config', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55f67c953638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1deded1520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55f67c953638] 6: ceph-osd(+0x3f7799) [0x55f67c953799] 7: ceph-osd(+0x3e8ab8) [0x55f67c944ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55f67cea0bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55f67ceb58fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55f67cf16e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55f67cf2010b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55f67cf20c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55f67cf251f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55f67cf26013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x55f67ce13bbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55f67ce156c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55f67cc31af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55f67cb8917d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55f67cade247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f67cd25609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55f67cae8c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55f67cfe4293] 23: ceph-osd(+0xa887f4) [0x55f67cfe47f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1dedf23b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f1dedfb5a00] 1 jobs: ['7600532'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/snaps-few-objects-localized}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a3ffdad638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f42178c8520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a3ffdad638] 6: ceph-osd(+0x3f7799) [0x55a3ffdad799] 7: ceph-osd(+0x3e8ab8) [0x55a3ffd9eab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55a4002fabd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55a40030f8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55a400370e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55a40037a10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55a40037ac88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55a40037f1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55a400380013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x55a40026dbbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55a40026f6c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55a40008baf6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55a3fffe317d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55a3fff38247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a40017f609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55a3fff42c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55a40043e293] 23: ceph-osd(+0xa887f4) [0x55a40043e7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f421791ab43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f42179aca00] 1 jobs: ['7600464'] suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564638f8c3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f53eea54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f53eeaa154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x564638f8c3df] 6: ceph-osd(+0x3f4540) [0x564638f8c540] 7: ceph-osd(+0x3e6182) [0x564638f7e182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5646394f3450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5646394fd49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56463956da61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56463957109b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x564639571c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56463955f9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x564639560833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56463946121d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x564639220492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5646391e7f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5646391de715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x564639119dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x564639364889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x564639135167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56463963597a] 23: ceph-osd(+0xa9df24) [0x564639635f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f53eea9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f53eea3f450] 1 jobs: ['7600494'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-pool-snaps}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5624cac3b638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8e6ab20520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5624cac3b638] 6: ceph-osd(+0x3f7799) [0x5624cac3b799] 7: ceph-osd(+0x3e8ab8) [0x5624cac2cab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5624cb188bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5624cb19d8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5624cb1fee71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5624cb20810b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5624cb208c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5624cb20d1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5624cb20e013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x5624cb0fbbbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5624cb0fd6c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5624caf19af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x5624cae7117d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5624cadc6247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5624cb00d609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5624cadd0c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5624cb2cc293] 23: ceph-osd(+0xa887f4) [0x5624cb2cc7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f8e6ab72b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f8e6ac04a00] 1 jobs: ['7600529'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/small-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56214b11c3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f8a3ac54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f8a3aca154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56214b11c3df] 6: ceph-osd(+0x3f4540) [0x56214b11c540] 7: ceph-osd(+0x3e6182) [0x56214b10e182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56214b683450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56214b68d49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56214b6fda61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56214b70109b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56214b701c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56214b6ef9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56214b6f0833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x56214b5f2212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x56214b5e6ed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x56214b426ed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x56214b37081d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56214b2a9dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56214b4f4889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56214b2c5167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56214b7c597a] 23: ceph-osd(+0xa9df24) [0x56214b7c5f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f8a3ac9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f8a3ac3f450] 1 jobs: ['7600527'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/thrash-rados/{thrash-rados', 'supported-random-distro$/{centos_latest}}', 'thrashosds-health}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ee3c74b638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f366b9f9520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ee3c74b638] 6: ceph-osd(+0x3f7799) [0x55ee3c74b799] 7: ceph-osd(+0x3e8ab8) [0x55ee3c73cab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ee3cc98bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55ee3ccad8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55ee3cd0ee71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ee3cd1810b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ee3cd18c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55ee3cd1d1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55ee3cd1e013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x55ee3cc0bbbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55ee3cc0d6c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55ee3ca29af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55ee3c98117d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ee3c8d6247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ee3cb1d609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55ee3c8e0c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55ee3cddc293] 23: ceph-osd(+0xa887f4) [0x55ee3cddc7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f366ba4bb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f366badda00] 1 jobs: ['7600472'] suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5651135643df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f29ed054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f29ed0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5651135643df] 6: ceph-osd(+0x3f4540) [0x565113564540] 7: ceph-osd(+0x3e6182) [0x565113556182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x565113acb450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x565113ad549d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x565113b45a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x565113b4909b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x565113b49c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x565113b379e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x565113b38833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x565113a3921d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5651137f8492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5651137bff47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5651137b6715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5651136f1dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56511393c889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56511370d167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x565113c0d97a] 23: ceph-osd(+0xa9df24) [0x565113c0df24] 24: /lib64/libc.so.6(+0x9f802) [0x7f29ed09f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f29ed03f450] 1 jobs: ['7600513'] suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}'] Failure: Command failed (workunit test rados/test.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f6948ddec49f204a3bb696c60ce2bd00b4465ea1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 6 jobs: ['7600539', '7600463', '7600502', '7600459', '7600474', '7600509'] suites intersection: ['rados'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/9-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/read', 'd-balancer/upmap-read', 'd-thrash/none', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/monthrash/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'thrashers/careful', 'thrashers/force-sync-many', 'thrashers/pggrow', 'thrashosds-health', 'validater/valgrind}', 'workloads/rados_api_tests}'] Assertion: 2024-03-14T23:51:06.440 INFO:journalctl@ceph.osd.1.smithi148.stdout:Mar 14 23:51:06 smithi148 ceph-c32e2bae-e25c-11ee-95c9-87774f69a715-osd-1[79353]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x56500d263d8f] 1 jobs: ['7600538'] suites: ['debug/mgr', 'mon_election/classic', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c443b27638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f9674271520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55c443b27638] 6: ceph-osd(+0x3f7799) [0x55c443b27799] 7: ceph-osd(+0x3e8ab8) [0x55c443b18ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55c444074bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55c4440898fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55c4440eae71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55c4440f410b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55c4440f4c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55c4440f91f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55c4440fa013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55c443feba7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55c443da0142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55c443d7628f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55c443d79e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55c443cb2247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c443ef9609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55c443cbcc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55c4441b8293] 23: ceph-osd(+0xa887f4) [0x55c4441b87f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f96742c3b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f9674355a00] 1 jobs: ['7600482'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/snaps-few-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ad4b93b638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f936a318520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ad4b93b638] 6: ceph-osd(+0x3f7799) [0x55ad4b93b799] 7: ceph-osd(+0x3e8ab8) [0x55ad4b92cab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ad4be88bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55ad4be9d8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55ad4befee71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ad4bf0810b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ad4bf08c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55ad4bf0d1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55ad4bf0e013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ad4bdffa7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ad4bbb4142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55ad4bb8a28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55ad4bb8de09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ad4bac6247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ad4bd0d609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55ad4bad0c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55ad4bfcc293] 23: ceph-osd(+0xa887f4) [0x55ad4bfcc7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f936a36ab43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f936a3fca00] 1 jobs: ['7600469'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/redirect_promote_tests}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x557e7ef81638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fe028185520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x557e7ef81638] 6: ceph-osd(+0x3f7799) [0x557e7ef81799] 7: ceph-osd(+0x3e8ab8) [0x557e7ef72ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x557e7f4cebd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x557e7f4e38fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x557e7f544e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x557e7f54e10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x557e7f54ec88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x557e7f5531f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x557e7f554013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x557e7f445a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x557e7f1fa142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x557e7f1d028f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x557e7f1d3e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x557e7f10c247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x557e7f353609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x557e7f116c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x557e7f612293] 23: ceph-osd(+0xa887f4) [0x557e7f6127f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fe0281d7b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fe028269a00] 1 jobs: ['7600455'] suites: ['mon_election/classic', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_1Shard_16Threads', 'settings/optimized', 'ubuntu_latest', 'workloads/fio_4M_rand_rw}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x556ca95023df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f3748454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f37484a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x556ca95023df] 6: ceph-osd(+0x3f4540) [0x556ca9502540] 7: ceph-osd(+0x3e6182) [0x556ca94f4182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x556ca9a69450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x556ca9a7349d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x556ca9ae3a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x556ca9ae709b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x556ca9ae7c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x556ca9ad59e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x556ca9ad6833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x556ca99d721d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x556ca9796492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x556ca975df47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x556ca9754715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x556ca968fdc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x556ca98da889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x556ca96ab167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x556ca9bab97a] 23: ceph-osd(+0xa9df24) [0x556ca9babf24] 24: /lib64/libc.so.6(+0x9f802) [0x7f374849f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f374843f450] 1 jobs: ['7600507'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56177de28638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f9911362520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56177de28638] 6: ceph-osd(+0x3f7799) [0x56177de28799] 7: ceph-osd(+0x3e8ab8) [0x56177de19ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56177e375bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x56177e38a8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x56177e3ebe71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56177e3f510b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56177e3f5c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x56177e3fa1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x56177e3fb013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56177e2eca7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56177e0a1142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x56177e07728f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x56177e07ae09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56177dfb3247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56177e1fa609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x56177dfbdc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x56177e4b9293] 23: ceph-osd(+0xa887f4) [0x56177e4b97f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f99113b4b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f9911446a00] 1 jobs: ['7600504'] suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5605aa246638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb8519db520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5605aa246638] 6: ceph-osd(+0x3f7799) [0x5605aa246799] 7: ceph-osd(+0x3e8ab8) [0x5605aa237ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5605aa793bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5605aa7a88fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5605aa809e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5605aa81310b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5605aa813c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5605aa8181f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5605aa819013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5605aa70aa7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5605aa4bf142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5605aa49528f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5605aa498e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5605aa3d1247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5605aa618609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5605aa3dbc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5605aa8d7293] 23: ceph-osd(+0xa887f4) [0x5605aa8d77f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb851a2db43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb851abfa00] 1 jobs: ['7600526'] suites: ['mon_election/classic', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/fio_4K_rand_read}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55c467c273df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7ff5b2e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7ff5b2ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55c467c273df] 6: ceph-osd(+0x3f4540) [0x55c467c27540] 7: ceph-osd(+0x3e6182) [0x55c467c19182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55c46818e450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55c46819849d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55c468208a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55c46820c09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55c46820cc78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55c4681fa9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55c4681fb833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x55c4680fd212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55c4680f1ed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55c467f31ed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x55c467e7b81d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55c467db4dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c467fff889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55c467dd0167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55c4682d097a] 23: ceph-osd(+0xa9df24) [0x55c4682d0f24] 24: /lib64/libc.so.6(+0x9f802) [0x7ff5b2e9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7ff5b2e3f450] 1 jobs: ['7600465'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect}'] Failure: wait_for_clean: failed before timeout expired 4 jobs: ['7600524', '7600466', '7600473', '7600458'] suites intersection: ['debug/mgr', 'rados/mgr/{clusters/{2-node-mgr}'] suites union: ['debug/mgr', 'mgr_ttl_cache/disable', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{bluestore-comp-snappy}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/crash}', 'tasks/failover}', 'tasks/prometheus}', 'tasks/workunits}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e1b83513df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f5f22854db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f5f228a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55e1b83513df] 6: ceph-osd(+0x3f4540) [0x55e1b8351540] 7: ceph-osd(+0x3e6182) [0x55e1b8343182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55e1b88b8450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55e1b88c249d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55e1b8932a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55e1b893609b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55e1b8936c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55e1b89249e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55e1b8925833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55e1b882621d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55e1b85e5492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55e1b85acf47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55e1b85a3715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e1b84dedc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e1b8729889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55e1b84fa167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55e1b89fa97a] 23: ceph-osd(+0xa9df24) [0x55e1b89faf24] 24: /lib64/libc.so.6(+0x9f802) [0x7f5f2289f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f5f2283f450] 1 jobs: ['7600499'] suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5646a37dc638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f48fb62a520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5646a37dc638] 6: ceph-osd(+0x3f7799) [0x5646a37dc799] 7: ceph-osd(+0x3e8ab8) [0x5646a37cdab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5646a3d29bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5646a3d3e8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5646a3d9fe71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5646a3da910b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5646a3da9c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5646a3dae1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5646a3daf013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5646a3ca0a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5646a3a55142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5646a3a2b28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5646a3a2ee09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5646a3967247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5646a3bae609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5646a3971c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5646a3e6d293] 23: ceph-osd(+0xa887f4) [0x5646a3e6d7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f48fb67cb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f48fb70ea00] 1 jobs: ['7600483'] suites: ['debug/mgr', 'mgr_ttl_cache/disable', 'mon_election/classic', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-comp-zlib}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/insights}'] Failure: Failed to reconnect to smithi153 1 jobs: ['7600522'] suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55897305d3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7eff69a54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7eff69aa154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55897305d3df] 6: ceph-osd(+0x3f4540) [0x55897305d540] 7: ceph-osd(+0x3e6182) [0x55897304f182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5589735c4450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5589735ce49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55897363ea61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55897364209b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x558973642c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x5589736309e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x558973631833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55897353221d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5589732f1492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x5589732b8f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x5589732af715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5589731eadc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x558973435889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x558973206167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55897370697a] 23: ceph-osd(+0xa9df24) [0x558973706f24] 24: /lib64/libc.so.6(+0x9f802) [0x7eff69a9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7eff69a3f450] 1 jobs: ['7600451'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/dedup-io-mixed}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5587c5514638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f19819c6520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5587c5514638] 6: ceph-osd(+0x3f7799) [0x5587c5514799] 7: ceph-osd(+0x3e8ab8) [0x5587c5505ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5587c5a61bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5587c5a768fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5587c5ad7e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5587c5ae110b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5587c5ae1c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5587c5ae61f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5587c5ae7013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x5587c59d4bbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x5587c59d66c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x5587c57f2af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x5587c574a17d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5587c569f247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5587c58e6609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5587c56a9c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5587c5ba5293] 23: ceph-osd(+0xa887f4) [0x5587c5ba57f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1981a18b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f1981aaaa00] 1 jobs: ['7600535'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/snaps-few-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x558ba33c73df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f3528654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f35286a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x558ba33c73df] 6: ceph-osd(+0x3f4540) [0x558ba33c7540] 7: ceph-osd(+0x3e6182) [0x558ba33b9182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x558ba392e450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x558ba393849d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x558ba39a8a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x558ba39ac09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x558ba39acc78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x558ba399a9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x558ba399b833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x558ba389c21d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x558ba365b492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x558ba3622f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x558ba3619715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x558ba3554dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x558ba379f889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x558ba3570167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x558ba3a7097a] 23: ceph-osd(+0xa9df24) [0x558ba3a70f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f352869f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f352863f450] 1 jobs: ['7600450'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56075218a3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f4fd8654db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f4fd86a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56075218a3df] 6: ceph-osd(+0x3f4540) [0x56075218a540] 7: ceph-osd(+0x3e6182) [0x56075217c182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5607526f1450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x5607526fb49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56075276ba61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56075276f09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56075276fc78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56075275d9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56075275e833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x560752660212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x560752654ed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x560752494ed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x5607523de81d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x560752317dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x560752562889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x560752333167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56075283397a] 23: ceph-osd(+0xa9df24) [0x560752833f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f4fd869f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f4fd863f450] 1 jobs: ['7600490'] suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/rados_striper}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5648ad64c638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7feeffd5f520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5648ad64c638] 6: ceph-osd(+0x3f7799) [0x5648ad64c799] 7: ceph-osd(+0x3e8ab8) [0x5648ad63dab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x5648adb99bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x5648adbae8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x5648adc0fe71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x5648adc1910b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x5648adc19c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x5648adc1e1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x5648adc1f013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x5648adb10a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5648ad8c5142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5648ad89b28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5648ad89ee09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x5648ad7d7247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5648ada1e609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5648ad7e1c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x5648adcdd293] 23: ceph-osd(+0xa887f4) [0x5648adcdd7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7feeffdb1b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7feeffe43a00] 1 jobs: ['7600541'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/sync', 'workloads/rados_5925}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5564387863df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f042ec54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f042eca154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x5564387863df] 6: ceph-osd(+0x3f4540) [0x556438786540] 7: ceph-osd(+0x3e6182) [0x556438778182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x556438ced450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x556438cf749d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x556438d67a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x556438d6b09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x556438d6bc78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x556438d599e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x556438d5a833] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xc82) [0x556438c5c212] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x556438c50ed7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x556438a90ed6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x80d) [0x5564389da81d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x556438913dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x556438b5e889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55643892f167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x556438e2f97a] 23: ceph-osd(+0xa9df24) [0x556438e2ff24] 24: /lib64/libc.so.6(+0x9f802) [0x7f042ec9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f042ec3f450] 1 jobs: ['7600462'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7f9b980ee0] 1 jobs: ['7600525'] suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{centos_latest}}'] Failure: failed to complete snap trimming before timeout 2 jobs: ['7600517', '7600468'] suites intersection: ['0-start', '1-tasks', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55dade83e638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fccb7544520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55dade83e638] 6: ceph-osd(+0x3f7799) [0x55dade83e799] 7: ceph-osd(+0x3e8ab8) [0x55dade82fab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55daded8bbd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55dadeda08fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55dadee01e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55dadee0b10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55dadee0bc88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55dadee101f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55dadee11013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55daded02a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55dadeab7142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55dadea8d28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55dadea90e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55dade9c9247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55dadec10609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55dade9d3c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55dadeecf293] 23: ceph-osd(+0xa887f4) [0x55dadeecf7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fccb7596b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fccb7628a00] 1 jobs: ['7600496'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56323ab0c3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1cd2254db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1cd22a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x56323ab0c3df] 6: ceph-osd(+0x3f4540) [0x56323ab0c540] 7: ceph-osd(+0x3e6182) [0x56323aafe182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x56323b073450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x56323b07d49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x56323b0eda61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x56323b0f109b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x56323b0f1c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x56323b0df9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x56323b0e0833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x56323afe121d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x56323ada0492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x56323ad67f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x56323ad5e715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x56323ac99dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56323aee4889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x56323acb5167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x56323b1b597a] 23: ceph-osd(+0xa9df24) [0x56323b1b5f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f1cd229f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f1cd223f450] 1 jobs: ['7600477'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55571bbe93df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f3382054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f33820a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55571bbe93df] 6: ceph-osd(+0x3f4540) [0x55571bbe9540] 7: ceph-osd(+0x3e6182) [0x55571bbdb182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55571c150450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55571c15a49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55571c1caa61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55571c1ce09b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55571c1cec78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55571c1bc9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55571c1bd833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55571c0be21d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55571be7d492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55571be44f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55571be3b715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55571bd76dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55571bfc1889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55571bd92167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55571c29297a] 23: ceph-osd(+0xa9df24) [0x55571c292f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f338209f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f338203f450] 1 jobs: ['7600460'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_latest}', 'thrashers/many', 'workloads/rados_mon_workunits}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5625039e7638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f95d18ae520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x5625039e7638] 6: ceph-osd(+0x3f7799) [0x5625039e7799] 7: ceph-osd(+0x3e8ab8) [0x5625039d8ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562503f34bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x562503f498fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x562503faae71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x562503fb410b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x562503fb4c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x562503fb91f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x562503fba013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x562503ea7bbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x562503ea96c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x562503cc5af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x562503c1d17d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x562503b72247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562503db9609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x562503b7cc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x562504078293] 23: ceph-osd(+0xa887f4) [0x5625040787f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f95d1900b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f95d1992a00] 1 jobs: ['7600510'] suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x562f75410638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4fb4c0e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x562f75410638] 6: ceph-osd(+0x3f7799) [0x562f75410799] 7: ceph-osd(+0x3e8ab8) [0x562f75401ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x562f7595dbd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x562f759728fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x562f759d3e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x562f759dd10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x562f759ddc88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x562f759e21f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x562f759e3013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x562f758d4a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x562f75689142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x562f7565f28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x562f75662e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x562f7559b247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562f757e2609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x562f755a5c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x562f75aa1293] 23: ceph-osd(+0xa887f4) [0x562f75aa17f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4fb4c60b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4fb4cf2a00] 1 jobs: ['7600512'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-1}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ac7e6123df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fb7d5454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fb7d54a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55ac7e6123df] 6: ceph-osd(+0x3f4540) [0x55ac7e612540] 7: ceph-osd(+0x3e6182) [0x55ac7e604182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ac7eb79450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55ac7eb8349d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55ac7ebf3a61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ac7ebf709b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ac7ebf7c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55ac7ebe59e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55ac7ebe6833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ac7eae721d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ac7e8a6492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55ac7e86df47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55ac7e864715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ac7e79fdc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ac7e9ea889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55ac7e7bb167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55ac7ecbb97a] 23: ceph-osd(+0xa9df24) [0x55ac7ecbbf24] 24: /lib64/libc.so.6(+0x9f802) [0x7fb7d549f802] 25: /lib64/libc.so.6(+0x3f450) [0x7fb7d543f450] 1 jobs: ['7600491'] suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55630767f638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb80ab6e520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55630767f638] 6: ceph-osd(+0x3f7799) [0x55630767f799] 7: ceph-osd(+0x3e8ab8) [0x556307670ab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x556307bccbd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x556307be18fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x556307c42e71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x556307c4c10b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x556307c4cc88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x556307c511f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x556307c52013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x556307b43a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x5563078f8142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x5563078ce28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x5563078d1e09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55630780a247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x556307a51609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x556307814c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x556307d10293] 23: ceph-osd(+0xa887f4) [0x556307d107f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb80abc0b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb80ac52a00] 1 jobs: ['7600452'] suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-small-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55fd9553b3df] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f6a77c54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f6a77ca154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55fd9553b3df] 6: ceph-osd(+0x3f4540) [0x55fd9553b540] 7: ceph-osd(+0x3e6182) [0x55fd9552d182] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55fd95aa2450] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6d) [0x55fd95aac49d] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13e1) [0x55fd95b1ca61] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55fd95b2009b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55fd95b20c78] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x13c5) [0x55fd95b0e9e5] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2e3) [0x55fd95b0f833] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55fd95a1021d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55fd957cf492] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xe07) [0x55fd95796f47] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3425) [0x55fd9578d715] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55fd956c8dc7] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd95913889] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x55fd956e4167] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0x55fd95be497a] 23: ceph-osd(+0xa9df24) [0x55fd95be4f24] 24: /lib64/libc.so.6(+0x9f802) [0x7f6a77c9f802] 25: /lib64/libc.so.6(+0x3f450) [0x7f6a77c3f450] 1 jobs: ['7600500'] suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/rados_workunit_loadgen_big}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a34a09a638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8551523520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55a34a09a638] 6: ceph-osd(+0x3f7799) [0x55a34a09a799] 7: ceph-osd(+0x3e8ab8) [0x55a34a08bab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55a34a5e7bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55a34a5fc8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55a34a65de71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55a34a66710b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55a34a667c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55a34a66c1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55a34a66d013] 15: (ReplicatedBackend::do_repop(boost::intrusive_ptr)+0xbfc) [0x55a34a55abbc] 16: (ReplicatedBackend::_handle_message(boost::intrusive_ptr)+0x1e7) [0x55a34a55c6c7] 17: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x55a34a378af6] 18: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x7ed) [0x55a34a2d017d] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55a34a225247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a34a46c609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55a34a22fc33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55a34a72b293] 23: ceph-osd(+0xa887f4) [0x55a34a72b7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f8551575b43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f8551607a00] 1 jobs: ['7600537'] suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-small-objects}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ebd8e7c638] ceph version 19.0.0-2027-gf6948dde (f6948ddec49f204a3bb696c60ce2bd00b4465ea1) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2e4cbe9520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55ebd8e7c638] 6: ceph-osd(+0x3f7799) [0x55ebd8e7c799] 7: ceph-osd(+0x3e8ab8) [0x55ebd8e6dab8] 8: (BlueStore::Blob::get_ref(BlueStore::Collection*, unsigned int, unsigned int)+0x120) [0x55ebd93c9bd0] 9: (BlueStore::ExtentMap::set_lextent(boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long, boost::intrusive_ptr, boost::intrusive::list, &BlueStore::OldExtent::old_extent_item> >*)+0x6e) [0x55ebd93de8fe] 10: (BlueStore::_do_alloc_write(BlueStore::TransContext*, boost::intrusive_ptr, boost::intrusive_ptr&, BlueStore::WriteContext*)+0x13f1) [0x55ebd943fe71] 11: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x2ab) [0x55ebd944910b] 12: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x98) [0x55ebd9449c88] 13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x10f1) [0x55ebd944e1f1] 14: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2f3) [0x55ebd944f013] 15: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x8ad) [0x55ebd9340a7d] 16: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x392) [0x55ebd90f5142] 17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xdcf) [0x55ebd90cb28f] 18: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55ebd90cee09] 19: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55ebd9007247] 20: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55ebd924e609] 21: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55ebd9011c33] 22: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55ebd950d293] 23: ceph-osd(+0xa887f4) [0x55ebd950d7f4] 24: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2e4cc3bb43] 25: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2e4cccda00] 1 jobs: ['7600453'] suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/libcephsqlite}']