Talks for Broadcom Inc. to buy SAS Institute Inc. have ended after the founders of the closely held software company changed their mind about a sale, people familiar with the matter said.