"It's a service, so it ramps up over time by it's nature," she told CNBC's Becky Quick. "This will be a play for the long run."
Rometty made her comments from Berkshire Hathaway's annual shareholder meeting in Omaha, Nebraska, where Watson, IBM's predictive data analytics service, was on display.
Berkshire Chairman and CEO Warren Buffett revealed to CNBC in 2011 that he had purchased about 64 million shares of IBM stock for about $10.7 billion. This past weekend, he said he had continued to build that position in the first quarter, purchasing about 3 million additional shares to bring Berkshire's total stake in IBM to roughly 78 or 79 million shares. The company has not yet publicly disclosed the exact figure for its latest purchases in Securities and Exchange Commission filings.
"I bought it because I thought we were getting our money's worth," he said.
Read more Buffett-related stories
Live Blog: Buffett on repurchases and dividends
Buffett: THIS is key to valuing stocks right now
Buffett helps us see the future: 4 blue chip CEOs
Buffett hits back at Clayton Homes lending gripes
Rometty stressed that Watson is central to IBM's growing data and analytics segment, which is a $17 billion business for the company and grew 12 percent last year.
The company has made a big push to use Watson's abilities to crunch data for cancer research and treatment, but it is also targeting companies of all sizes with a program called Watson Analytics, which Rometty said is meant to "democratize" the service for any business user.
Rometty said the Watson program is core to IBM's future, adding that the company made the decision early on not to spin off parts of the service for short-term gain.
"Could you have sold it? Would you have want to sell pieces of it as a product? To me, that would have been a short-term bump in revenue. But that's what it is. It's a long-term play," she said.