Kevin Costner rattled "Yellowstone" fans last week when he confirmed that he's leaving Montana rancher John Dutton behind after five seasons. Costner recently sat down with "CBS Mornings" co-anchor Gayle King to discuss his decision to leave the hit show after months of speculation.

The Oscar-winning actor-director told King that "Horizon: An American Saga" – a four-part film series about the American West – always came second to "Yellowstone." Still, it was difficult to juggle the demanding schedules of both projects.

"I wanted to work more than once a year and it was important that I made room for 'Yellowstone' and made room for 'Horizon,' but people ran through deadlines. They were busy; they had a lot to do," explained Costner in an interview on "CBS Mornings" that aired Thursday, adding that he'd love to go back to "Yellowstone" if he feels like "the story is going the right way."

Costner said he put the "Horizon" film franchise "into the holes," shooting the first chapter in just 52 days. In comparison, Costner said, his award-winning 1990 film "Dances with Wolves" – his directorial debut – took around 106 days.

"'Yellowstone' was my first priority," Costner reiterated. "But at a certain point when you make commitments you just have to live with them."   

In an interview with "CBS Sunday Morning" that aired earlier this month, Costner said he "did everything that [he] was contracted to do with 'Yellowstone.'" 

Rumors swirled that an alleged feud between him and "Yellowstone" co-creator Taylor Sheridan also contributed to his decision to walk away from the role. But Costner told King that he has nothing but love for the show and reflected on his early days working with Sheridan.

"Before anybody, it was [Taylor Sheridan] and myself. The truth never changes; that love was very strong. The idea of going back… I would do that in a second if I felt it was something I could do or wanted to do. I would make it fit. Nothing changes there. There's no games with me. There's just the way you choose to live your life," he told King.

Costner said he was hoping to continue after committing to five seasons as John Dutton, but there are a lot of different circumstances that arose that made it difficult. 

"I'm pretty even the way I operate," he said.

But Costner's not necessarily ready to officially close the chapter on John Dutton just yet.

"I've never missed any obligations in my entire career. So the idea [of returning] is wide open for me. It just happens to be, can I fall in love with the writing? Can I fall in love with the part to go forward?" Costner said.

Fans have expressed online that they can't imagine "Yellowstone" without the iconic raspy-voiced patriarch, crediting Costner for playing a huge part in the show's success.

In an Instagram post last week, Costner said he realized he couldn't continue filming for "Yellowstone" after spending the past year and a half directing and starring in "Horizon." He said the character will always have a special place in his heart.

"It was something that really changed me. I loved it, and I know you loved it. I just wanted to let you know that I won't be returning. I love the relationship we've been able to develop, and I'll see you at the movies," he said in a video that has millions of views. 

Chapter 1 of "Horizon: An American Saga" is hitting theaters on Friday, June 28. Chapter 2 is set to be released in August and the others will follow.

Paramount Network is a subsidiary of Paramount Global, CBS News' parent company.