Spain's socialists say they would vote 'no' to another PP government
The ruling conservative Popular party came first with 123 seats in Sunday's election but fell far short of a 176 majority needed to govern alone.
Mr Rajoy, seeking a second term, said he will "try to form a stable government" but has no immediately obvious partner with which to do that.
Spanish political parties face what are likely to be tough negotiations to form a government after the country voted in a fragmented Parliament that bade farewell to the dominating two-party scene of recent decades.
The PSOE followed with 90 seats and 22 percent of the vote, then Podemos with 69 seats and almost 21 percent, and finally the centrists Ciudadanos with 40 seats or nearly 14 percent.
"Spain is not going to be the same anymore", said jubilant Podemos leader Pablo Iglesias.
Spain's benchmark Ibex 35 index was down 2 percent in trading Monday, suggesting investor jitters following the result.
Created less than two years ago, the anti-establishment party quickly went on to win five seats in European elections, and has now become the parliament's third political force.
Rajoy's best chance would seem to be with Ciudadanos - due to their shared center-right, business-friendly politics - but the party has ruled out supporting Rajoy as prime minister, chiefly because of corruption cases in his party.
But several senior Socialist officials have already said the party should be in opposition and would reject any government led by Rajoy or the PP. The nominee must garner a majority of deputies' votes in Parliament in a first round to take office, or the most votes in the second round. If the candidate is not immediately successful, Parliament has two months to elect a prime minister or call a new election.