Addressing a gathering at Diu in the Union Territory of Daman and Diu, Singh said, "all our neighbours are good, except one".
"We want to maintain good relations with all our neighbours. Our sages said the whole world is one family....We accept our neighbours as friends. All neighbours are good except one which does not agree. You know which one," he said, alluding to Pakistan.
"If (this neighbour is) not agreeing with us today, it will have to do so tomorrow. The entire international fraternity is putting pressure on it to leave the path of terrorism. It is helping terrorists sneak into India, conspiring to break up India despite being a neighbour. It should look at its own weaknesses," Singh said.
He lauded the armed forces for giving a 'befitting reply' to terrorists.
"We told the Border Security Force in September 2014 itself that we should maintain good relations with neighbours, and also with Pakistan because it is our neighbour.
I told the BSF and armed forces not to fire the first bullet, but when the other side fires, don't count how many bullets you fire (in retaliation)," he said to a cheering crowd.
Singh was here to inaugurate various projects. Diu will soon become a Smart City with the Centre allocating Rs 1,400 crore for its development, he said.
India cannot become a developed country until all states develop equally, "irrespective of whether they are big or small, or which party is ruling them", Singh said.
"This is why we have a federal structure, and our government is moving ahead with this idea," he said.
He also mentioned the Fugitive Economic Offenders Bill, the ordinance of which has been approved by the Cabinet.
"The opposition blames us, saying `they (fraudsters) fled abroad, Nirav (Modi), Choksi.... I say this happened even before (the BJP came to power). The question is what the government is doing now. Our government has prepared a bill, Fugitive Economic Offenders Bill.
"If an absconder, however big he is, flees India, his property was not seized earlier. Now with this Bill, his entire property will be seized," the Union home minister said.