Some governments try to control the movements of their own and other citizens. For example, many Muslim countries will not allow entry to people with an Israeli visa in their passport, and it is illegal for US citizens to visit Cuba. (To help foreigners get around these restrictions, Israel and Cuba do not require visitors to have their passports stamped upon entry, making it difficult for Muslim nations or the US to tell if their citizens broke the law by visiting those countries.)
Sometimes countries have a reciprocal agreement that a visa is not needed under certain conditions, e.g. that the visit is for tourism and not for longer than three months. No visa is required for travelling between European Union countries, where EU citizens have full freedom of movement and work.
A few countries have agreements allowing for cross-border travel without passports (but with identification). These include the EU countries of the Schengen Group, the United States and Canada, and Great Britain and Eire.
In most European countries, the passport belongs to the citizen, who has a right to travel to any country that will accept him or her. The only exception is that passports may have to be temporarily surrendered by people on bail and awaiting trial if there is a risk that they might abscond. The situation is different in the US, where only about a quarter of citizens have a passport anyway. The passport is the property of the state, and can be withdrawn arbitrarily. Prominent people with left-wing views, such as Paul Robeson, have been prevented from traveling abroad by this method.