Support for Ms Rousseff has fallen to single-digit figures in recent polls.
Many voters have accused her of failing to stamp out corruption and blame her for the economy's worst slump in 25 years.
Marchers took over Copacabana beach in Rio and also demonstrated outside congress in the capital Brasilia.
Many wore the yellow shirts of the Brazilian football team, and sang the national anthem, carrying banners saying "Dilma Out".
Police said about 137,000 people took part, but tens of the thousands of others were also involved in a demonstration in Sao Paulo.
The national day of action is the third major protest against Ms Rousseff and her left-wing Workers' Part this year. Hundreds of thousands took part in demonstrations in March and April.
"We want things to change and if the people don't go in the street that's impossible," said retired engineer Elino Alves de Moraes, who joined the march in Brasilia.
Ms Rousseff is less than a year into her second term as president.
People took to the streets to demand the impeachment of Dilma Rousseff, even though the perceived mood across the country in recent days has been one of reconciliation.
There's been an acceptance that to remove a democratically elected president might do more harm than good to a country which only emerged from military dictatorship three decades ago.
Overwhelmingly white and middle class, the protesters in Sao Paulo were having none of that argument.
In generally jovial mood, they condemned the ruling Workers Party for its role in the unprecedented corruption scandal surrounding Petrobras.
Ms Rousseff may have more to worry about further down the line, if Brazil's economy continues to decline after a decade of growth.
If inflation creeps above 10% and the economy goes into recession, that could alienate not only the privileged protesters of Sao Paulo, but her own working class base.
Anti-government protesters say Ms Rouseff must have known about a corruption scandal in the state oil firm, Petrobras, as alleged bribery took place when she was head of the company.
She was exonerated in an investigation by the attorney general and denies involvement.
However, several senior members of her government have been implicated.
Government austerity measures are also hugely unpopular with the electorate, correspondents say, as are rising unemployment and inflation rates.
A survey by the Brazilian company Datafolha showed support for Ms Rousseff's impeachment was strongest in the poorest areas, which backed her in the last election.
There have also been demonstrations in recent months showing support for the embattled leader, with many claiming calls for her impeachment amount to a coup attempt.