Opponents blame Ms Rousseff for a severe recession and a corruption scandal also engulfing her predecessor, Luiz Inacio Lula da Silva.
She has said she has no intention of resigning but her opponents hope a large turnout on the streets will lead to her impeachment.
She faces charges over masking the budget deficit, which she denies.
Crowds in Sao Paulo at the peak of the march reached 1.4 million, the city’s state security office said.
Smaller marches took place in many cities across the coutnry.
The demonstrations were peaceful, with tens of thousands wearing the national colours of yellow and green and holding banners that read “Dilma out” and “Stop with corruption”.
“I support her impeachment and new elections because the presidential vote in 2014 was financed with dirty money from corruption,” said protester Alexandre Cortes, a 39-year-old engineer in Sao Paulo.
Correspondents have described the crisis as the worst in Brazil for many years.
The BBC’s Wyre Davies says Ms Rousseff, leader of the governing Workers’ Party has come under increasing pressure in recent months over a huge corruption investigation, centred on the state-controlled oil company Petrobras.
So far, the Petrobras inquiry has implicated some of the wealthiest people in Brazil and politicians from all parties.
Prosecutors say contractors skimmed billions of dollars from the company to illegally finance politicians including those from the Workers’ Party and its allies – allegations the party denies.
Ms Rousseff was head of the Petrobras board at the time of many of the infractions and opponents have accused her of involvement.
She has denied any knowledge of the corruption scheme and is not being investigated.
The demonstrators also directed their anger towards Ms Rousseff’s political mentor and presidential predecessor, Luiz Inacio Lula da Silva.
In Sao Paulo, large inflatable dolls of President Rousseff wearing a sash saying “impeachment” and Lula in prison clothes were paraded through the streets.
Dilma Rousseff under pressure – Daniel Gallas, BBC News, Brazil
The size of Sunday’s protests matters, because it could help define the shape of things to come for President Dilma Rousseff.
Her future in office depends on her support from Congress, where she is facing an impeachment process.
If she loses backing from MPs and Senators, she could be out of her job before her term ends, on January 1st 2019.
Support in Congress is very volatile – some of the same politicians and parties who are in the ruling coalition today were previously in governments lead by the opposition.
Sunday’s protests could serve as a good excuse for some of those parties to turn against her at a moment she is very unpopular.
Pressure on Lula
Last week Sao Paulo State prosecutors asked a judge to place Lula in “preventive custody” after filing charges accusing him of failing to declare ownership of a luxury sea-front penthouse in the seaside resort of Guaruja.
Lula, 70, denies any wrongdoing and says the claims are politically motivated. He says he never owned the flat.
Protesters also expressed frustration that Brazil – a rising economic star on the world stage only a few years ago had wasted a historic opportunity with many holding the PT government responsible.
Brazil recently lost its investment-grade status. President Rousseff has accused the opposition of paralysing the Congress with bickering making it impossible to pass the financial measures needed to jump-start the economy