New Jersey leaders and residents gathered outside St. Luke’s Methodist Church in Long Branch Wednesday to express concerns with the potential rise of white supremacy around the country.

The gathering comes as President Donald Trump faces criticism for his remarks Tuesday after the protests in Charlottesville, Virginia.

A large gathering of self-proclaimed neo-Nazis and white supremacists converged on the town to protest the moving of a statue of Confederate Gen. Robert E. Lee. Counterprotests broke out, and the two sides clashed. The whole incident came to a head when one of the protesters against the removal of the statue allegedly ran down a group of counterprotesters in his car. Heather Heyer, 32, was killed.

President Trump said that he condemned bigotry, but then made comments Tuesday that noted that there were "very fine people on both sides,” including those in the hate group. Some have taken his comments to be a defense of racist beliefs.

Vietnam veteran Avery Grant attended the meeting in Long Branch Tuesday. Grant, who is African-American, said that he is worried about what is to come.

“I’m here to support anything we can to keep us from going backwards,” he said. “I sat at a couple of ‘Whites Only’ counters way back in Louisville, Kentucky…I’m too told to do that now.”

Grant said that the country needs to stop the hate now before it gets out of hand.

The mayor of Long Branch said that he cannot support President Trump.

“To think that the president of the United States can’t come out and condemn Nazis and Klansmen. Given a second chance he makes it worse. He’s not my president,” said Mayor Adam Schneider.

But some people have a different view elsewhere in New Jersey. President Trump gained a lot of votes in Lakewood. He won the town by a margin of 50 percent, the highest in New Jersey. Lakewood resident Geraldine Lucas voted for the president.

“He’s doing the best he can. He is only one person and he gets too much criticism,” she said. “Let him do his job.”

Lucas said that she was upset by the violence in Charlottesville, but that it is not the president’s fault.

“Parents have to control their children and raise them properly,” she said.

Back at the meeting in Long Branch, Rep. Frank Pallone showed photographs of KKK marches in the town decades ago.

“If anyone thinks it can’t happen, it can and it has happened here,” he says.