Washington (CNN) -- Even as he charges into his re-election battle, President Barack Obama has a festering weakness on a flank that, by all accounts, ought to be rock solid: the one held by African-American Democrats. Their support for his programs, belief in his leadership and enthusiasm for another term is softening just as he needs it most.

In the past few weeks, the president has been chastised by the Congressional Black Caucus for avoiding troubled inner-city districts, peppered with tough questions on BET, and suffered plummeting poll numbers among black voters.

That last item is critical. During the 2008 election, their turnout was massive and their support almost unanimous. Now, however, a Washington Post/ABC News poll finds that black voters with a "strongly favorable" view of Obama dropped from 86% to 58% in just five months.

"Patience is running out," says Reginald Daniel, a professor of sociology at the University of California Santa Barbara. Daniel has written extensively about race and politics. "People are disappointed because I think their expectations were way too high to begin with."

Many political analysts sum up those "expectations run amok" this way: White voters heralded the election of a black president as a sign that long-standing racial gaps were closing, and they expected Obama to play a post-racial role; a president for everyone, who just happened to be black.

Meanwhile, many African-American voters just as clearly saw his election as a great leap forward and hoped the presence of a black president in the Oval Office would bring a new level of understanding, acknowledgment, and relief for their community's problems.

"Finding a middle ground is almost an impossibility," Daniels says. "It's just the worst position to be in."

The driving problem, however, is neither black nor white -- it is green. Joblessness, bad for everyone, is much worse in African-American communities, where unemployment is pushing 17%, the worst since the 1980s. That is what triggered the revolt in the CBC and those stinging questions during that TV interview.

The president is pushing back, pointing out that his initiatives, such as health care reform and the recent jobs bill, especially help lower-income families, many of whom are black. On BET, he quickly dismissed talk about a dearth of specific programs for minorities.

"What people are saying all across the country is we are hurting and we've been hurting for a long time," the president said. "The question is: How can we make sure the economy is working for every single person?"

He also insisted that even if black leaders are grousing, it's not really about him or his policies. "There's always going to be somebody who is critical of the president of the United States." And at a CBC fundraising dinner he raised eyebrows by telling members to quit complaining and start "marching" with him for change.

But if the reaction of Rep. Maxine Waters, D-California, is any gauge, he'll have to do a lot more selling. She summed up his comments in a word: "Curious."

To be sure, most African-Americans still support Obama. Political analysts do not expect a massive shift of their votes to the Republicans; and even the president's most adamant black critics often follow their barbs with a quick salve, saying African-American voters ultimately will not oppose him.

Still, the danger for the White House is not that black citizens will vote against Obama, but that they won't vote at all.

A lackluster turnout in just a few key states could tip the electoral balance against him. What's more, every moment he spends making sure black Democrats come to the polls increases his risk of alienating white voters, and is time lost winning over independents, whom he also must have.

Politically, it is hard for a candidate to tiptoe through such a black-and-white minefield. For a president, it is even tougher.