Here's a nice section from Ben Macintyre's “A Spy Among Friends: Kim Philby and the Great Betrayal”:

Harvey's Ladies' and Gentlemen's Oyster Saloon started serving steamed oysters, broiled lobster, and crab imperial in 1820 and had continued to do so, in colossal quantities, ever since. In 1863, notwithstanding the Civil War, Harvey’s diners were getting through five hundred wagonloads of oysters a week. Every president since Ulysses S. Grant had dined there, and the restaurant enjoyed an unrivaled reputation as the place to be seen for people of power and influence. The black waiters in pressed white uniforms were discreet, the martinis potent, the napkins stiff as cardboard, and the tables spaced far enough apart to ensure privacy for the most secret conversations. Ladies entered by a separate entrance and were not permitted in the main dining room. Most evenings, FBI director J. Edgar Hoover could be seen at his corner table, eating with Clyde Tolson, his deputy and possibly his lover. Hoover was said to be addicted to Harvey’s oysters; he never paid for his meals.

Interesting: ladies get first billing but separate entrance and no main dining room.

Here's a little history of Harvey's, which “no longer exists in the city.” (That second photo of the exterior looks like something out of a Wes Anderson movie.) This piece suggests Harvey's does exist; it simply “relocated to the suburbs,” but it doesn't say which suburbs.