Most think of Shelter Island on Long Island's East End as an idealistic, picturesque beach town. But the island's majestic Sylvester Manor was once a working plantation staffed by those living in slavery.

Sylvester Manor is a 17th century estate on Shelter Island, about a quarter-mile ferry ride from the Hamptons.

In 1652, Nathaniel Sylvester, who lived in England, bought the property and moved there with his teenage wife. He also brought enslaved African people to supply the sugar cane operations in Barbados.

Many don't know that Long Island was home to the largest population of enslaved people in the North.

"These are all sort of facts of northern slavery that are not really commonly known and are just starting to come to light for the public ... to be taught in schools to be talked about in communities," says DonnaMarie Barnes, Sylvester Manor curator.

The manor is the home where the enslaved people actually lived. Much of it is in its original form: the wallpaper hasn't been touched and the dining room is the same as it was hundreds of years ago.

There is also the "slave staircase" which Barnes says, "Winds to the first-floor bedrooms and then to the attic spaces, where the enslaved and indigenous people were housed in the manor house under conditions that were stifling hot in the summertime and humid to freezing cold in the winter."

Sylvester Manor spans several hundred acres. Many people think it's beautiful and flock there by the hundreds. But it was a very different experience hundreds of years ago.

"For the African people who were brought here from the West Indies, this was a beautiful place, it was a foreign climate and it was a prison," says Barnes. "They were imprisoned on this island against their will and it doesn't really matter how beautiful the sunsets were , there was no escape."

There is also a burial ground believed to be for many of those who were enslaved. There are no tombstones to tell who they were.

It is now the mission of Sylvester Manor to tell the story of slavery that existed on Long Island and in New York. Sylvester Manor is now a nonprofit educational farm with the mission to preserve, cultivate and share the land, buildings and rich history behind it.

"This is the history of not just Shelter Island and not just Sylvestor Manor, but the history of eastern Long Island, of all Long Island, of New York and of the United States," says Barnes. "This is not Black history, this is American History."