According to the United States Census Bureau, the town has a total area of 96.0 km² (37.1 mi²). 24.9 km² (9.6 mi²) of it is land and 71.1 km² (27.4 mi²) of it is water. The total area is 74.02% water.
As of the census of 2000, there are 4,317 people, 2,068 households, and 1,244 families residing in the town. The population density is 173.1/km² (448.2/mi²). There are 3,970 housing units at an average density of 159.2/km² (412.2/mi²). The racial makeup of the town is 96.92% White, 0.95% African American, 0.28% Native American, 0.28% Asian, 0.00% Pacific Islander, 0.37% from other races, and 1.20% from two or more races. 0.53% of the population are Hispanic or Latino of any race.
There are 2,068 households out of which 21.5% have children under the age of 18 living with them, 48.8% are married couples living together, 8.3% have a female householder with no husband present, and 39.8% are non-families. 33.8% of all households are made up of individuals and 13.9% have someone living alone who is 65 years of age or older. The average household size is 2.08 and the average family size is 2.63.
In the town the population is spread out with 18.0% under the age of 18, 5.5% from 18 to 24, 24.7% from 25 to 44, 30.9% from 45 to 64, and 20.9% who are 65 years of age or older. The median age is 46 years. For every 100 females there are 94.4 males. For every 100 females age 18 and over, there are 93.8 males.
The median income for a household in the town is $28,514, and the median income for a family is $33,425. Males have a median income of $27,075 versus $20,859 for females. The per capita income for the town is $20,367. 12.7% of the population and 9.7% of families are below the poverty line. Out of the total people living in poverty, 22.9% are under the age of 18 and 9.4% are 65 or older.