According to the United States Census Bureau, the town has a total area of 20.2 km² (7.8 mi²); none of it is covered by water.
As of the census of 2000, there are 4,584 people, 1,627 households, and 1,277 families residing in the town. The population density is 226.6/km² (586.7/mi²). There are 1,670 housing units at an average density of 82.6/km² (213.7/mi²). The racial makeup of the town is 91.84% White, 0.17% African American, 1.22% Native American, 1.29% Asian, 0.02% Pacific Islander, 3.08% from other races, and 2.38% from two or more races. 9.21% of the population are Hispanic or Latino of any race.
There are 1,627 households out of which 36.3% have children under the age of 18 living with them, 68.0% are married couples living together, 7.4% have a female householder with no husband present, and 21.5% are non-families. 15.5% of all households are made up of individuals and 7.0% have someone living alone who is 65 years of age or older. The average household size is 2.78 and the average family size is 3.09.
In the town the population is spread out with 26.2% under the age of 18, 4.7% from 18 to 24, 25.5% from 25 to 44, 28.8% from 45 to 64, and 14.9% who are 65 years of age or older. The median age is 42 years. For every 100 females there are 90.7 males. For every 100 females age 18 and over, there are 88.4 males.
The median income for a household in the town is $80,284, and the median income for a family is $84,467. Males have a median income of $56,286 versus $45,688 for females. The per capita income for the town is $33,811. 5.5% of the population and 3.0% of families are below the poverty line. Out of the total people living in poverty, 7.1% are under the age of 18 and 2.4% are 65 or older.
Michael Jackson's ranch Neverland is located at the Figueroa Mountain Road in Los Olivos, in the Santa Ynez Valley.