The colony center is the heart of the colony. It is where the colonists live and units and buildings are produced. The Colony Center has a number of inherent economic upgrades to get your economy started independent of annexed sectors. When the colony center is taken over by an enemy all annexed sectors are lost with it.
Each colony sector can have a limited number of sectors annexed to the colony as provinces. To be annexed the sector must be located within two sectors of the colony center as well as being adjacent to another sector of that same colony. The number of provinces a colony can have depends on the number of colonists:
Annexing a sector is done by moving a unit onto the sector flag and choosing the “Annex Sector as Province to Colony” button.