Freeform edges is an Advanced Setting available since OpenTTD 0.7.0 or r15190. If enabled, the classic limitation that the map edges have to be water is lifted. This also means you can now terraform at the map borders to your liking. To cope with this new feature, you can now select the border type (Water, or Freeform) for each border individually when generating a new map using TerraGenesis. Or select random to let the game choose for you.