The rocks in Varvsberget are the same as in the other table mountains (basement, sandstone, alum shale, limestone, clay shale, and dolerite), but the mountain is part of the same limestone plateau as Gerumsberget and Plantaberget. The actual visible mountain thus consists only of clay shale and dolerite. Varvsberget is connected to Gerumsberget to the south.
The name is derived from the ancient village of Varv to the east of the mountain, but a number of other names have been used throughout the centuries: Fårdalsberget, Granberget, and Kungslebaberget. There are traces of several ancient fortifications here, such as the castle Lenaborg on the northern slope, where Birger Jarl resided in the 13th century.
