Named for the colonial proprietor of Maine, Sir Ferdinando Gorges, it was constructed to support existing forts, including Fort Preble in South Portland and Fort Scammel built on nearby House Island in 1808. Congress, however, did not fund construction of Fort Gorges until 1857. The walls of the fort were begun the next year, and when the American Civil War broke out in 1861, work quickly advanced.

The fort was designed by Colonel Reuben Staples Smart. The chief architect in charge of construction was Thomas Lincoln Casey, who later became Chief of Engineers. It is similar in size and construction to Fort Sumter, but is built of granite instead of brick.

In 1865, when the war ended, the fort was finally completed. Eventually, modern explosives made the fort obsolete by the time it was ready for service. A modernization plan was begun in 1869, but funding was cut off in 1876, with the third level of the fort still unfinished. During the modernization project, sod-covered sand was added to the top level of the fort to protect gun encasements and powder magazines from attacks.

The fort was last used by the military during World War II, when it was used to store submarine mines. It was acquired by the city of Portland in 1960 and placed on the National Register of Historic Places.