Leader Sir Ed Davey celebrated by knocking over a wall of blue bricks, claiming his party was now the “main threat” to the Tories in many areas.
The party’s candidate Sarah Green won by 8,028 votes from the Tories, with the Green Party in third place.
Labour had the worst by-election result in the party’s history, with 622 votes.
The Conservatives had comfortably held the Chesham and Amersham constituency in southern England since its creation in 1974.
The result is a major boost for the Liberal Democrats after their poor performance at the 2019 general election and brings their total number of MPs at Westminster to 12.
Prime Minister Boris Johnson described the result for his party as “disappointing”.
Appearing alongside Ms Green at a victory rally in the Buckinghamshire constituency, Sir Ed said: “This is a great result, a huge swing to us. There are many Conservatives across the country who are now worried.
“People have been talking about a red wall, well after Chesham and Amersham and Sarah Green’s victory they’ll be talking about a blue wall, and how the Lib Dems are the main threat to the Conservatives.
“People here felt ignored and taken for granted, and we listened. Sarah Green is going to take these concerns to Parliament and hold this appalling Tory government to account.”
In her victory speech in the early hours of Friday, Ms Green said: “Tonight the voice of Chesham and Amersham is unmistakable. Together we have said, ‘enough is enough, we will be heard and this government will listen’.”