It is wise to save your gems. Especially in the beginning of the game. It can be attracting to use them to speed things up, but better is to save them for a new builder hut.
The time you win with a extra builders hut is much more then the time you win by spending your precious gems.
With the help of some achievements you can get that 3th and 4th builders hut without buying gems!