Playing with the mass and friction is a good first step. At some point, though, math is going to be your culprit. If you've taken a course in calculus, then you're probably aware of the issues in finite difference approximations of the derivatives used in a physics model.
If you've never taken a course in calculus, the short version is this: The fewer frames per second you run the physics model at, the less stable and poorer the model is. You can raise Stencyl's default 100 ticks per second, although it comes at the cost of speed. There is another parameter, the arguments of world.step, that you can adjust. You'll have to go into Engine.hx to adjust them. Raising the parameters improves stability, but once again comes at the cost of speed.
Fortunately, there are other ways you can "help" the model achieve stabler towers that might even lower CPU requirements. The Joints Extension is a real help here, since you can use it to simulate mortar/glue/other binding materials used in real world structures. I found that a combination of using weld joints and a "foundation" (that can't move) greatly improves how large a structure you can build. You can even break the joint when an object is hit if you still want the structures to collapse.
Here's something I wrote up a few years ago with some more details on using the Joints Extension for towers.
http://community.stencyl.com/index.php/topic,44376.30.html
