The big plus for gold is that it doesn't corrode or tarnish. However, its higher cost is a disadvantage, and it's less conductive than silver.
Another problem with gold contacts is the potential for oxidation from organic compounds, simple versions of which are present in the atmosphere. In vapor form, they can be absorbed by the gold contact surface. Sliding or fretting between contacts causes a chemical reaction between the gold, which acts as a catalyst, and organic compounds. This reaction produces long-chain compounds or polymers.
One way to get around these drawbacks is to place a thin gold layer over a copper contact. Doing so demands barrier plating. Due to the cost of gold, the plating is usually thin, on the order of 1 to 2 microns. Without a barrier between the two materials, copper can migrate to the gold surface, where the exposed copper begins to oxidize forming a copper-oxide layer that acts as an insulator. The barrier layer is usually nickel.
Silver contacts, on the other hand, need no such barrier because the silver plating is much thicker, 2 to 5 microns. A barrier also isn't necessary in gold over silver contacts because the silver is thick enough to act as a barrier.
Some applications such as high-reliability edge-connectors require 30 microns of gold plating by specification. The thick plating is needed because of the high-force, high-wear nature of insertions and disconnections. However, a 30-micron gold plating isn't reasonable for switch contacts. First, a plating of this thickness would eliminate the spring characteristics of the contacts, and there just isn't enough room for such a thick plating. Moreover, a switch with 30 microns of gold on the contacts would be more expensive.
Soft mechanical wear is another disadvantage of using gold. It's much softer than silver or silver alloys and gold wears much easier by sliding friction.