They ride on broomsticks – or backwards on goats, according to the German artist Albrecht Dürer. They hold midnight sabbaths. And now they are taking over the Scottish National Gallery of Modern Art in Edinburgh.

One of the major art shows of 2013 in Scotland will be an exhibition at Edinburgh's modern art museum called Witches and Wicked Bodies. In a bold departure for a modernist venue, it will span 500 years of art history, including Dürer. But why are witches modern art all of a sudden?

They are "modern" because they provoke anxiety. The image of the witch was a focus for Freudian terrors long before Freud came along to analyse them. This European myth is a figure of darkness on to which some of the greatest artists have projected some of their most bizarre – and hence "modern" – imaginings.

In his painting Witchcraft, also known as Allegory of Hercules, the 16th-century painter Dosso Dossi depicts a sinister group of people at a nocturnal banquet. Is witchcraft going on here, or is that a romantic title? Dossi was fascinated by the idea of the witch. His painting Circe portrays an ancient Greek enchantress from Homer's Odyssey. Dossi puts her in a contemporary Italian landscape, gives her a tablet inscribed with what look like spells, and makes her a beautiful sorceress.

Sex and magic are an eerie combination in portrayals of witchcraft. The German artist Hans Baldung Grien returned, obsessively, to fantasies of witches having sexual relations with Satan, his demons, and each other. Centuries later, the underground film-maker Kenneth Anger was to weave sex and magic into his ritualistic films, including Invocation of My Demon Brother.

In his 1940 painting the Robing of the Bride, the surrealist Max Ernst imagines a fantastical transformation of woman into bird, under the tutelage of his personal demon Loplop. It is a sensual demonstration of how the occult and the unconscious still flourish in modern art. But while for Ernst magic is fascinating, for artists in the age of Dürer and Dossi it was both alluring and terrifying: flesh and blood people were executed for this fictional crime.

That is why Francisco de Goya's images of witchcraft are the most troubling of all. An advocate of Englightenment and reason in a land of powerful religious traditions, Goya at the start of the 19th century gave dark life to folk beliefs in witchcraft. His paintings and prints make the potency of night terrors real and frightening – but what frightens us is not witchcraft but superstition, madness and human irrationality.

Black magic has been a disconcertingly central theme of western art for centuries. Out of primitive beliefs, the imagination grows strange fruit.