A "conversation piece" is an informal group portrait in which the sitters are shown at full length on a scale much smaller than life. The advantage of this format is that it allows the painter to place the sitters in a domestic setting – usually their own home or garden – surrounded by property and possessions to indicate their social status or financial success.

Unlike straightforward portraiture, therefore, the painter of conversation pieces must be able to paint interiors, still lifes, flowers and landscapes. A direct outgrowth of Dutch 17th-century domestic portraiture, the genre became fashionable in England in the 1720s and '30s, when Hogarth, to take one example, painted more than two dozen such group portraits.

There was a revival of interest in the conversation piece in the 1760s, in part because of the new King George III's taste for family portraits by the German painter Johann Zoffany, but also because the English artist Nathaniel Dance made a speciality of painting English gentlemen on the Grand Tour.

It is often said that the conversation piece is associated with portraits of middle-class sitters. That this is only partially true is demonstrated by an enchanting show of paintings from the Royal Collection at the Queen's Gallery in Edinburgh, where many of the very best examples of the genre show royal sitters.

The challenge of the conversation piece is to create a composition in which a group of people (who may or may not be related to each other) look as though they are interacting naturally. That this isn't easy to achieve is clear from one of the first pictures in the exhibition, Hendrick Pot's 1632 double portrait of King Charles I with Queen Henrietta Maria, with the two-year-old Prince of Wales, later Charles II. The young couple are placed at the left and right of the composition, separated by a long table on which are displayed two essential royal props: the crown and sceptre (to tell you who these people are), and the baby (to tell you that the royal line will continue).

At first glance, the picture, for all its charm, doesn't seem to fit the definition of a conversation piece because there isn't much interaction between the adults in it. But look again, and you see that what makes it different from an ordinary royal or dynastic portrait is the element of humour the artist introduces by showing the baby propped up so unsteadily at his mother's side that he looks as though he is about to topple over.

And that's another thing: children are often an important ingredient in a conversation piece. In this show the monarch most closely associated with the genre, George III, had a happy marriage and a large family. Zoffany's pictures give us a fascinating glimpse into the domestic life of the royal family in the first part of the king's long reign. In his 1764 double portrait of the two eldest princes, George, Prince of Wales, and Frederick, later Duke of York, the little boys are shown playing with their pet dog in what is clearly a recently redecorated room in Buckingham House, which the king had acquired only in 1762.

Since there is nothing about the dress or deportment of the boys that indicates their royal status, that information is conveyed by the setting and accessories. With its bright red walls, newly gilded chairs formally arranged against the walls, and wall-to-wall carpeting that looks like something you'd see in a Midland Hotel, there's nothing cosy or lived-in about the room.

On the walls behind them are portraits of their parents, but more interesting to me was the presence on another wall of Van Dyck's double portrait of two children of Charles I – a detail that suggests the Hanoverian kings felt secure enough to see themselves as the legitimate successors of the Stuarts.

On the other hand, one reason that Zoffany may have felt able to show the room as it really looked – including the Van Dyck – is that he was painting a private image that he knew would never be exhibited or engraved.

Infinitely more complex is the composition of Zoffany's amazing group portrait showing the Academicians of the Royal Academy. With dozens of figures arranged in a semi-circle around a life model, what's remarkable is how much variety the artist manages to infuse into the life-like poses and gestures in a composition that at once pays tribute to Raphael's School of Athens and undermines the Dutch tradition of formal group portraiture.

The term "conversation piece" comes from a French word meaning a polite social gathering. But I have a theory (with no evidence to support it) that it refers to a conversation in the English sense of the word – the conversation about the picture that ensues when the viewer remarks on the humorous details or unruly behaviour of the people in it. Unlike formal portraiture, the conversation piece often has a social function: to break the ice.

One of the last examples in this exhibition is Edwin Landseer's Windsor Castle in Modern Times, which shows the young Queen Victoria, Prince Albert and their eldest child Victoria, Princess Royal, at home surrounded by their pets and the dead game that Albert has just brought home from the day's shoot.

Whatever political message the picture was intended to convey – and much has been made of Queen Victoria's wish to replace memories of her dissolute uncles with images of her own happy marriage and contented home life – it would be very hard to enter a room in which this picture hung and not find yourself remarking on the touching and comic details: the dandy dinmont terrier who sits up to beg, the prince's muddy boots, the dead kingfisher that the sturdy little princess examines with such curiosity.

The quality and variety of the works on view in Edinburgh is so extraordinary that I'm not sure you'd learn any more about the genre in a larger show. If you go, I highly recommend the audio guide by the Surveyor of the Queen's Pictures, Desmond Shawe-Taylor, who also wrote the first-class catalogue.