Compartmentalizing a composition with discreet areas recessed into the surface where a motif is repeated, is a strategy that was in use more than 1500 hundred years ago but has become more common in contemporary art. This kind of strategy requires a certain way of looking and a particular kind of thinking. Most paintings are constructed around a particular motif or subject such as a flower or landscape etc which is constituted as a unified "whole." Some paintings are created as a field where the whole surface is unified by a dominant color or pattern or both without any distinctly recognizable motif. To divide the surface up into discrete areas in order to draw attention to a particular symbol or shape by repeating it and then augmenting that by use of its adjoining areas, is another kind of approach used often in abstraction. Mise en abyme is a way then, of making an intervention in the composition and emphasizing its critical components to the viewer.
Another great example is "Las Meninas" painted by Diego Velazquez in 1656 where he very skillfully divides up the picture into at least four separate areas that echo the outer shape of the canvas. The rear of the large canvas can be seen with the artist standing in front of it and looking forward directly at the viewer. Then there is the strong shape of the pictures at the back of the room hanging on the wall. On the same wall it is just possible to make out what looks like a portrait or maybe a mirror reflection of the King and Queen. Finally there is the figure illuminated in the doorway at the back of the room. This is a wonderful early modern study of squares or rectangles within a square/rectangle and remains one of the cleverest paintings ever devised playing with perception and a tremendous awareness about and confidence in what the artist is doing. There are many interpretations of this picture but i'm primarily interested in the use of the square echo throughout the painting.
One of the greatest Modern artists to exploit mise en abyme in his paintings is Rene Magritte. His work "Representation" from 1937 is really simple but a very clever painting. The nude torso echoes the shape of the frame in such a tight relationship that they become one and the same to complete the illusion. The female torso so completely fills the frame as to seem to come bulging out into our space as if through a portal in time. There is no room for the eye to get past or beyond the torso stuffed into the frame; no rest from the advancing image as it surges into view. There is a certain strange mix of menace and anticipation when viewing this painting. Here the idea of the "insert" or "motif" is so central, so immediate, it totally envelopes the viewer in suspense. This effect is only possible because of the frame sympathizing totally with the human form.
As intriguing as "mirror art" may be, i'm more interested in the "recessed" image as "contrast" in the composition, not endless multiple reflections of an original subject acting as critique for "presence/absence" or location of the "subject." In the gallery below i've provided three images that demonstrate the use of shape(s) within the art object that reflect or echo the outer edge of it. The first, 'Untitled" by Karl Holmquist, a Swedish artist, uses the simplicity of a checkerboard pattern to emphasize the diamond shape of the object. The black and beige diamonds conspire through repetition to not only create an illusion of depth but draw our attention to the very "edginess" of the edge.
The middle image is "Black Square" by Kasimir Malevich, possibly the most well known and important abstract work of art. The central black square was meant to express "pure" feeling surrounded by the white "void." The black form echoing the white surround echoing the edge or border of the work is about as close as you can get to the "sublime" theoretically if not experientially! For the time in history that this was created it was absolutely "new" and totally baffled viewers who had never witnessed anything like it before. It is the negation of everything and yet the ratification of everything in the same visual "instant."
The last image is an encaustic work of mine "Untitled 160907" from 2007 that demonstrates again the echoing of the outer perimeter by the use of interior regression of the same shape into the appearance of perceptual infinity. Each square or outline of a square serves to emphasize or heighten the "squareness" of the art object. Repetition becomes important in the attempt to create visual profundity out of simplicity.
Below is an interesting essay on the use and meaning of mise en abyme in Medieval art by Stuart Whatling of the Courtald Institute of Art.
Below is a YouTube video spoof of dynamic mise en abyme.