Work in progress, likely to update this over time.
All writers, fundamentally, are custodians of attention. Shakespeare is perhaps the most able manager of audience attention that has ever lived, and my personal interest in him stems from wondering what we can learn from him as a writer. Hamlet is an interesting case study because his project is to revive an old genre, which is very hard to do at a mature point in a genre’s history, and yet he does it so spectacularly that Hamlet is commonly regarded not just as Shakespeare's first undisputed masterpiece but also as a high-water mark of all world literature.
As a manager of attention, Shakespeare is doing everything that he can possibly do to in Hamlet keep your brain away from things that are sand traps for writers. He is turning your attention into knots and making balloon animals out of it so that you are not focused on the things that are often deadly for a story: cliche, unwanted violations of the suspension of disbelief, or other drains on narrative energy. He is incredibly, incredibly good at this. You do not generally spend your time while watching Hamlet thinking about how tired the use of a ghost is, or god this is yet another delay before revenge is fulfilled, or the overuse of an antic disposition, or other ways in which you feel that you have seen this story before. In fact, one of the quiet miracles of the play is that the things that you think about generally lag behind where the play is at. Hamlet is in fact so good at managing its audience’s attention that it always feels like it is one step ahead of you while you experience it. Perhaps the best analogy for this is that the play is to its audience as Hamlet is to Polonius.
It’s worth discussing how Shakespeare accomplishes this. The main aesthetic principle that distinguishes him as a writer is oppositional force. In Lit and Writing 101 classes you’re taught about plot conflicts as drivers of story — man vs. nature, man vs. self, man vs. society, etc. — but Shakespeare’s work answers the question: what if you could bring that same potential for energy into every element of a narrative: its characters, its ideas, its language, its scenic structures, etc. Shakespeare constructs the elements of his stories by fusing opposite ideas in extremely subtle ways, and these all contribute to productive, invisible tension throughout his work. This is the engine of his plays. An example of this in his characterizations is how, as Norman Rabkin notes, Brutus in Julius Caesar can be seen both as the best and worst of men simultaneously. An example of this in what might be called the ideational space of his plays is the scene where Hamlet meets the ghost of his father, which can satisfy otherwise mutually exclusive Catholic and Protestant readings at the same time. One of Lear’s final lines is “O, you are men of stones,” which glosses simultaneously as “you are men of worth and strength” and “you are worthless things.” Many other examples abound, but these each stand in for a larger galaxy of phenomena where tension between opposites translates into real narrative frisson.
There is another related phenomenon operating as well, which is that the play, again like its title character, is full of deliberate slights of hand, misdirection, and general mischief. Pretty much every scene contains an example, but one of the largest is when Hamlet delivers the “to be or not to be” speech. Pondering suicide makes sense here characterologically, but not, as Stephen Booth notes, in terms of the actual plot of the play, where an audience has been conditioned to assume that the commandment of the ghost — Hamlet’s main plotline — is more important than the secondary intrigue raised in the subplot about Hamlet's possible interest in Ophelia. When Hamlet then tells Ophelia that she should not have believed him, he speaks for the play. Like so many other places in Hamlet, this scene both frustrates and satisfies simultaneously, and that oppositional tension is also something that is invisibly exciting.
Shakespeare also realizes something truly astounding in Hamlet, one of the most brilliant things that a writer has ever figured out: that those places where genre cliches ordinarily enter into the consciousness of your audience as violations of the suspension of disbelief — see the prior list of cliches in revenge stories — are in fact potential ways to deepen your narrative and to generate interest and excitement. By introducing ideas into his stories that conflict with genre conventions — but doing so in a way that an audience doesn’t actively observe — he is generating the kind of invisible oppositional energy that gives frisson to a plot conflict. He is taking something that could be a drain on an audience’s attention and converting it into a source of excitement instead. The moment in the “to be or not to be" speech when Hamlet refers to death as “the undiscovered country from whose bourne / no traveller returns” is enormously complex, and is deepened not just by its affect of existential gravity, but also, as many have observed, by the fact that it is logically incompatible with a plotline wherein a ghost returns from the grave.