Asked about his new album title, Kendrick Lamar demurred, “That will be taught in college courses someday”. You could call him arrogant if you like, but not before you call him right. To walk through a campus town is to see a tidal wave of well-to-do white people in Notorious B.I.G. t-shirts, to know that those institutions place the words of Tupac up there with the established cannon in their curriculums, to know that no matter how many knuckle-draggers might kick off about a festival headliner, that truthfully they lost that war a long time ago. Kendrick probably isn’t going to have to wait too long anyway: Georgia Regents University already has his last album on the syllabus. You can probably spin a better undergraduate piece out of Kendrick’s title than Harper Lee’s anyway.

Which is a roundabout way of confronting the elephant in the room: I am not his target audience here. I know this, and you can probably take one look at the name and guess. To Pimp A Butterfly is an album that examines – in extreme, sometimes excruciating detail – the African-American experience. It is an album that considers what it means to have a black President but to watch the brutality of Ferguson unfold, that looks back at a history of oppression and asks if anything has ever really changed, that asks what it means to achieve your dreams and attain stardom but know that there is still part of you that will always be outside, part of you that still owes something to where you came from.

So what right do I, a white man in England, have to go digging through this? To me, the answer is simple. Even as I must acknowledge my distance (or, more crucially, my complicity), this is an album that demands a response. It asks questions of every listener, that makes great demands of everyone from Kendrick downwards. Kendrick knows he is not just speaking to the community back home in Compton, but delivering his messages to the world. This is an album that refuses to be ignored, that understands the realpolitik of progress and shouts from the rooftops, barriers be damned. When the question is asked, do you answer it or just ignore it? Or, as Kendrick puts it: “when shit hit the fan, is you still a fan?”

If the advance word on good kid, M.A.A.D. city was that it would be the album that was going to resurrect the increasingly unfashionable art of the skit, then this is the album that sets out to restore nineties rap CD bloat in full. The short film of good kid has been upgraded for Kendrick’s take on the Great American Novel: plenty will take a look at the seventy-eight minute run-time and wonder where the cuts could have been made, but the terrifying truth is that every second of this album is almost alarmingly dense. To Pimp A Butterfly is a cavalcade of thoughts, styles, memories, regrets, sounds, concepts, protests – hell, fitting all this in on just the one CD is its own kind of miracle.

It’s an approach most readily apparent in the beats on the album, which must stand as the strangest, most stylistically diverse on any major label rap album since Outkast’s prime. To Pimp A Butterfly plays out as an all-encompassing rewrite of the history of black American music: lurching G-funk rolls into swinging ragtime, modernist hip-hop into quiet storm balladry, and throughout it all, a heavy dose of squalling jazz. Flying Lotus was only directly involved on one song – the crammed, frantic opener ‘Wesley’s Theory,’ which somehow finds the time to fit in George Clinton and Dr. Dre as it sketches out the exhilaration of a black man gaining power in America and also the great challenges he inevitably faces to stay there – but the fingerprints of the Brainfeeder collective can be found across the album, especially in Thundercat’s fluid bass-playing.

Indeed, Kendrick’s appearance on Flying Lotus’s ‘Never Catch Me’ proves to have been a greater hint at where Kendrick was heading than anyone might have expected: very few tracks here end in the same place that they start, and even the more straightforward tracks here, like the militant funk of ‘King Kunta,’ the Pharrell-produced ‘Alright’ (which is, suffice to say, a far more awkward beast than anything we’ve heard from him of late) or the solemn gospel of ‘How Much A Dollar Cost’ stand proudly separate from anything else likely to get even remotely close to the radio. Productions blur and smudge together, weaving in and out to create a grand collage that pays its respects to a hundred years of popular music whilst still looking forward. The experience of listening to To Pimp A Butterfly is to be un-stuck in time, to experience the past, present and the faint suggestions of the future at once, a swarm of voices all vying for attention, all speaking in different tongues yet all saying the very same script.

The only contemporary point of reference for Kendrick’s achievement here is D’Angelo’s astonishing comeback record last year, Black Messiah. Both are a direct address from and to the black community in the wake of the traumas of Trayvon Martin and Eric Garner, both call upon iconic figures from the past and present of music in pursuit of an auterial vision, both feel the pull of spirituality and the need for redemption, and both give a contemporary retro-fit to older sounds. But D’Angelo’s R&B vision is a less fiery, less intense thing: there remains something aspirational in his work, the politics of love and desire outweighing racial and social concerns in his writing. Black Messiah deliberately leaves a way out, looks forward to a brighter tomorrow, the rage and anger of ‘1,000 Deaths’ and ‘The Charade’ giving way to the guarded optimism of ‘Another Life,’ a path to peace found in the touch of another, your other.

D’Angelo returned after years of self-destruction in the wilderness redeemed and renewed, now an elder statesman imparting a message of hope and forbearance. The 27-year old Kendrick however is on a far more complicated, far more urgent mission – he looks at the grim fates of his rap forefathers, he finds himself on the precipice that D’Angelo found himself pulled over after the success of Voodoo, and asks himself if there can ever be another chance to say what he needs to say, tries to let out every possible thought or message before one door or another slams shut.

Indeed, one immediate effect of To Pimp A Butterfly is to re-write the end of good kid, M.A.A.D. city in a far bleaker hue. Compton was the triumphant credit roll, the sound of Kendrick joining the company of Dr. Dre and claiming kinship with his hometown whilst in absentia, having risen above the crime and the gang warfare. To Pimp A Butterfly finds Kendrick wrestling with survivor’s guilt and with his new-found fame: suspicious of power, suspicious of women and most of all suspicious of himself. He breaks down in a hotel room, curses himself for not being there for a dying friend, is tempted by Luci (guessed her real name yet?) and even cast down by God himself for his transgressions. If his music once seemed the answer to his immediate problems, To Pimp A Butterfly sees it creating a whole raft of new ones – and, more crucially, failing to fix the problems of those he left behind in Compton.

More than any other post-fame freak-out (Me Against The World, My Beautiful Dark Twisted Fantasy, virtually every single Drake song from his first mixtape onwards), To Pimp A Butterfly is a record about a lack of power, about aiming for the skies only to hit an impenetrable brick wall. If even Obama couldn’t change things, who can? Check that front cover again: it might look like liberation, the neighbourhood partying over the body of a white judge, but look again. That’s still the White House, built by slaves, looming unchanged therein the background after all. Ask yourself if anything has really changed, if maybe the progress that has been made is far more superficial and fragile than the official narrative would have it. If money, fame and authority change nothing, what can?

This then is the heart of the revisitings of To Pimp A Butterfly. If, Kendrick proposes, a culture and a history can be placed in a new context, if it can be properly understood and assessed, then the possibility of learning and progressing is open. So we get the familial wisdom of ‘You Ain’t Gotta Lie (Momma Said),’ the call-out to audiences and critics on Hood Politics (“Critics want to mention that they miss when hip-hop was rappin’ / [ ] if you did, then Killer Mike’d be platinum”), the reflections on slavery and origins on ‘Complexion (A Zulu Love),’ all backed by this fluid new-old production. Kendrick confronts his own demons so he can gain the strength to confront the demons on the outside, and he confronts them to offer himself as an example.

The whole of To Pimp A Butterfly is a series of mirror images and dualities, questions and answers that do not necessarily match but are given anyway so that the listener may find their own illumination. Aside from the obvious point that this album, far more than the hiring of Sasha Frere-Jones, looks set to be the real breakthrough moment for (formerly Rap) Genius, it is one that invites analysis in a very rare way. The broad strokes of Kendrick’s message – that it is necessary to conquer the enemy within oneself before you take on the world, that tracing a line from now back to the past throws up far more disturbing similarities than it does differences – are immediately apparent, but it’s the details and complexities that Kendrick throws in that will keep this album in the conversation for months and years to come. The allusions and references come thick and fast: yams as a symbol of (sometimes destructive) African-American masculinity and power, the interrogation of our confused relationships with cultural heroes – remind me to take the day off social media when people start honing in on the Michael Jackson lines – and the futility and nihilism of rivalries power struggles in politics and on the streets (the DemoCrips and ReBloodlicans Kendrick dismisses on Hood Politics). To Pimp A Butterfly reflects the intellectual effort and self-examination that Kendrick had to undergo to make it – a similar effort, it is clear, is expected from the listener.

It’s in this context that, at last, the real master plan of ‘i’ is revealed. On its release last year, it seemed to many listeners just far too easy, far too simplistic a get out to just boast “I love myself!” over an Isley Brothers sample whilst Ferguson turned into a grotesque police state. Here however, coming hot on the heels of the fury and vitriol of ‘The Blacker The Berry’ and the homilies of ‘You Ain’t Gotta Lie (Momma Said),’ it acts as a rare moment of certainty amidst an album full of soul-searching and declamation, a necessary moment of triumph and jubilation. To find something celebratory after all the chaos and tumult of the last hour, it seems, might be some small yet vital victory, and to anyone listening who still hasn’t figured it out, the album edit fades into a pseudo-live scenario where Kendrick appeals for calm at a rowdy performance, calling for people to rise to their best and to unify, to keep hope alive no matter what. A throwaway lead single finds itself reworked into the secret pivot of the album, and the sweet sugar to help the medicine go down – a small shaft of light bursting through the dark.

So after all that, we arrive at – but where else? – The End. ‘Mortal Man’ is a reflection of Kendrick’s grand ambitions and dreams, but also an acknowledgement of vulnerability. Can he be the person he aspired to be, or will his flaws come to define him? Can he outrun the odds, or is he doomed to an untimely end like so many of his heroes? Mortal Man frames all that comes before it as a desperate attempt to get it all out there, to make something important before the chance slips away and the need to right so many wrongs, because really, as it turns out, this was all a fan letter all along.

The moment when the music stops, when the poetry stops and the voice of Tupac Shakur comes in is a seismic pull of the rug, even when you know what comes. It is as bold a statement as you can get – the great new pretender, directly asking for advice and wisdom from the old master who, even for just a few minutes, he brings back to life – and one ends the album on a haunting, doom-laden note. Tupac Shakur left us great works alright, but he was shot down before he had the chance to the yawning contradictions between the sensitive street poet of Dear Mama and the man jailed for sexual violence, to resolve the stupid, terrible feud that would end his life, to resolve much of anything. Kendrick’s triumph at the end of good kid, M.A.A.D. city was to plot an escape from the cycle of violence that Tupac never could, yet that spectre doesn’t fade so easily. It’s a stark reminder to artist and listener alike of the challenges that lay ahead, a reminder that the past can still speak back.

But of course, this is just one reading of To Pimp A Butterfly, one reading from someone far removed from the fray. This is a work of rich multitudes, lyrically and musically, with a wealth of treasures to unpack. If something can be said of it with certainty though, it is this: the promise Kendrick made in that Control verse to kill the competition? Well, try and imagine anyone else putting out an album like this. Maybe you might spot a few of them enrolled in the To Pimp A Butterfly night class in a year from now.