Log probability, entropy, and intuition about uncertainty in random events

Probability is hard thing for humans to think about. Setting that aside, there are a whole bunch of fields that care about log probability. Log probability is an elemental quantity of information theory. Entropy is a measure of uncertainty, and at the core of information theory and data compression/coding. Expected log probability is entropy:
H(X) = -\sum_x p(x) \log p(x) = -\mathop{\mathbb{E}} \log p(x)

For a uniform distribution this can be simplified even further. A fair coin toss, or die throw, for example, has uniform probability of heads/tails or any number, and we get:
H(X) = -\sum \frac{1}{n} \log \frac{1}{n} = -n \frac{1}{n} \log \frac{1}{n} =  -\log \frac{1}{n} = \log{n},
where n = 2 for a coin flip and n = 8 for a 8-sided die (because there are 2 and 8 possible values, respectively). So with a base-2 log, we get H(X_{coin}) = \log{2} = 1 for the coin flip, and H(X_{d8}) = \log{8} = 3 for a 8-sided die.

One thing that might not be immediately obvious is that this allows us to compare different types of events to each other. We can now compare the uncertainty in a coin flip and the uncertainty in a 8-sided die. H(X_{d8}) = 3 H(X_{coin}), so it takes 3 coin flips to have the same uncertainty. In fact, this means you could simulate an 8-sided die with 3 coin flips, but not 2 coin flips with some sort of tree structure: the first flip determines if the die is in 1-4 or 5-8, the next if it is in 1-2 or 3-4 if the first flip was heads, or 5-6 vs 7-8 for tails on the first flip, and the last flip resolves which of those two numbers the die ends up on.

You probably would have been able to come up with this scheme to simulate die throws from coinflips without knowing about how entropy is formulated. I find this interesting for a couple reasons. First, this means there may be something intuitive about entropy that we have in our brains that we can dig up. The second is that this gives us a formal way to verify and check what we intuited about randomness. For this post I want to focus on intuition.

The first time you are presented with entropy, you might wonder why we take the log of probability. That would be a funny thing to do without a reason. Why couldn’t I say, ‘take your logs and build a door and get out, I’ll just take the square or root instead and use that for my measure of uncertainty’, and continue with my life? It turns out there are reasons. I wanted to use this post to capture those reasons and the reference.

If you look at Shannon’s A Mathematical Theory of Communication, you will find a proof-based solution that’s quite nice. Even after looking at it if you haven’t looked at convexity-based proof in a while, it can still be somewhat unintuitive why there needs to be a logarithm involved. Here is a less formal and incomplete explanation that would have been useful for me to get more perspective on the problem.

There are a few desirable properties of entropy that Shannon describes. For example, entropy should be highest when all the events are equally likely. Another one of them is how combining independent events like coin flips or dice rolls combine the possibilities. This means that the number of outcomes is exponential on the number coin flips or die rolls. So if I compute the entropy of one coin flip and another coinflip, and add them together, the sum should be the same value if I were to compute a single entropy on those two coinflips together.

If you want a measure that grows linearly on number of coin flips or die rolls that achieves this property, then taking the logarithm of the number of combinations gives you just that. No other function that isn’t a logarithm will do that. This is because the number of possibilities of n coinflips is exponential. Notice that \log_2(2^n) = n, where n is the number of coin tosses and 2^n are the possible outcomes for n coin tosses). So the log inverts the exponent added by the combinations of multiple events, which gives entropy the linear property on n.

Nostalgia, Music, and Utility Functions

We accept that certain events that happened in the early course of our lives influence us with a permanence that lives on in our identity. For many people music will be one of those events. In the opening of High Fidelity the main character sarcastically warns about the dangers of music for kids. In this post I want to consider the after effects (such as nostalgia) for music after the novelty wears off. “Sentimental music has this great way of taking you back somewhere at the same time that it takes you forward, so you feel nostalgic and hopeful all at the same time.” — also from High Fidelity, and closer to today’s topic.

The older I get, the less surprises there seem to be. I probably should restate that. There are still many surprises, but the intensity and the excitement — the amount something grips me — seems to go down with age. This is a bit depressing if you look at it one way, but I don’t think it has to be. Rather it seems like it is a natural consequence of understanding more about the world. The most obvious analogue is film and literature – at first you can start with the classics and they are all amazing. Then eventually you get to a point where you can see how those classics influenced the newer works, getting more perspective and insight into the world. This is quite an interesting feeling too. But the novel spark of that first foray into the arts is really something special.

Where did those bright eyed, wonderful times go? I’m reminded of them most viscerally when I put on music. Sometimes I talk or write about experimental computer music. But the particular music that does this for me is only a few specific bands and genres that are now very far from me culturally – the teenage and college years of grunge, pop-punk, emo, and indie rock, from The Descendents to the popular garden state soundtrack. Near 40, I still feel a certain strong and invincible euphoria when listening to these genres, even though it’s been years since I listened. If I get into it and shake my head and air-drum it is even better. The music has wired up a strong pathway for belonging and excitement in my brain and then neurons were set to be read only, to last for much longer than I would have thought. When I am 80, will it still be there?

At the same time, the act of listening to this music feels more than a bit weird. Perhaps it feels masturbatory or escapist. There is a disconnect between who I am now and who I was then, and the music is an instant portal that revives those youthful brain pathways.

And if we take it one step further, where did those bright eyes and times of wonder come from? The act of falling in love with a (sub)-culture because of someone you loved is one of the slyest and most curious phenomenon that is near the heart of western individualism. The association with a beautiful face that you might get to see if you go to the next concert. Culture is much more than a sexually transmitted meme. The acceptance from dressing a certain way and having a crew is really powerful, and dancing in unison to a beat is a primal joy even if the footsteps are different. For me, it was always the elusive nature and the promise that glory was just around the corner. The Cat Power song goes “It must just be the colors and kids that keep me alive, because the music is boring me to death”, which is something that a popular artist might be able to say after overwhelming success. But I was always on the outskirts, never really feeling in the middle of the culture that I loved. The colors, the kids, they were great, but the thing the lasted for me was the music. And I think, just as with the friendship paradox, this is the case for most people.

What would it take to find something like that again? Is it a matter of age, hormones, and college or can we recreate that somehow? Would I even want to if I could? Why can’t I, say, take these nostalgic feelings from music that I grew out of and ‘install’ them into music and things I am interested in now?

This is related to rewriting your utility function, which is a very scary thing that will literally annihilate your identity if you get it wrong. Just as we are programmed through evolution to love doing certain things – to feel joy from the wind going through your hair on a good walk, we are programmed through society and culture to a lesser extent during our developmental youth. But we do not get to swap the way we feel about walking through nature and thinking about high dimensional spaces. The latter has an intellectual pleasure for sure, but it is less ‘natural’. If we could do this, the world would be very different in very little time. I am not sure if we would converge or diverge as a species, but I would posit that if you could make being good at your job feel like dancing for the average person, productivity would blow up globally and previous social problems would disappear in place of newer probably scarier problems. But this is a rabbit hole that has many tropes and discussions such as wireheading that I would like to leave aside for now. I just want to note that it seems that technology is still a ways off from this, but this feeling I get when I listen to the right music is a sneak preview. It is frightening and feels great at the same time.

Waialae Ave (fairly dry)

This weekend I went back to basics and did some very tape-y music on the electro-acoustically traditional themes of traffic and water. That being said I left my usual mode of synthesis-based composition at the door: this is a memoir piece that I made in Audacity. I haven’t been making much music lately, so my goal was to have fun making something that I’ll enjoy listening to in a few hours using the recordings of rain, wind, and traffic that I’ve been collecting.

The Audacity of Nostalgia

The Audacity of Nostalgia


The main traffic in rain was recorded outside of “The Curb” coffee shop in Kaimuki’s Waialae ave, the street where I spent most weekends of my youth hustling to sell magic cards to buy used video games in addition to 50 cent “ice cake” at the crack seed store.

Sitting at a fancy-but-gritty coffee shop that hadn’t existed a few years ago, recording the rain, I found myself comparing the smell of the rain and the air to the other places that I’ve recorded rain and similar relaxing noise in California.

Waialae Avenue is the main street that runs through Kaimuki in Oahu, where I grew up. Though it is constantly changing, I noticed that the initial scent rain there was markedly different, and possibly unique, evoking a strong nostalgia. Looking for an explanation, I discovered the phenomenon “Petrichor”, which involves accumlation of oils and bacteria in rock and stone during dry periods that are released via aerosols caused by raindrops. I became curious to know if the geographical and historical bacteria might cause this scent to have a unique signature.

From this line of thinking, I also became interested the acoustic signature of an area. I took recordings of mostly-water based sounds, from Waialae Avenue’s rain, the drizzle from a chanterelle hunt, Big Sur’s and waves, the Yuba River’s flow, and the airport that took me between these areas, and applied basic processing to emphasize their similarities and differences.

I wanted to combine some of these recordings in a piece that meshes them together, similar to how my consciousness hopped about from the roar of California’s Big Sur coast and the drizzle of my fall chanterelle hunts in the East Bay. There is also the motion of the Yuba river and wind of Ocean Beach in there. Also, I’ve enjoyed hearing “Kaze wo Atsumete” at SLC when taking a non-non-stop trip back from HNL to SFO, so it felt appropriate to put here.

On the subject of rain and computer music, I did get to train a tensorflow-wavenet model on my crappy hardware to get a rain-like signal out. It sounds somewhat like rain, but it turns out rain is similar to white noise anyway, so I probably should have started with something that had more tones in it. Hopefully I’ll probably write a post about using wavenet soon. Maybe after I get a 1080, so I won’t need to wait many days for a model.

All Songs Considered Not Considered Harmful But Rather A Bit Limited In Scope

AllSongsConsideredNPR, you are great. You have shows that cover literally everything, from debating whether AI will be become skynet or Her, to fixing my car while getting relationship advice (RIP Tommy). I spend 3 hours a day commuting like a real American, and most of that time goes to podcasts from NPR.

Despite the title of this post, I like All Songs Considered quite a bit. Let me address you in the second person. I like the genuine excitement that you have for music and shows, especially because of the your two main hosts are a model to look up to, being older than stereotypical ages of people that get excited about new music and shows. Even when you had show a few years back of the worst songs of all time where you teamed up with Carrie Brownstein to try to make me feel bad for liking everything (We Built This City – Yes, We didn’t start the fire- yes, Baby I love your way – DEFINITELY YES). I don’t even know to what extent you were trolling, but it made me more aware about one thing.

I like everything now. I have the things I really love, but I also pretty much like everything else that I hear. I think this is okay. When I go to a vinyl garage sale, I buy 75% stuff I want and 25% stuff I don’t even know and I am not ever unhappy with it (to be fair it helps that it costs $1/LP). It’s completely fine.

Couldnt be as picky when there were only 4 mp3s on the internet

Couldnt be as picky when there were only 4 mp3s on the internet

Remember in high school and college when ‘liking everything (‘except country’ – as the phrase went)’ and a mishmash of random MP3s was code for “I don’t know care about music and I’m not cool, but I know how to dl warez (aol ‘zarew’ channel and hotline represent)”? Well, I was never much of an elitist, but I understand how that sentiment came about – it’s because it takes time to learn about a certain kind of music, and at freshman year of college, you won’t probably be able to know deeply about more than a few unless you are the Kid That Knew All Music. Now college is long past for me, I’ve seriously studied and listened to much more music since. And now I’m the guy who likes everything. But I still don’t feel that I will ever have that expert feel (or ‘hipster tone’ when I am jealous) to talking about music. But I appreciate the expert knowledge of the band’s background, influences and similar artists that folks like the All Songs can share with me. It’s so nice to just be able to enjoy things that are presented to you in this format without doing any research-like brain activity.

But what is up with the title ‘All Songs’? I love hearing about Weezer, Leonard Cohen, Radiohead, Neutral Milk Hotel, but it does make things more like “Indie Rock, Folk, Female Hiphop and some Electronic Songs Considered”. If you had such a title, I would be 100% satisfied with the show. But as it stands I am 95% satisfied. 5% means that at some time during the show I wonder ‘what would it be like if they played something without conventional rhythm next. When I started listening, you mentioned ‘experimental’ artists sometimes so I expected some artists that really try to do something different with music. And there is much non-experimental music that you don’t play, not because it’s not accessible, but for some other reason.

My dream podcast would have hosts like you guys that also included equal amounts of classical (old and avant-garde), jazz, harsh noise, computer music, and maybe some hawaiian music. Those are just the styles that have been in my interests, but it would be nice to get the other styles as well, giving the show more of an ethnomusicology feel then a hipster bent. I wish it could be less comfortable, not only for the audiences, but also for the hosts. One of the things that’s great about your podcast is that it introduces new music, and I feel like that’s at odds with the need to be an expert on the subject. In a nutshell, the show could do with more ‘curiosity’.

To be fair, you do occasionally play more ‘friendly’ noise/drone/experimental stuff like Fennesz and Deafheaven, and that’s a great start. But it’s once in a blue moon and that stuff is still closely connected to the indie rock world. I do understand that if you play both ‘harsh’ and ‘eclectic’ pieces that you risk losing users, but isn’t that true for any change you make? Lately NPR podcasts have been booming and it does seem like the goal of many is to maximize listeners, but if that is really the main concern, then maybe the show needs a new name. Titles are important in subtle ways. As it is I notice that when I listen to your podcast I feel that I am opening up my horizons, but then when I think about the problem of only playing the music you have expertise on is actually kind of limiting. And even a little scary because I don’t always realize the latter, maybe in part due to the subconscious effect of the title.

Just to test the idea I am really curious as to what would happen if you sprinkled these tracks over several shows. These are not new songs, but it’s a start to get the conversation going:

Playlist with all songs here. Single songs:

– David Dunn – Mimus Ployglottos

– Kalapana – The Hurt

– Merzbow – Rainbow Electronics

– György Ligeti – Désordre (Etude No. 1)

– Takeda Wataru – Seikatsu No Gara

– Paul Koonce – The Flywheel Dream

John Coltrane & Johnny Hartmann

Most of those songs are fairly old, and of well-established artists in these styles. I used them for examples, but my ideal show would present these kinds of classics for reference as well as find newer stuff in those styles and introduce them to me.

What would happened if you got a more even distribution of songs by getting other listeners to also suggest styles? Would it become noise? I don’t think so. I think it would give the songs you already play more perspective. I think it would make you feel more vulnerable – but as a lot of this new podcast world is showing, vulnerability is nice and can actually help drive the show.

All that being said, given the age of this show I’m sure you’ve had this discussion before, maybe even on the show. I’m curious to know what was the result of that discussion. Obviously, I’m not a radio producer and can’t say what’s best for your show. It’s a lot easier for me to describe my ideal show and its ideals. I originally had the title ‘All Songs Considered Considered Harmful’, but that’s just because its hard to resist and would make for nice hacker news clickbait. You’re doing a great job anyway, and I’ll be listening.

Systems

The world is watching: one cannot walk through a meadow or forest without a ripple of report spreading out from one’s passage. The thrush darts back, the jay squalls, a beetle scuttles under the grasses, and the signal is passed along. Every creature knows when a hawk is cruising or a human strolling. The information passed through the system is intelligence.” – Gary Snyder, The Practice of the Wild

A while ago on Stack Overflow, I asked a potentially off-topic question about metaphors for memory allocation (it was less-horrible a crime to post opinion questions at the time). Some entertained my question, but I thought the most thought provoking comment was that it was a horrible idea to tie any real-world analogies because of the possible confusion.

ceci n'est pas une (binary) tree

ceci n’est pas une (binary) tree

Years later, I remember this question often. In the time that has passed, I have made the answer my own strawman argument that I can go to battle with to give various interests in my life metaphorical continuity. On the one side, we already have decided it’s useful to call data structures by names of similar pre-computer concepts, like trees, memory, lists, containers, (not to mention both ‘structures’ and ‘data’) – we don’t need to come up with brand new words for each of these things, because the power of abstraction in its ability to be repurposed and recycled. I have not met programmers that are confused about where the chlorophyll exists in the leaves of the binary tree, nor people that think they need to sprinkle water on their computer to insert a new node in a tree.

That’s fun to think about, because it’s a nerdy and quick evidence-based answer to why analogy is useful for programmers. But lately, I find myself thinking about how other systems in my life are alike.

Besides computers, I am currently interested in these systems below. In fact, the word ‘systems’ and ‘network’ pop up in most of my hobbies:

  • Engine/Car repair – appreciating what does what in my car (quite literally, appreciating ‘engineering’) helps me to feel more comfortable with what is probably the most expensive thing besides a house that I’ll ever own. Running into various space and tool constraint problems in replacing my head gasket/valve seals/timing belt feels very similar to debugging software when you need to come up with clever ways to get the info you want to solve the bug.
  • Chanterelles have roots that create a network between trees

    Chanterelles have roots that create a network between trees

    Ecological Systems – the interaction between plant/mushroom/animal/climate/terrain is so amazing that it goes far beyond the basic darwinistic view of a single organism’s fitness (did you know trees use mychorrizal networks (like a mushroom root) to transfer nutrients from the more abundant trees to the lesser ones?). Darwin said some pretty cool stuff in the origin of species about how every few feet comes a different distribution of plants and animals that are best suited to the environment. As I see more and more connections between the species, it seems more and more like the ecological system itself behaves as an organism itself.
  • Fitness – understanding the interaction/balance of muscles has been useful for my improving posture and well-being. At first, I thought that weightlifting and talking about muscles was just for serious strong bros, but after swimming and jogging didn’t really help me out, looking into muscular anatomy showed me some clear muscular imbalances that would be obviously helped out with weightlifting. This is a study of the body and the interaction between its parts – a very complex constraint solving problem – and the connections were more counterintuitive than I imagined. I had no idea for a long time that my hip angle is what would cause me to hunch over, and I didn’t even have the physical self awareness to note that I was walking around with my hips fully extended in pelvic-thrust mode. And of course, the exercise and progression of the body is an interesting system of its own that ties in others, including nutrition.

Becoming more familiar with all of these systems has brought me happiness, and has counterintuitively made the world more simple as I invest more time into understanding their internal structures and meaning. All of them seemed like impenetrable opaque black boxes when I first started looking at them, and I may have even had the impression that it wasn’t worth digging in because it was more efficient to leave the specialist knowledge to the specialist. It’s worth mentioning that my feelings about this are similar to the argument against ‘leave cooking to the chef’ in Zen and the Art of Motorcycle Maintenance.

Shepherd's tone scales are a good analog of Penrose stairs

Shepherd’s tone scales are a good analog of Penrose stairs

My master’s thesis and a large amount of my music was about finding similar structures in music and software. At the time I remember thinking that this was a very special connection, and that I just-so-happened to be lucky enough to be working with both music and software. But the real lesson was that seeming unrelated systems have deep connections, and that the noticing of these connections is a very powerful thing that can add depth and perspective to a subject one is interested in.

Although it’s always tempting to be a purist when coming up with bizarre theories, some systems are more compatible than other systems. This happens often in music – for example, a shepherd’s scale (a scale that seems to get higher and higher) is a great analog to Penrose steps (that Escher made famous), but my idea to make an audio-only version of space invaders never panned out because some things just won’t translate very well even if you really want them to. I’ve heard anecdotally that composer Gyorgy Ligeti, who was himself interested in visual phenomena, made it a point to tell his students that one shouldn’t force a sonic representation of a visual phenomenon just because the metaphor sounds good – and I would agree that there’s a fair deal of art in these connections.