The choice of a window in a discrete Fourier transform (DFT) is an art due to tradeoffs that it implies. When the time domain signal with a single frequency (e.g. a sine-tone) is transformed into the frequency domain, it creates a lot of other ‘phantom’ frequencies called sidelobes. The no-window choice of the rectangular window has a -13 dB sidelobe whenever the frequencies of the signal are not perfectly divisible by the window length (but when they are there are no sidelobes). In practice, for audio processing, I see Hann and Hamming windows being used. The Hamming window is my personal favorite due to a very high sidelobe rejection of over -40 dB, and has a raised edge, which destroys less information. Also Richard Hamming is cool (cf. see: Bell labs – Shannon’s manager and the famous asker of the ‘Hamming Question’) .

When I think about windows I usually think about the Fourier transform of these windows. But they don’t tell the full story. Phase, as usual, complicates everything. Notice how the sidelobes are themselves periodic. Without even analyzing anything, this implies there is some complex rotation going on, and the DFT is just ‘sampling’ this rotation at regular intervals. This is why you can use a rectangular window on a signal with a certain frequency and get no windowing artifacts or sidelobes – in this case the sampling/aliasing of the window happens to fall exactly where the complex values are the same. So as you move away from the exactly-aligned frequency to other frequency, the sidelobes ‘pop’ into existence.

Generally, DSP practitioners just pick one window for an application and stick to it. In theory, we could use an adaptive window to reduce sidelobe confusion, but it is not simple. Using an convolution directly on the waveform in ML processing is effectively assuming a fixed window, (and often comes with windowed preprocessing) as in TasNet or wav2vec. Something about this feels like a Bayesian problem since the choice of window provides different uncertainties conditional on the application and properties of the signal. It’s fun to think about how an adaptive window might work. Adaptive filters are fairly common, of course (and virtually all ML-processing is adaptive), but somehow the first thing that touches the signal – the window – is more difficult to touch. Perhaps this has to do the the relative stability that a fixed window brings. But if you look at picking the window for each frame to reduce the uncertainty. The ‘easy’ way out, is to use a very high sampling rate and/or Mel spectrogram, which smoothes out the sidelobes considerably for the high frequencies. Anyway this is a brainstorm, and something to look at further in the future.

The other thing I want to dig further into is the complex phasor for these windows. In other words, how the real and imaginary values evolve with respect to frequency for the window. My guess is that it looks a lot more steady than the sidelobes in the typical magnitude frequency domain that we commonly look at, especially if we looked at the Dolph-Chebyschev window which is designed to have flat sidelobes. This is not a big leap in perspective, and probably something people have looked at, but I don’t recall it being discussed in my signal processing education. I’d like a 3blue1brown-style animation of this.

‘Uncertainty’ as a word is very useful in talking about probability theory to the general public. Because we associate fear and doubt and other emotions with uncertainty in the casual usage, the word immediately gets across the point that at least some of the variance is due to the observer’s mental state, which translates well to the concept of prior information. But how much of the uncertainty in a random variable or parameter is due to uncertainty, and how much is actually in the process.

Edwin Thompson Jaynes’ chapter 16 of Probability Theory: The Logic of Science (16.7 What is real, the probability or the phenomenon?) is very explicit in saying that believing randomness exists in the real world is a mind projection fallacy. The projection is that because we things appear uncertain to us, we believe the generating process itself is random, rather than appreciating our lack of information as the source of the uncertainty. We had a discussion in our reading group about this – Jaynes being a physicist and all, it seems like a strong statement to make without qualifiers, since ‘true randomness’ might exist in quantum processes, like entanglement/wave functions, when an observer ‘samples the distribution’ by taking a measurement.

But if we take a step back, my reading group buddy mentioned that Jaynes’ field was closer to thermodynamics, not quantum mechanics. So maybe he was only considering particular ‘higher level’ aspects of the real world that have deterministic properties. But in many fields treating processes as stochastic is fairly common. Perhaps they don’t consider it as part of the real world or not, but usually humans don’t care about the territory as much as the map because the map is literally what we see. I suppose the problem with this approach is when we encounter Monty-Hall like paradoxes, which are surprisingly infrequent in the real world, probably because things are tangled up and correlated for the most part. Below are some examples from my world where the stochastic process is considered. I don’t find these problematic, but are kind of interesting to think about.

In discussions with machine learning/signal processing folk, sometimes I hear the distinction between stationary noise and signal, or between voiced and unvoiced speech as deterministic and stochastic, with attempts to model as such. My own group at Google even used such a strategy for speech synthesis. Here, ‘stochastic’ and ‘uncertain’ are interchangeable. If we knew the sound pressure levels of all points in the past millisecond within the 21.4 cm (343 m/s speed of sound divided by 16 kHz sample rate), we would be able to predict the next sample at the center of the sphere with higher accuracy, even if it was Gaussian noise. Jaynes believes this uncertainty can be reduced to zero with enough information. For this case, it’s much easier than quantum mechanics to see it as a deterministic process, since sound is mostly just pressure waves moving linearly through space.

Another connection from my perspective is the connection to process based ‘academic’ music, such as John Cage or Christian Wolff, and of course Iannis Xenakis, who explicitly references stochastic processes. Here, I think the term stochastic process tends to refer to explicit (as in Xenakis’ granular synthesis) or implicit (as in Cage’s sheet tossing in cartridge music or die rolling in other pieces). Die rolling music goes back at least as far as to Mozart’s Würfelspiel, but I think Mozart thought about it more as a parlor trick than an appreciation of randomness. The Cage vs Xenakis style can be considered from Jaynes’ pure determinism stance, and gets even crazier if you consider the folks that believe consciousness arises from quantum processes, since Cage/Wolfe often use the signalling/interaction between musicians to ‘generate entropy’.

I find that statisticians, or at least statistics literature tends to much more opinionated than other areas of STEM, like computer science and math to the point where it is quasi-religious, but I’m curious if insiders e.g. in pure math feel otherwise. Recent examples are Jaynes and Pearl, which make interesting arguments but survey histories with some preaching for at least a quarter of the text. It makes it interesting to read, but also difficult to know if I’ve processed it well. This book is full of great examples that I feel I will need to look at from other perspectives.

At the end of the day though, I’m uncertain (no pun intended) if the real world contains random processes, and how these might bubble up to ‘real randomness’ in larger processes (or if the ‘randomness’ is aggregated in some central limit theorem-like or law of large numbers-type of way that ‘cancels it out’). I am certain that uncertainty exists in my mind, and that a good amount of it could be reduced with the right information. But I also like the information theory/coding problem where we treat some sources of uncertainty as things we don’t care about the fine details of (the precise order of grains of sand doesn’t matter for an image of a family on a beach). In this case we care about the grains of sand having some plausible structure (not all the black grains clustered together, but uniformly spread out, with some hills). This maps well to the way classical GANs or VAEs operate by injecting noise to fill in these details that capture an incredible amount of entropy as far as the raw signal goes, but is constrated the way recent GANS, Conformers, or MAEs don’t typically use any ‘input noise’ at all to generate all of the fine structure.

This was fun but it’s getting a bit rambley now, and I’m done for the day. I guess that’s what happens when I read and try to connect everything that I’ve done.

Like everyone else, I’ve been playing with ChatGPT for many different tasks. Here is a short story that I used ChatGPT to create. It seems able to do the basic request, which was to use mysticism to tie longevity biotech progress and mycology together with a metaphor in a vaguely sci-fi style.  I decided the theme while writing, and asked it to write and rewrite sections with my direction, and I hand picked which paragraphs to use for the final result which I’m sharing below.  The full sequence of prompts and responses is included at the end, along with some discussion.

The story is reasonable, and doesn’t have any obvious mistakes at the sentence level. I was impressed that it kept global structural integrity as well as it did, given the length of the story (~2900 words). Generally, I thought the story and quality of writing was better than something I could come up with in an hour. It starts half of the paragraphs with ‘As I’; it has clear storytelling flaws; it doesn’t always respect literary devices. For example it gives up metaphors by exploring metaphysics on the story too easily. It did, however, use apt metaphors at other times: how nature represented by the forest can be both powerful, beautiful, scary, and if we look closely at it we can find solutions. On the other hand, I’m not a creative writer, and perhaps this story is of much lower quality than I realize. Maybe that’s the sweet spot: better than a total newbie, but not an expert. A lot of this task was about maintaining a certain style, and from the way the GPT-family works, it’s reasonable to think it could do this well.

At the same time, it misinterpreted or failed to acknowledge several instructions. I had to ask it to rewrite sections. Still, it was an interesting process that had me glued to the screen for an hour. Other ChatGPT tasks such as ‘help me research subject x’, ‘write code with me’ or ‘be my chat companion’ were less interesting, and also felt less powerful, and had more obvious mistakes (example: when I was researching standing desks, the agent was certain that standing desks were used by having people stand on top of the desk surface). Anything that requires expertise seems a risky thing to ask of ChatGPT.

One immediate application this gives us today is the ability for people that aren’t so actively doing media creation to try it out and get an appreciation of the feeling for writing, as opposed to actually using it to publish stories. If you want to publish it, there’s also a reduced sense of the typical apprehension of releasing a part of something you’ve worked on into the world because you can always blame it on ChatGPT. I, personally woudn’t feel comfortable sharing this story otherwise.

Depending on the medium, it might not even be that bad for publishing. I think novels and essays are usually quite refined and may be too high a bar to challenge at this point, but someone could make the argument if these large language models are used for assisting in creative endeavors with lower requirements, such as some Netflix series or movies. Netflix movies like Bright already feel like they are the result of taking current social trends and topics as input to some machine learning algorithm that spits out a script, and are often quite bad. Although here, the problem may not be so simple. In Bullshit Jobs, David Graeber mentioned an inside story from recent media producers explaining why so many films and TV shows are mediocre. It had to do with so many unnecessary ‘creative executives’ that don’t do anything for the script, except make small changes to be able to claim they did something. I have no idea how true that is, but this is more of a societal issue. But I could totally see passing the scripts through ChatGPT and getting better stories than some of the current ones.

I have many more ChatGPT interactions, and want to share them, but since I’ve already posted on it I’ll give it some time to see if that feeling survives the hype.

Complete prompt/response:

I’d like to write a scifi story with you.  But first I want to know your limitations.  Since I’ll be asking you to edit and rewrite sentences, how many words or sentences or chapters can I expect you to be able to remember?  This will be important for keeping the story consistent and focused.

ChatGPT: As a large language model trained by OpenAI, I am not capable of remembering specific sentences or chapters of a story. My purpose is to assist with generating text based on the input and prompts provided to me. I can help with editing and rewriting sentences to improve the clarity and consistency of the story, but I don’t have the ability to remember and keep track of specific details across multiple interactions. My main goal is to help you with your writing and to provide suggestions and ideas based on the information you give me.

Me: I will only ask you to remember my text input from this conversation, but I still need to know how many tokens you can remember.

ChatGPT: As a language model, I don’t have the ability to remember specific sentences or pieces of text. My primary function is to generate text based on the input and prompts provided to me. I can provide suggestions and ideas to help with your writing, but I don’t have the ability to remember and keep track of specific details across multiple interactions. My main goal is to assist with the writing process and to help improve the clarity and consistency of the text.

Me: For example, if I give you an entire chapter of say, 2000 words, can you keep that all in memory, or will it be problematic?

ChatGPT: As a language model, I don’t have the ability to remember specific sentences or pieces of text, including entire chapters of a story. My primary function is to generate text based on the input and prompts provided to me. I can provide suggestions and ideas to help with your writing, but I don’t have the ability to remember and keep track of specific details across multiple interactions. My main goal is to assist with the writing process and to help improve the clarity and consistency of the text. If you have a specific chapter that you would like me to help with, you can provide the text to me and I can assist with editing and rewriting sentences to improve the flow and coherence of the chapter. However, I won’t be able to remember the chapter in its entirety across multiple interactions.

Me: Ok fuck it, let’s go.  I hope swearing is okay.

ChatGPT: As a language model, I am a neutral AI assistant and don’t have personal preferences or opinions on language. My primary goal is to assist with your writing and to provide suggestions and ideas to help improve the clarity and coherence of the text. Whether or not to use swear words in your story is entirely up to you as the author. My role is to help with the writing process and to provide feedback and suggestions based on the information and prompts you provide to me.

Me: For this short story, the first word of every sentence will combine to be meaningful comentary on this story.  After this input sequence, I will begin writing the story, and you can edit or continue the story in your response.  If I have requests or commands, {I’ll use curly brackets to indicate this as separate from the story}.  You can also use those curly brackets to give me feedback or directions.   Do you think this is a good system to collaborate, or would you advise some modification to it?

ChatGPT: This is a creative and unique way to approach collaborative writing. Using the first word of each sentence to create a separate commentary on the story is an interesting concept, and I’m looking forward to seeing how it will develop in the story. One potential issue with this approach is that it may be difficult to maintain a consistent and coherent narrative while also crafting the commentary using the first words of each sentence. It may be helpful to plan out the overall structure and direction of the story before beginning to write, and to consider how the commentary will fit in with the narrative. Using the curly brackets to indicate requests and commands is a good way to keep track of these, and I will be sure to follow any instructions provided in this way. Overall, I think this is a promising approach to collaborative writing, and I’m excited to see what you come up with. Let me know if you have any specific requests or ideas for the story, and I will do my best to assist.

Me: When I woke up this morning, I heard the rain hitting the window and the roof.  It’s the only sound I’m used to.

ChatGPT: I’m not sure what I was expecting to see when I got out of bed and made my way to the kitchen, but the sight of my empty fridge and the dirty dishes piled up in the sink was almost a relief. At least it was familiar. As I poured myself a cup of coffee and looked out the window, I couldn’t help but wonder what my life would be like if I had made different choices. But it’s too late for that now. All I can do is try to make the best of what I have.

Me: {rewrite that paragraph respecting the rule that the first word of sentences is its own commentary and should be grammatically correct}

ChatGPT: When I woke up this morning, I heard the rain hitting the window and the roof. It’s the only sound I’m used to. Although I didn’t know what to expect when I got out of bed and made my way to the kitchen, the sight of my empty fridge and the dirty dishes piled up in the sink was almost a relief. At least it was familiar. As I poured myself a cup of coffee and looked out the window, I couldn’t help but wonder what my life would be like if I had made different choices. But it’s too late for that now. All I can do is try to make the best of what I have.

Me: {add a hint of murakamian mysticism from something outside the window}

ChatGPT: When I woke up this morning, I heard the rain hitting the window and the roof. It’s the only sound I’m used to. As I made my way to the kitchen, I couldn’t help but notice the strange light coming from outside the window. It was as if the rain was glowing, casting an otherworldly glow on the street below. I poured myself a cup of coffee and stepped outside onto the balcony, where I was greeted by the sight of a fox standing in the rain, staring up at me with piercing eyes. In that moment, I felt a strange sense of connection to something beyond my own world, as if the fox was trying to communicate something to me.

Me: I would have guessed that foxes don’t care much for the rain.  But then again, the stone-moss petrichor that the light wind pushed around suggested it hadn’t been raining for so long.  

ChatGPT: As I stood on the balcony, watching the fox in the rain, I realized that I had been wrong about the fox’s reaction to the weather. The fox didn’t seem to mind the rain at all, and instead seemed to be enjoying the fresh scent of the stone-moss petrichor that the light wind was blowing around. It occurred to me that it must have been a while since it last rained, and the fox was taking advantage of the moment to explore and enjoy the world around it. I couldn’t help but feel a sense of curiosity and wonder at the fox’s behavior, and I found myself wanting to know more about this mysterious creature.

Me: The fox was at a distance far enough that it wouldn’t feel threatened by anything I did.  When I’m out in the woods, sometimes I make a loud hoot to let the animals know I am there.  It’s entirely possible this fox and I had communicated in the past.

ChatGPT: As I watched the fox from the balcony, I wondered if it had ever noticed me before. The fox was far enough away that it didn’t seem to be threatened by my presence, and I couldn’t help but wonder if it had somehow sensed me even before I stepped outside. I remembered the times when I had been out in the woods and made a loud hoot to let the animals know I was there. It was entirely possible that this fox and I had communicated in this way before, and that it was responding to my presence now. The thought of this possibility filled me with a sense of excitement and curiosity, and I found myself wanting to know more about the fox and its behavior.

Me: I see that you are treating my text as if it’s guidance and not the actual story.  That’s fine, let’s continue like this.  Add a faint and looming sense of danger in the next three paragraphs.

ChatGPT: As I continued to watch the fox from the balcony, I couldn’t shake the feeling that there was something off about the situation. Despite the peacefulness of the rain and the fox’s calm demeanor, I couldn’t shake the sense of unease that was growing inside me. It was as if something was lurking just out of sight, waiting to reveal itself at any moment. I felt my heart rate increase and my palms start to sweat, and I couldn’t help but wonder what was causing this sudden sense of danger.

I tried to tell myself that I was just being paranoid, but the feeling only grew stronger. The fox seemed to sense it too, and it turned its head to look in the direction of the woods behind my apartment building. I followed its gaze, but I couldn’t see anything out of the ordinary. Still, the sense of danger was palpable, and I couldn’t shake the feeling that something was about to happen.

I considered going back inside and locking the door, but something held me back. I was drawn to the fox, and I felt a strange sense of connection to it. As I watched it, I realized that it was the only thing keeping me grounded in the face of the looming danger. I took a deep breath and tried to steady my nerves, determined to face whatever was coming my way.

Me: I felt existential dread from the sudden fear that this world would cease to exist if I went back inside the apartment.  Or worse, that the entire world consisted entirely of what was in front of me.

ChatGPT: As I stood on the balcony, facing the fox and the sense of danger that surrounded us, I felt a sudden wave of existential dread wash over me. The fear that this world would cease to exist if I went back inside the apartment, or that the entire world consisted only of what was in front of me, was overwhelming. I felt my chest tighten and my breath quicken, and I was overcome with a sense of panic and despair.

In that moment, I realized that I had to do something to break free from this feeling. I couldn’t just stand there and let myself be consumed by fear and uncertainty. I took a deep breath and tried to focus on the fox, using it as a anchor to keep myself grounded in the face of the looming danger. Slowly but surely, the fear began to recede, and I was able to think more clearly.

I realized that I needed to take action if I wanted to survive this ordeal. I couldn’t just stand there and let the danger consume me. I had to face it head on, no matter how scared I was. With renewed determination, I took a step forward and began to move towards the woods, ready to face whatever was waiting for me.

Me: It’s quite complicated and ridiculous looking to get down to the ground from my balcony in my pajamas, but this seemed more and more logical because I couldn’t remember anything else.  

ChatGPT: As I made my way towards the woods, I couldn’t help but notice how ridiculous I must have looked, trying to navigate the difficult terrain in my pajamas. But in that moment, nothing else mattered except for my desire to face the danger and protect the fox. Despite the complications and potential embarrassment, I continued to push forward, determined to find out what was lurking in the woods.

As I neared the edge of the woods, I felt my heart pounding in my chest. I was filled with a sense of determination and fear, unsure of what I would find but knowing that I had to face it. I took a deep breath and stepped into the woods, ready to confront whatever was waiting for me.

The sense of danger was palpable as I made my way deeper into the woods. The fox followed close behind me, its presence providing a sense of comfort and reassurance. Despite the fear and uncertainty, I felt a sense of determination and strength that I hadn’t known I possessed. I was ready to face whatever was waiting for me, and I was determined to emerge from this ordeal victorious.

Me: Add a hint that the world is simulated.  This is a short story and will end within the next 20 paragraphs

ChatGPT: As I made my way deeper into the woods, I couldn’t shake the feeling that there was something strange and surreal about the world around me. The trees seemed too perfectly placed, the sky too clear and blue, and the sounds of the forest too muted and distant. It was as if I was walking through a carefully crafted simulation, and the fear and danger that I had felt earlier only added to the sense of unreality.

I tried to push these thoughts to the back of my mind, focusing instead on the task at hand. I had to find out what was lurking in the woods, and I had to protect the fox at all costs. But as I continued to move forward, the sense of unease only grew stronger. It was as if the entire world was conspiring against me, determined to keep me from uncovering the truth.

Despite these doubts and fears, I continued to push forward, drawn by a sense of curiosity and determination. I had to know what was waiting for me in the depths of the woods, and I was determined to find out, no matter the cost. As I moved closer to the center of the forest, I couldn’t shake the feeling that I was on the brink of uncovering a great secret, one that would change everything I thought I knew about the world around me.

Me: I searched for some of my dearest memories

ChatGPT: As I made my way through the woods, I found myself thinking about some of my dearest memories. I remembered the time when I was a child, playing in the park with my friends. The sun was shining, the birds were singing, and life was full of possibility. I felt a sense of nostalgia and longing, and I couldn’t help but wonder if I would ever be able to experience that kind of joy and freedom again.

I also thought about my first love, the person who had taught me what it was to truly care for someone. We had spent hours talking, laughing, and dreaming, and I had never felt more alive. But in the end, it hadn’t been enough, and we had gone our separate ways. I still thought about that person sometimes, and I couldn’t help but wonder what might have been.

As I continued to search for the source of the danger in the woods, I realized that my memories were both a source of comfort and pain. They reminded me of who I used to be, and they made me long for the things that I had lost. But at the same time, they also gave me strength and determination. They reminded me that I had faced challenges before, and that I was capable of overcoming them. And with that knowledge, I continued to push forward, determined to uncover the truth and protect the fox.

All disciplines have interesting histories that explain their development. For some reason, different studies seem to ‘value’ their history differently. Art and music students are required to study art and music history with multiple dedicated classes. Computer science and mathematics students do not have to study math or computer science history in a typical undergraduate program (if you are lucky, maybe one or two classes). Instead, if the computer science or math student is lucky, they get a charismatic professor who is a good story teller and fits in anecdotes about the creators of the topics being studied, or a reference to a book about this.

Presumably, the history of art is useful to an artist, not only for telling interesting stories to their students one day, but also to understand something deeper about where new art comes from. A deeper understanding may be useful for a number of things, including how to go about creating novel art. The creation of new abstract concepts in math and art have certain high-level similarities – they provide a new way to look at things. Perhaps the poster child for this type of thinking is Xenakis, whose existence united architecture, statistics, algorithms, and music, where knowledge across the fields had an interesting synergy. But even if we ignore cross-discipline examples, I think we will find that the innovators typically have had an interest in the history. Is the assumption for the sciences that this is correlation, and not causation? That argument can be made, but it seems less likely to me. Or is the assumption that in the interest of time, most undergraduate students don’t need to be innovators, and rather just need to understand how to solve the damn equation, not how it came to be?

Perhaps this is a straw-man. I may be generalizing from an unrepresentative experience, and it has been a number of years since I was in school. It seems like folks pursuing graduate degrees in math/computer science have more understanding of history, and because this information is easier to come by today than 20 years ago, people that self-study also pick this up naturally. In that case all I can say is that I was not aware of so much of the history involved in computer science, and I wish I had started my studies with something like James Gleick’s The Information.

Information theory is a relatively new field of the sciences. Of course, it did not spring out of nowhere. There are a few history-oriented books that describe its formation, but there is not much. Gleick’s coverage is by far the widest I’ve seen.

The book has an excellent cast of characters, starting out with long distance communication with African drums, Babbage/Lovelace and early computers, and Laplace. As the book develops the more typical founding characters in Information theory appear, with Maxwell and his heat daemons, Clausius and his entropy, Morse with his codes, Hartley, Shannon, Wiener, Turing, and Kolmogorov. What makes the book’s presentation special is the depth in which each character is gone into. There are a large amount of supporting characters and competitors that I hadn’t heard of, which provides great context for the developments. Naturally, a large amount of time is spent of the juicy rivalries such as the Shannon-Wiener relationship, but also how it fit into the rest of the world, e.g., how Kolmogorov felt about them.

I was introduced to a range of new connections that I was not aware of, including the Schrodinger (yes, that Schrodinger) connection to molecular biology and What is Life?. There were also nice teasers for the parts of info theory I haven’t had exposure to such as Quantum computing and Schorr/Feynman’s thoughts on it. There also deeper ties to fundamental math history such as the early developments in greek and arabic states from Aristotle to al-Khwarizmi. I was also unaware of the amount of obsoleted infrastructure required for telegraph networks, and the book spends a good time talking about the logistics of this kind of thing.

I very much enjoyed this book, although it still misses a few important areas. Notably, Kullback’s application of information theory to statistics, as well as Bayesian statistics and the related information criteria are not mentioned. Deep learning is also not mentioned, but the book was published in 2011, before the recent surge. Naturally, Gleick also discusses the fictional works of Borges. Unfortunately as much as I enjoy Borges, I found this to be the weakest part of the book.

At 426 pages, Gleick’s presentation is almost entirely conceptual and non-technical, so I think this would be an great bedtime read for anyone interested in the topic that isn’t in a rush. For faster and a more technical approach, one might consider John Pierce’s book.