09
Dec 10
01:10
Listen to your computer: lstn
Lstn is another project I’ve been working on with Institute for Algorhythmics.
It is a program that sonifies real-time debugging data of other programs. If you were interested in the FuckingAudacity or FuckingWebBrowser demos you’ll probably like this one. Those were sonification from the inside out, and lstn is about external sonification:
We sonify opcodes, callstack, and active memory. The sonification of active memory is direct, but sonifying opcodes and callstack info is more tricky to do ‘mostly honestly,’ and we’re still trying to figure it out. The project is still in alpha stage.
You can use lstn to attach to pretty much any program on the mac besides itself.
You can also use lstn to do old-school sonification of chunks of memory (as opposed to jumping around wherever the opcodes go.) The sonification of memory this way has a suprising amount of periodicity in it, creating both tones, and even things like crecendi and decrecendi.
There is built in osc support, but we’re not sure yet what data to send, as my impression is that osc might choke if you send too much stuff at irregular intervals.
I’m hoping to work more on the display to draw a visual map of the memory and program counter and also some tree-like structure for visualizing the callstack.
Another feature to note is the speed control which reduces the speed of the program being attached to. This is important because our ears won’t be able to catch things that happen at rates higher than the sample rate. I hope as the sonification methods become more meaningful that this feature will have more value.
This project started last February, but has only gotten this far because I keep changing OS’s and computers. The program needs different source for each of these combinations, since the opcodes and opcode format for i386, i386_64 and ppc are different. As of right now it works on ppc best and i386 (core duo) pretty well. on ppc you can simultaneously attach to many different processes, but I haven’t gotten that to work on core duo yet.
This project has taught me a lot more about cpu internals and program structure than any other program, (or class for that matter) has given me. So if you’re a programmer and this kind of stuff tickles your whatever, I suggest taking a look at the source and asking me about it. Otherwise just hold on till we get a real release.
robermann
December 9, 2010
9:44 am
It sounds interesting. Which project is lstn, among those listed in http://algorhythmics.svn.sourceforge.net/viewvc/algorhythmics/ ?
mike
December 9, 2010
7:40 pm
sorry for not mentioning it. It’s ‘FuckingFucker’ (I may do a release that lets you fiddle with the memory of the program being attached to as well).
robermann
December 10, 2010
12:54 am
Could you briefly explain how you are going to sonify the opcodes? For example, the following lines (*) are quite obscure:
analysis->cursor = (float) (threadState.FUCKINGEIP-analysis->minpc)/analysis->maxpc;
analysis->phase = ((float)info->opcodebyte/256.0)*M_PI;
analysis->freq = 0.1+exp2cursor((float)info->opcodebyte/256.0)*5.0;
analysis->buflen =((float)info->opcodebyte/256.0)*kFFManAudioBufferSize*0.5+512;
FuckingFuckerMan::Instance()->GetOpcodeAnalysis()->bufstart = randint(kFFManAudioBufferSize-FuckingFuckerMan::Instance()->GetOpcodeAnalysis()->buflen);
It seems that the function “FuckingPortAudioCallback” uses in same way that buffer. Does in your purposes every opcode act like a sample number, conveniently trimmed between +1 and -1 values?
—————
(*) from FuckingFuckerMan.cpp, lines 886s
mike
December 11, 2010
7:42 pm
current opcode sonification creates an impulse in a game of life style buffer, where the opcode determines where in the buffer it lands. imo this is currently a weak part of the sonification. the effect it ends up having is making a ‘flickering’ noise when an (any) opcode is found.
also, unlike your java vm style sonification we are just sampling the process, not getting every opcode, so we can’t rely on having a regular number of opcodes per output sample – esp. when the threads are sleeping.
the real meat is loading memory pointed at by the edx register in the hopes that it is a pointer that will give us some active memory. this seems to work, because when using it on audacity with the right speed control you can get clips of the audio being analyzed.
also, i think the stack-depth based sonification will have a good result, but i’m still trying to figure out what to do. i want to do something nicer than just mapping to a frequency that takes some advantage of the tree/graph like structure that callstacks have over time.
robermann
December 12, 2010
5:30 pm
maybe for every call stack recursion you coud add a tremolo, or a glissando whose speed depends on the loops number… just ideas.