well you don't read the residual, you plot it. the ASCII plots have limited rendering resolution compared to a proper plotting system, like where you can zoom in and stuff.
> I'm also curious what the state-of-the-art is for these sorts of tests. Does anyone have insight into what e.g., appleton's test suite looks like?
I don't know, but if I were to make an educated guess, maybe rendering stuff to actual audio files is a common approach? That way when something goes wrong, they can inspect it in a standard waveform editor?
For audio you might also consider U+2581—U+2588, LOWER ONE EIGHTH BLOCK through FULL BLOCK. And then if you really want to go all out there are sixels, but that’s basically just an image; you probably lose the easy ability to compare them. On the other hand they’re not available in every terminal.
These blocks are great if you need more vertical than horizontal resolution. There are also newer symbols in the Unicode 13 spec that are just the line without the fill.
The same group wants to include a couple others, from different platforms, but the Unicode Script Ad Hoc Group is concerned the new batch may not be as meaningful as the first one.
In short, because text is much easier to deal with than bitmaps, and there is much more tooling that "just works" for text than actual graphics, like Expecto's textual diffing in assertations. @MayeulC said it well: https://news.ycombinator.com/item?id=28856884
That’s interesting. Do you think sixels could work for the baseline tests? Would it be feasible to have them display nicely in an IDE, like VS Code or Visual Studio?
I don’t see why sixels couldn’t work. You’d probably want a tool to decode them, diff the images, and then output another sixel image. I’m admittedly not sure of such a tool existing off the shelf though.
I’m not aware of text editors supporting sixels, which could make preparing the tests a challenge. Certainly, you could imagine a text editor supporting them, but I’m not aware of one that does personally.
I will concede that for your specific use case, an off the shelf ASCII plotting library probably involves less custom tooling.
The problem is NOT THE FORMAT, the problem is the lack of tooling: links and w3m are among the rare text browsers that can display images in the console.
It's just a matter of the browser sending the image to the terminal in some format it can understand, but if that hasn't be thought about as a possibility (say, for text reflow issues) it's going to be far more complicated than just adding a new format, as you will have to work both on say the text reflow issues (ex: how do you select the size of the placeholder, when expressed in characters?), on top of the picture display issues.
Said differently, it would be easier to have console IDE that supported graphics if any format whatsoever (sixel, kitty...) was supported by a console IDE; we could then argue about the ideal format.
Arguing about the ideal format BEFORE letting the ecosystem grow using whatever solution there is only results in a negative loop.
It's like if a startup was arguing about the ideal technological stack even before trying to find a product market fit!!
Personally, I do not care much about sixels, kitty or iterm format - all I want is to see some kind of support for a format that's popular enough for tools using it to emerge.
Yes, it would be better if that supported format was the option that had the greatest chance of succeeding, but right now, that is a very remote concern: first we need tools, then if in the worst case they are for a "bad" format, we can write transcoders to whatever format people prefer!
Right now, there is rarely any "input" to transcode (how many console tools support say iTerm format?), so we have a much bigger bootstrapping problem.
> an off the shelf ASCII plotting library probably involves less custom tooling
With a terminal like msys2 or xterm, no custom tooling is required: just use the regular gnuplot after doing the export for the desired resolution, font, and font size.
gnuplot is far more standard than plotting library that often require special Unicode fonts on top of requiring you to use their specific format.
Thanks, I like to think so! I didn't see other people doing much audio programming in F#, so I figured someone would be interested in seeing what it can look like.
That's true that it quantizes (aka bins) the samples, so it isn't right for tests that need to be 100% sample-perfect, at least vertically speaking. I suppose it is a compromise between a few tradeoffs - easy readability just from looking at the code itself (you could do images, but then there's a separate file you have to keep track of, or you're looking at binary data as a float[]) vs strict correctness. The evaluation of these tradeoffs would definitely depend on what you're doing, and in my case, most of the potential bugs are going to relate to horizontal time resolution, not vertical sample depth resolution.
If the precise values of these floats is important in your domain (which it very well may be), a combination of approaches would probably be good!
Would love to hear how well this approach works for you guys. Keep me updated :)
I'm not sure it makes sense to separate "vertical" correctness from "horizontal" correctness when it comes to "did the feature behave" though; to extend the example in TFA, if your fade progress went from 0->0.99 but then stopped before it actually reached 1 for some reason, you might find that you still had a (small, but still present) signal on the output, which, if the peak-peak amplitude was < 0.1, the test wouldn't catch.
Obviously any time you're working with floating-point sample data the precise values of floats will almost always not be bit-accurate against what your model predicts (sometimes even if that model is a previous run of the same system with the same inputs as in this case); it's about defining an acceptable deviation. I guess what I'm saying is that for audio software, a peak-peak error of 0.1 equates to a signal at -20 dBFS (ref DBFS@1.0) (which of course is quite a large amount of error for an audio signal), so perhaps using higher-resolution graphs would be a good idea.
Fair points here. Unfortunately adding more vertical resolution starts to get a little unwieldy to navigate through. Maybe it could start using different characters to multiply the resolution to something sufficiently less forgiving of errors. If it could choose between even 3 chars, for example, it would effectively squash 3 possible values into one line, tripling the resolution.
I think more resolution may give you more false negatives, which might not be helpful. We’ve used similar tools for integration testing at work and the smallest usually irrelevant change can bust the reference cases, due to the high-detail in the reference, which means going through all the changed tests and then seeing that everything is still fine.
For this, just thinking about sound, I wonder if you could invert the reference wave form and add it to the test to see how well it cancels? Then instead of just knowing there was a diff, you could get measurements of the degree of the diff.
Not yet, but it certainly could be. Would it be useful to publish the helper classes that render the waves out to ASCII? That's really the guts of the thing. After that, you just use whatever testing framework you want to do the actual diffing (in my case Expecto for F#).
