Isn’t a lack of snapshot files a downside? How do you review snapshot differences tied to a merge request? Being able to review code changes right along side the test changes and snapshot differences is a big plus for normal snapshot files in my book.
Hi there, maker of Touca here. Thanks for sharing your thoughts! Being able to visualize the differences along side pull requests is helpful. We currently provide that insight for each commit using Github Actions job summary and plan to add a GitHub App in the future to report differences as a PR comment. Touca also allows locally storing the captured data points for users who like to do so .
We argue that visualizing the differences shouldn't require storing snapshot files in version control because 1. they are inconvenient to maintain 2. what you can store in them is limited to what your application produces as output. I've written a bit about this in our docs here: https://touca.io/docs/guides/vs-snapshot/
Would really appreciate hearing if you find this argument reasonable.
Similar concern here about the lack of snapshot files. Can "locally stored" captured data points be version controlled, even if it's some special Touca binary that makes no sense in GitHub?
I'm coming at this from a "how would I integrate this into my existing corporate setup" perspective. I'm concerned less about UX and more just how it works.
If I interact with Touca via a local client that doesn't to the Internet, and the actual data points are exchanged entirely via Git, that's a much easier sell than another server that I have to ask SREs to maintain. And being able to say "it's just Git + a local client" makes it easier to reason about concerns like "what if we switch to GitLab" or "how would this integrate into our own bespoke CI/deploy pipeline".
Hi Andrew, to clarify, we do support generating local snapshot files in either JSON or Binary with FlatBuffers format (see https://touca.io/docs/sdk/capturing/#locally-storing-test-re...). We also provide `touca results` which helps with managing binary result files (see https://touca.io/docs/cli/results/). So it is possible to forgo the Touca server and just use the SDKs to capture values of interesting variables or runtime of important functions.
Storing test results in Git and comparing them via external tools may make it easier for teams to get started with Touca. But we care a lot about the developer experience and think that having a remote server take care of storage, comparison, visualization, and reporting could make a lot of sense for teams building serious software.
Touca has several components that work great together but you can certainly pick and choose and don't have to use all of them. We like Touca to do as much work as possible including integration with CI so that tests run continuously. But that integration is also optional. If teams have a different CI than GitHub Actions, they could call `touca test` as a CI step instead of using our plugins.
I’d argue that integration into GitHub via actions isn’t good enough—what about being able to review diffs using the git CLI? Or investigating snapshots with git blame?
I agree GitHub Actions job summary does not take the place of a GitHub App. The latter is part of our Q1 roadmap.
Reviewing diffs using `git diff` or `git blame` is not part of our roadmap. In fact we want to replace that model so that we can provide more insights and better summaries than just showing the diff between two textual outputs. Currently, you can use the Touca CLI to capture test results and get feedback whether they are any different. But you'd need to use our web interface to see and manage the comparison results in details. We're working to expand the CLI so that you could see the diff between any two versions right in the terminal.
