I'm a big fan of CyberChef. One of its most useful features is "magic" and turning on "intensive mode". This will automatically detect the encoding used and can often detect 2-3 levels of encoding.
If you like that, try FTFY https://ftfy.readthedocs.io/en/latest/ which can automatically repair a huge range of ways that Unicode text can be broken by re-encoding. This would be a great addition to CyberChef if they could reproduce it in JS.
Classic example from the docs:
> ftfy can fix multiple layers of mojibake simultaneously:
>>> ftfy.fix_text('The Mona Lisa doesn’t have eyebrows.')
"The Mona Lisa doesn't have eyebrows."
Any hints on challenge #5? FromHex returns something that looks like it has the bz2 compression header. Trying to decompress w/ bz2 doesn't seem to work though.
42 5a 68 39 31 41 59 26 53 59 34 3d 45 44 3d 31 37 3d 44 45 3d 30 30 3d 30 30 28 3d 44 46 3d 38 30 3d 30 30 3d 31 30 68 3d 30 37 3d 46 37 3d 46 30 3d 30 43 3d 30 30 66 3d 30 30 3d 33 46 3d 45 46 3d 44 46 3d 46 30 30 3d 30 30 3d 0d 0a 3d 44 41 3d 38 31 48 3d 43 34 3d 43 38 3d 44 30 68 3d 30 33 43 3d 30 38 3d 30 30 3d 30 30 3d 43 30 3d 43 38 32 3d 30 30 3d 30 43 46 3d 38 33 21 3d 39 30 3d 31 38 35 4f 3d 31 34 3d 46 34 46 3d 39 34 3d 46 43 52 6f 42 3d 0d 0a 3d 39 45 3d 41 37 3d 41 36 6a 46 3d 39 41 63 3d 31 32 6d 46 3d 30 34 3a 32 3d 43 38 52 5d 52 7a 68 3d 31 30 3d 31 31 3d 30 31 3d 41 41 3d 31 46 3d 38 36 3d 31 45 3d 42 30 52 3d 42 41 3d 30 42 5e 3d 31 36 25 3d 41 46 3d 30 37 3d 0d 0a 3d 46 31 3d 44 37 38 74 3d 43 46 3d 42 37 3d 31 36 3d 39 35 3d 38 42 3d 30 35 3d 45 41 3d 39 34 30 44 3d 46 31 3d 31 34 3d 43 33 3d 32 32 62 6a 3d 32 32 3d 38 43 6d 3d 41 41 3d 45 35 3d 41 34 3d 44 45 3d 39 34 53 3d 42 34 2e 3d 0d 0a 3d 41 36 3d 42 33 3d 41 43 3d 31 42 3d 38 30 3d 30 38 3d 33 46 46 3d 44 41 2c 49 3d 43 35 3d 42 38 09 28 3d 41 32 72 35 3d 30 38 40 3d 46 41 2a 3d 31 45 61 3d 44 45 5e 3d 39 45 3d 44 42 3d 31 41 3d 43 33 3d 38 42 26 3d 0d 0a 3d 43 44 3d 39 32 3d 44 30 3d 43 32 3d 31 32 3d 41 41 2a 3d 30 33 48 3d 46 38 45 3d 45 38 30 3d 41 35 3d 43 35 40 3d 44 45 3d 46 30 3d 31 37 3d 43 38 3d 44 34 3d 41 30 3d 39 46 3d 43 45 3d 46 34 20 76 3d 30 36 2c 3d 0d 0a 43 6e 3d 45 37 3d 39 30 3d 44 39 3d 43 31 64 3d 30 35 3d 42 39 3d 30 30 58 3d 44 32 48 3d 46 41 3d 41 46 3d 41 44 3d 38 38 3d 38 30 30 3d 31 38 3d 46 37 3d 39 30 3d 31 32 4d 3c 3e 76 3d 30 45 6b 3d 38 42 5b 3d 42 46 3d 0d 0a 3d 45 36 3d 31 30 3d 38 46 3d 44 31 3d 44 31 3d 45 41 3d 45 42 3d 39 44 44 3d 44 30 3d 44 31 3d 43 34 74 3d 38 43 2b 3d 41 38 5e 3d 43 42 31 3a 50 3d 42 37 3d 30 38 3d 42 43 09 3d 0d 0a 3d 41 32 3d 31 37 3d 41 37 3d 31 46 3d 46 31 77 24 53 3d 38 35 09 3d 30 33 4e 3d 44 31 7d 3d 45 30
I just got through this one and was having the same block as you, took me a while to connect the dots. There's a step missing, take a look at some possible encodings before you decompress. Hope that helps without giving too much away, good luck!
afaik those are user IDs of Chinese(?) messengers. Devs or others there like to use those as usernames because the are (of course) mostly unique and also enable others to easily contact them.
I don't think in this case it'll be messenger IDs (and if it is definitely not chinese) as GCHQ is the UK's equivalent of the NSA.
They're a pretty secretive bunch, when they present in person they don't use real names, and if you go to their headquarters you have to leave all electronics at the door (did a talk there once and had to buy myself an analog watch for the day!)
Chinese messenger ID-as-username mostly starts from q(QQ - counterpart FB Messenger) and s(Sina Weibo - counterpart FB/Twitter). Others(a - admob, b - bilibili etc.) are quite rare tbh so I don't think that's the case here.
Flutter was a promising choice as it'd give me Windows, Mac OS and Linux build from a single codebase - and even the possibility of orgs running this internally as a self-hosted webapp. But one of the ideas I had was to bundle CyberChef with my app and open in a webview. Turns out, Flutter doesn't support webviews on desktop platforms at all. https://github.com/nileshtrivedi/devtoolbox/issues/4
You should check out the WebViewX plugin, for simple websites it works without any drawbacks, even though it's pretty hard to find by just searching for it
I have one of its predecessors (2GB USB 2.0 flash drives). Costed around $60 back then and it was worth it. There's hardly a day i don't use it.
After a very long time of daily usage one of the knifes broke. Totally my own fault. I sent Victorinox an email to thank them for their excellent product that lasted a long time of abuse. They replied to mail the thing to them so they could get it repaired for free. Everything was replaced, except the flash drive. Awesome thing, awesome service!
It's amazing that some companies can still do that. But then you get things like L.L. Bean ending policies because assholes are buying at Goodwill and then sending them in to be refurbed for free.
It's a static website. If you want to use it as an application you can download the compiled page (https://gchq.github.io/CyberChef/CyberChef_v9.32.3.zip) and create a shortcut to the 'Cyberchef.html' file contained within.
ht-editor was a fantastic editor[1], similar to hiew[2] on windows. Unfortunately ht-editor codebase is a bit hard to extend and it's based on really old modified binutils headers. I was trying to update it, but it's probably better to just write it from scratch, it's still much more straightforward to use that most modern cli hex editors.
What is with russians and their love for advanced windows cli tools? IDA Pro, hiew and far manager[3] come to mind.
This is actually very useful. I've been spending the past few weeks working on a cross-platform, native dev assistant app like this written with GTK3 and Rust. It's been a really good exercise in designing meaningful GUIs and, well, usable code. I don't have anything to show right now, but when I get it to an MVP state I'll be sure to share it with everyone!
I'm curious to hear what things people want to see out of an app like this. What utilities are you constantly Googling for that you'd rather have on your desktop?
Something I run into a lot is a JSON-like blob of text I'd like to be formatted as JSON (new lines and indentation). Most JSON formatters choke on improper JSON (understandably). It would be great to have a tool that was more lax. Like browser support for terrible HTML lax haha. If I could paste JSON-like strings into a text area and have it fix and format it as best as can be, that would be great. Some examples of non-JSON syntax to handle would be like single quotes instead of double, arbitrary JSON nodes (not necessarily wrapped in `{}`), some pre or post text (some non-JSON text at the beginning or end), comments amongst the JSON, etc. Another JSON aspect that would be useful is something to escape/unescape JSON (specifically double quotes). I deal a lot with JSON that includes escaped JSON in values and it would be great to have some better way to visualize and process those blobs. Sublime Text has a nice plugin to handle some of the escaping/unescaping[1].
I feel like I am forever having to hack together things to parse json to csv. It feels like there is never an easy path (indeed jq is frustrating, I usually end up just solving in python)
Not a fan, I have tried say formatting a large JSON API results (with no formatting) and it just runs out of memory because:
1) the string is huge like 64mb+
2) it tries to do syntax highlighting and blows up big time on the resulting formatted value.
For a utility tool it's awful finnicky on real world data and I think tries to be pretty at times rather than useful, or perhaps a case where a browser makes a poor Editor.
We might need to try fine tuning some limits. If CyberChef thinks it will have an issue rendering some text, we'll stop trying to display it and offer the user the ability to render only a part of the result or download the file.
Here's an example of me trying to format a 100mb file.
I always wondered if any of the online tools like JSONlint or any other tools that manipulate data would save it or try to extract data as it is often used to indent or decode potentially sensitive data.
It is very useful. I have issues with how both GCHQ and NSA operate, but I also really like the tools they provide like Cyberchef and Ghidra as they automate a lot of workflows.
The 'magic' command in cyberchef is pretty magic, especially for obscure (to me) utf and language encodings, though I have certainly run down more than a few rabbit holes because it detects the file magic for 'inflate' compression in pretty much everything, which would be perfect for hiding embedded files because of the number of false positives that appear when you're looking for them, and it's just the algorithm someone of a certain vintage who was serious about hiding something from everyone but someone else of that era would use.
The image analysis stuff is great. I use palette randomization for detecting embeds, and the entropy analyzer/visualizer is great fast method for detecting encrypted and zipped payloads. I like that it's fast, and it's there on the web so I can use it on anything.
