Fq: Extracting Structured Data from Binary Files

Do you remember Jq, which allows you to extract data from any software output and format it in JSON?

Here is the same concept: having data formatted in JSON, but this time extracted from a binary file. Fq can thus display the data in its hex form and “transform” it into JSON, which is practical for outputting data from media such as MP3, MP4, FLAC, JPEG, etc., or even listing the values and functions included in a program.

For example, to output the header of an MP3, you can use the command:

fq '.frames[1].header | tovalue' fichier.mp3

We can also why not, extract the first JPEG image encountered in the binary:

fq 'first(.. | select(format=="jpeg")) | tobytes' file > file.jpeg

This makes it possible to feed databases or websites by extracting data from binary files, network packet captures, etc.

For example, for network frames, we can recover TCP frames that have HTTP GET headers like this (from a PCAP file):

fq'.tcp_connections | grep("GET/.* HTTP/1.?")'file.pcap

…etc., etc.

In short, a tool halfway between Jq and GDB (the debugger).

If you’re interested, the entire doc can be found here.