Hi everyone,
I have created a new libray which uses Heatshrink compression to compress files on an SD card.
I use this library on the electron to compress files before sending them to an FTP server. This saves about 50% of data.
It uses the excellent SDFat library.
The library lives at: github
That’s awesome!
@rickkas7 Wanted to ping you just so you know this is available. Sounds perfect for your FTP upload library, especially if your uploading image files from a camera.
Hmm, compressing already compressed JPGs (at least my cams provide these) won't win you a lot.
Ahh, never really thought about the JPG already being compressed.
I wonder if compression would many any difference at all. Even a little reduction in size would be helpful when using 3G to send data.
Just curious if there is software available to decompress the files. The info on GitHub is a bit lacking, and Googling it doesn’t seem to bring any useful results.
I have an electron project that could really use the compression this seems to accomplish, but I’m at a loss as to how to decompress the data once it is uploaded to the cloud.
@MichaelTriad, the data is shrunk using the heatshrink library. A binary version that can be run on a linux computer can be found in the bin directory. I see now that I did not include a description on how the run the heatshrink program.
Copy heatshrink to /usr/local/bin on a linux pc. Now you can call
heatshrink -d -w 10 -l 4 $compressedfilename $filename
To make it easy I added the following script to /usr/local/bin, called heat:
#!/bin/bash
name=$1
lzname=${name%.*}
echo "extracting $name to $lzname"
heatshrink -d -w 10 -l 4 $name $lzname
This will decompress the heatshrinked data using the parameters that are the same as used during compression. So by running
heat shrunkenfile.txt.lz
The file will be decompressed and the last extension removed, so another file in the same folder will be created that is called shrunkenfile.txt
Hmm, after some trial and error it appears that the Raspbian version of Linux can’t run that. Going to try with a virtual machine now that I have a little better idea of where to put what and all.
Ah right, that is the case yes. You need a x86 processor to run the binary. The other solution is to download the source from github and compile the source yourself using make.
Run:
git clone https://github.com/atomicobject/heatshrink.git
cd heatshrink
make
and if make and the right compiler is installed, you will get a heatshrink binary compatible with your platform.
There a quite a few tutorials on internet for compiling from source on Raspbian, so I suggest you read one of those if this explanation is too short.
I will update the readme so the next person has an answer to those questions.
It took a bit, having to deal with some idiosyncrasies of a virtual machine and permissions to access things, but I got it to work. A file compressed with the library on an Electron was successfully decompressed and perfectly matched the original.
I usually like using LInux, but there are times it tries my patience!
Thanks for your input and for creating the library!
I have a new question regarding the compression library: could it be used on a directory to not only compress the data, but combine the files in the directory? (Someone on the project wants a bunch of individual log files instead of one long one.)
I don’t know if this is possible. I believe my code is not build to do this. We do several log files as well, but just compress them one by one and then send them over FTP.