No more paper

I just decided I am really tired of filing all the bits of paper that collect in piles around here every year around 'tax time'. Also, I don't want to own a file cabinet anymore. It's heavy and bulky.

Capturing documents

First I need to get the docs into the computer. This includes anything paper that I want to file. Receipts, bills, articles torn from magazines, notes to myself.

The computer aka "the doc server" needs bullet-proof and secure backups, so I need some way to encrypt the docs and ship them off somewhere. Only critical docs need this treatment. I think I can afford slightly less bullet proof backups for my photo and music collections.

Currently I am using cloud storage for this.

Digital docs

I can already access many personal docs online.

I have to do this in a timely way though. they don't all archive everything forever. Plus I want them all accessible in one system.

I need to program the doc server to grab every relevant personal doc and archive it. This means bank and credit card records, and bills from various entities like the power company and the phone company.

I'd like to stash email in this server, too.

I also own a substantial number of e-books and e-magazines that I'd like indexed.

Paper to image

Flatbed scanner - I have an HP. I need to drag it out of the closet.

ADF scanner 2013-May - a friend gave me a Canon Pixma MX330 which has an "ADF" = automatic document feeder. AWESOME. It probably ran out of ink so the former owner dumped it. Inkjet printers are a scourge. I only need the ADF part right now. (Update: Yes, I bought ink and it works fine as a printer too.)

Digital camera - it's been suggested that you can use a camera and tripod. My eyesight is not that good. It might be useful to consider its ability to capture other things than paper docs. Also I'd like to store photos.

Image to words

Comparing OCR software - conclusion, tesseract is better than gocr.

Setting up tesseract in xsane

Would be great but requires an external script at the moment.

Scanner tips

sudo apt-get install xsane tesseract-ocr

XSANE scanning

Using xsane is tedious, too many things to remember and too many things to hack around. Fun for experimenting or doing special projects. xsane scans each one to a PNM file, sequentially numbered. To save the scanned images in a single file, select 'Multipage' (Ctrl+M).

• source = Automatic Document Feed
• set Number of pages
• Gray
• 150 DPI

Command line scanning

Find out where the scanner is hiding:
% scanimage -L
device `hp3900:libusb:005:010' is a Hewlett-Packard Scanjet 3970 flatbed scanner

Get the options for this scanner
% scanimage -d hp3900:libusb:005:010 --help

% scanimage -L
device `pixma:04A91737_22F601' is a CANON Canon PIXMA MX330 multi-function peripheral

Use a 150 dpi resolution. Gray scale not color. Write the output to a TIFF file. Since there is only once scanner I don't have to specify the device.

• 150dpi, 8.5x11 in pixels: 1275x1650, or about 500 x 640 in mm

Flatbed mode example

scanimage --progress -x 500 -y 640 --source Flatbed --resolution 150 --mode Gray --format=tiff  > page.tif

ADF mode example

scanimage --progress -x 500 -y 640 --source 'Automatic Document Feeder' --resolution 150 --mode Gray --format=tiff  > out.tif

Multipage ADF example: will produce 9 files numbered starting with out10.tif

scanimage --progress -x 500 -y 640 --source 'Automatic Document Feeder' --resolution 150 --mode Gray --format=tiff --batch-start=10 --batch-count=9

Compression for tiff

tiffcp -c zip name.tif name.z.tif
rm name.tif

Linux image viewers

View the result; pick an image viewer.

sudo apt-get install gimp paul mirage imview gimageview

Gimp -- works but is a heavy side
paul -- forget it
mirage -- ok
imview -- does not scale image to fit screen
gimageview -- (run 'gimv') controls not intuitive
gwenview -- the KDE viewer

OCR

"tesseract billname.pnm billname" will generate a text file "billname.txt"

convert doc.pdf doc.pnm

Photographic images

(and movies, I suppose)

Thinking along the same lines, I'd also like to store...

Audio data

Let's get rid of the CD collection, too.

Update 12/2008: My music collection is now on my hard drive.

Storing documents

Ideally I want all my documents in a pile, just like I do with paper. Then I want to be able search through the pile. The difference is that I want to be able to do the searching MUCH MUCH FASTER.

So I want at least the metadata stored in a SQL database. The docs themselves can be stored in BLOBS or files. I don't care as long as I can still get at them in 10 or 20 years when I need them. I think files would be best. Document File Formats

Indexing documents

Fine, so the metadata can live in a database. I need to decide what the best way to access the data is.

This journey started when I ran across the announcement for a release of phpMyArchive so I better check it out. This project has some very good ideas about making data storage and indexing.

I remember evaluating Swish-E so I will look at it again too. Swish-E indexes text docs.

I also played with MnoGoSearch for a while. It indexes documents and puts the index into a database. For articles it could go like this: Scan article. Attach title to it. OCR it. Then run indexer on it.