No more paper

I just decided I am really tired of filing all the bits of paper that collect in piles around here every year around 'tax time'. Also, I don't want to own a file cabinet anymore. It's heavy and bulky.

Capturing documents

First I need to get the docs into the computer. This includes anything paper that I want to file. Receipts, bills, articles torn from magazines, notes to myself.

The computer aka "the doc server" needs bullet-proof and secure backups, so I need some way to encrypt the docs and ship them off somewhere. Only critical docs need this treatment. I think I can afford slightly less bullet proof backups for my photo and music collections.

Digital docs

I can already access many personal docs online.

I have to do this in a timely way though. they don't all archive everything forever. Plus I want them all accessible in one system.

I need to program the doc server to grab every relevant personal doc and archive it. This means bank and credit card records, and bills from various entities like the power company and the phone company.

I'd like to stash email in this server, too.

I also own a substantial number of e-books and e-magazines that I'd like indexed.

Paper to image

Flatbed scanner - Yep, I have one of these somewhere. I need to drag it out of the closet. If things go well I will probably want to trade it in for one with a feeder on it.

Digital camera - it's been suggested that you can use a camera and tripod. My eyesight is not that good. It might be useful to consider its ability to capture other things than paper docs. Also I'd like to store photos.

Image to words

OCR via Tesseract My idea is not to make a perfect readable copy of the original but just to be able to grab enough keywords for indexing.

Scanner tips

I am currently scanning a manual that I want to share on this site. Scanning a long paper document one page at a time with xsane is not good for my sanity. So I use the scanimage program.

Find out where the scanner is hiding:
% scanimage -L
device `hp3900:libusb:005:010' is a Hewlett-Packard Scanjet 3970 flatbed scanner b

Get the options for this scanner
% scanimage -d hp3900:libusb:005:010 --help

Now measure the printed area, for example 180 x 250mm and the margin widths, top and side. Use a 150 dpi resolution. Greyscale not color. Write the output to a TIFF file. Since there is only once scanner I don't have to specify the device.

scanimage --progress -x 200 -y 250 -l 22 -t 20 --resolution 300 --mode Gray --opt_nowarmup=yes --format=tiff > page.tif

View the result; pick an image viewer.

sudo apt-get install gimp paul mirage imview gimageview

Gimp -- works but is a heavy side
paul -- forget it
mirage -- ok
imview -- does not scale image to fit screen
gimageview -- (run 'gimv') controls not intuitive

Photographic images

(and movies, I suppose)

Thinking along the same lines, I'd also like to store...

Audio data

Let's get rid of the CD collection, too.

Update 12/2008: My music collection is now on my hard drive.

Storing documents

I don't want things to be stored in a hierarchy or a taxonomy or anything ending in the letter 'y'. I want it all in a pile, just like I do with paper. Then I want to be able search through the pile. The difference is that I want to be able to do the searching MUCH MUCH FASTER.

So I want at least the metadata stored in a SQL database. The docs themselves can be stored in BLOBS or files. I don't care as long as I can still get at them in 10 or 20 years when I need them. I think files would be best. Document File Formats

Indexing documents

Fine, so the metadata can live in a database. I need to decide what the best way to access the data is.

This journey started when I ran across the announcement for a release of phpMyArchive so I better check it out. This project has some very good ideas about making data storage and indexing.

I remember evaluating Swish-E so I will look at it again too. Swish-E indexes text docs.

I also played with MnoGoSearch for a while. It indexes documents and puts the index into a database. For articles it could go like this: Scan article. Attach title to it. OCR it. Then run indexer on it.