Doing some thinking about personal data and where it is stored, especially since starting my CLI productivity tools exploration. This is the kind of thing companies take very seriously and you should too. It’s important to keep control of your own data and to store it in such a way as to have it still available and viable many years in the future.

I have lost data because software vendors change file formats and don’t maintain backward compatibility. I have lost data for unknown reasons when storing files in the cloud. I have not had any catastrophic hard drive failures, but that’s no reason not to have good backups.

With all that in mind, I’ve put together some rules of thumb about how I will store data and what file formats to use. This is helping me review to review the information I’m storing and to keep it in good shape for the future.

Guidelines about data

  • Text is the closest thing to a universal file format. Use text when possible.
    • Text should have the greatest longevity and be available in the future.
  • Sometimes a database is the correct choice over text. In that case, prefer SQLite.
  • Other text formats such as CSV files or JSON files can also be good in the right circumstances.
  • Use Markdown for bringing formatting into text files. Restructured text would be my 2nd choice.
  • Use conversion tools like pandoc to convert text to other formats. An example
  • Leverage fzf, grep or other fuzzy finding tools to quickly navigate among text files.
  • I don’t want my data to be controlled by someone else.
  • My data should be stored in my infrastructure.
  • Use cloud storage only for backup or remote access. Prefer hosting my own cloud.
  • Avoid lock-in to a specific vendor or software product due to proprietary file formats.
  • Own your own media when possible.
  • Keep good backups. Renew storage media periodically.

Long Term Data

Identify types of data that should be maintained.

  • Notes (Markdown preferred) - able to link to other files
  • Calendar and scheduling
  • Tasks, TO-DO lists
  • Contact information
  • Media (eBooks, music, movies, photos)
  • Recipe / cookbook database

Binary Data Files

It isn’t always possible to have text files. Need other kinds of files for different kinds of data. These include things like:

  • Music files: .mp3, .flac, .ogg
  • Video files: .mp4
  • Pictures: .jpg, .webm, .png
  • Documents: .pdf (can also be generated from Markdown)
  • eBook: .epub
  • Compression: .zip

Strategy

I really like Thunderbird. It manages multiple email boxes easily. It also has calendaring and contact management support. The solutions will need to allow Thunderbird to access the data.

The other big consideration is to pull my data away from the cloud (Primarily Google in my case).

Notes

Right now I use Obsidian for my note taking needs. I am also playing around with nb notebooks. Both of these tools use Markdown files for the data. This makes it easy to migrate the data in the future if the need arises.

Calendars

I am looking for an improved method of keeping my calendars. I’ve used Google Calendar for a very long time. Migrating away from that hasn’t been easy. I did have a Nextcloud instance running, but experienced corruption when I upgraded. It was also overkill for my current needs. vdirsyncer worked well for synchronizing between Google and local data.

Trying out some self-hosted tools to see which works the best. Currently exploring Radicale. This looks promising.

Tasks

Taskwarrior is my primary task tool. I don’t currently have a way to sync it between my devices, but that hasn’t been a problem so far. Ultimately, that will be the goal.

Contacts

Abook and vdirsyncer worked well with Google. More work needed here.

Media

This is in the best shape. I’m using Jellyfin for movies, videos, pictures, and music. eBooks are managed with Calibre and I use Calibre Web to self host them on my network. Syncthing keeps everything copied across devices as needed.

Recipe Database

I’m self-hosing Mealie for my recipe database. I love it. It is a great recipe database that even my wife uses. I also have copies of some of the recipes in my Obsidian vault.