Also check out gitit (http://gitit.net), written by the same author as pandoc. It's a git-backed wiki that uses pandoc to let you write in a variety of formats. Markdown + latex for math works well for taking notes in a wiki format. The content is stored as flat files in the repo that you can just export as pdfs using pandoc if you wanted to.

After a few weeks of trying to get Github's gollum gem to do what I want, this might be what I've been looking for.

Yep.I use pandoc to convert all the notes I take to a variety of formats for all my different devices and needs. For example Pdf for highly consistent and pretty viewing on a tablet, html as an all around backup, and others as needed. Pandoc is fantastic!