Sentdiff: Diff for Writing

gourneau · on March 2, 2009

Howdy, thanks for posting this.

It would be awesome to visualize wikipedia edits overtime. I don't really care about what the text says, just how blocks of it change over time. I am after the ascetics present in the ever flowing change of data. I think your script might be a good starting point. Think of the videos of a flowers growing, that compress months into a few seconds. Do something similar with wikipedia edits.

keenerd · on March 2, 2009

I've been doing something similar with my blog, except it is currently for people who do care what the text says. The diff algo is tricky, I should have built up a larger corpus of material before designing.

It looks at paragraphs, sentences, sub-sentence structures, words. It even draws little sparkgraph-ish diagrams. It is not really that long (250 lines by wc) but it has been a huge time sink for tweaking.

For an example of some heavy editing: http://kmkeen.com/inabow/2009-01-07-11-22-00.html

ashr · on March 2, 2009

I was hoping to see a compact implementation of diff algorithm. However, the script seems to be relying on using the 'diff' utility already present. Not a bad thing, but I was expecting to see something else.

jackowayed · on March 2, 2009

I went with a don't-reinvent-the-wheel approach. Anything I did would have at least doubled the time to write the script and probably yielded a diff half as good.

dhotson · on March 2, 2009

This might be worth a look for a decent diff algorithm:

http://code.google.com/p/google-diff-match-patch/

akkartik · on March 2, 2009

Also this: http://bramcohen.livejournal.com/37690.html

jackowayed · on March 2, 2009

I wrote this as well as submitting it.

Any suggestions, thoughts, etc. would be greatly appreciated.

ntoshev · on March 2, 2009

I would approach this problem differently: tokenize the text and diff the stream of tokens (as opposed to a stream of characters).

akkartik · on March 2, 2009

Oh my eyes.. :) I suggest replacing main with

    def file_with_split_sentences(old)
      contents = File.read(old).split_keep_after(/[\.!\?] *[A-Z\n]/, 2)
      begin newname = "#{old}-sdiff-#{random_chars 5}"; end while File.exists?(newname)
      File.open(newname, "w") { |file| file.write contents.join("\n")}
      newname
    end

    fnames = ARGV[-2..-1].collect{|f| file_with_split_sentences f.chomp}
    system "diff #{ARGV[0...-2].join " "} #{fnames.join " "}"
    "rm #{fnames.join " "}"

socmoth · on March 2, 2009

i like this and needed it about a week ago. i eventually had to write my own script.

i also found dwdiff, which you may like because it is very similar and very unixy http://www.linux.com/feature/114176

boucher · on March 2, 2009

Why sentences and not words?

anewaccountname · on March 2, 2009

Maybe he didn't design this for phonologists? I doubt anyone else has too much use for intra-word diffs.

jrockway · on March 2, 2009

I think it would be useful. Consider this example.

The original sentence is: "He went to the stoar." Person A sends this to person B for review. While Person B is reviewing this, Person A changes the sentence to "Bill went to the stoar.". Then, Person B sends back the spelling correction; "He went to the store.".

If the sentence translated to one line for `diff` to work on, this would be a merge conflict. If each word was its own line, this would merge cleanly.

keenerd · on March 2, 2009

You've just described dwdiff. Not the smartest diff algo, though its strong point is producing diffs of prose that are human readable. I used it as the core of a wiki with 'perfect' collision resolution. Two people could edit at the same time, and the 2nd to hit save would have their work merged in with dwdiff instead of having their work automatically discarded.

jackowayed · on March 2, 2009

hmmm

It might sort of drown you (a 15-word sentence totally changed to a different 15-word sentence would take up 30 lines. My default terminal size holds 36 lines.

But really because I never thought of it. Might give it a try and see if it's better.

wanderingmarker · on March 2, 2009

For Emacs users: ediff highlights changed words. I've used it to track changes in text.

albertcardona · on March 2, 2009

I could use this as part of git itself for comparing latex document revisions. The current line-oriented diff has all the problems that sentdiff tries to solve.

crux_ · on March 2, 2009

http://www.robmar.net/TexDiff/

Handy!

jackowayed · on March 2, 2009

yeah, I was thinking it would be cool as an option for git diffs or something for when you have text files under git.

Never really looked at the git source code, so I'm not sure how easy it would be to do.

bbb · on March 2, 2009

Git has something very much like that already built in. Try

git diff --color-words

albertcardona · on March 2, 2009

I know about --color-words, it's reasonable. But not optimal. Sentence diff plus color words would be great.

(and a pony--I know, why not implement it myself?)