It isn't just an 80/20 problem. Imagine that you replace every linked list, binary tree, trie, etc., with a generic directed graph datatype. The resulting performance would be abysmal. The resulting notation would be horrid, too.

Here's our nice linked list:

  def last_element(ll):
      last = ll
      while ll is not None:
          last = ll
          ll = ll.next
      return last

And here's an implementation with generic graph notation:

  def last_element(g):
      for v, deg in g.out_degree:
          if deg == 0:
              return v
      return None

There are several problems with this; most importantly, there can be silent failures when g is not a linked list. But it also throws out a useful abstraction where a list is equivalent to a node, so I wrote a horrid implementation that takes O(n) regardless of the position in the list. And then comes all the baggage of representation, because you can't just represent a node with a pointer anymore.

When your data structure better reflects the, well, structure of your data, you can go faster and safer. There's a reason we teach undergrads about these specific datatypes and don't just sweep it all under a rug with "it's a graph!"

I think it's more like discarding a 20% solution that can't do the last 80%.