Not at all. I've written/cleaned-up a few dozen Backbone apps. They work great when they're small and are generally a mess once they scale. An example: how do you use a datepicker component in your pure state->DOM system and guarantee setup/cleanup of DOM event listeners?
As for when you'd want to read from the DOM in a well architected app, the easy example is drag and drop. Unless your drop targets have fixed sizes or your needs are simple enough to purely get away with mouseenter/mouseleave events then you need to query the DOM for node sizes. Other situations I've run into are text breaking and rich text editing.
Yes, if you need to add a behavior to the UI which "can be dropped anywhere", such as drag-drop, or rich text editing, then as I said, you're going to have to read from the DOM and otherwise try to adapt "to any reasonable environment" with varying degrees of success.
So what? This is orthogonal to the problem of rendering the basic UI as a pure function of the state.
You can store the state of of components inside components, there are many ways you can do it. React does it. We've implemented it in our framework: http://platform.qbix.com/guide/tools
As for how you would remove event listeners when a component is removed, that needs be coded of course. It would be nice for the developer not to have to write all that repetitive code. Of course, that also means you'll have to know when a node is removed from the DOM, and in browsers that don't support mutation listeners you'd be hard pressed to know that unless the user uses YOUR standard methods to control the DOM. All around it's a better solution (if you control the development of the app) to architect the app so that the model goes through YOUR framework to update the view.
As for when you'd want to read from the DOM in a well architected app, the easy example is drag and drop. Unless your drop targets have fixed sizes or your needs are simple enough to purely get away with mouseenter/mouseleave events then you need to query the DOM for node sizes. Other situations I've run into are text breaking and rich text editing.