Yes, but you can have as many active sessions as you need instead of just two.
Incognito/InPrivate/whet-ever-other-browsers-call-the-feature tabs/windows share the same session so that gives you at most two active sessions: normal and incognito.
The "new session" option in IE can have many more. For a lot of workflows two is enough, but sometimes I want something like "a couple of distinct base users, a manager, and an admin" for testing more complex user workflows.
I don't know how edge does incognito but in other browsers, incognito kills the tab history, so if you accidentally close a tab you can't trivially restore it with ctrl+shift+t or whatever.
Containers keep the cookies and other data, yes, so you don't need to re-login every time you start the browser.