Matlab is good at prototyping what one might actually build in another language. Its basically a regular python like language that has lots of built-ins for doing stuff with vectors and matrices and plotting things nicely.

Mathematica on the other hand is more of a tool for doing things symbolically (x's & y's). You can integrate, differentiate, find roots, etc all symbolically. It does have tools for doing numerical integrations and such, but they are much more of a black box than matlab's and are very hard to port out of mathematica. Some math folks don't like mathematica because of its black-box attitude, which is understandable. I use it sometimes for calculus.

Don't do linear algebra with mathematica. Matlab is waaay better at it.

I used Mathematica extensively for my physics Ph.D. at Caltech (the same degree and school as Stephen Wolfram---though he finished in two years at the age of 20, instead of six at 29...) I've only used MATLAB sparingly, so I'm can't offer much of a comparison, but Mathematica is excellent at linear algebra: http://reference.wolfram.com/mathematica/guide/MatricesAndLi...

for plotting, matlab is a bit 'lower level' where you map a function over a range and then plot the list of return values

for linear algebra, matlab is more succinct, it has "'" transpose and "^-1" for inverse, so you can type x = A'(AA')^-1*b

You can do that in Mathematica, too: http://documents.wolfram.com/mathematica/functions/Transpose

Mathematica has an extensive set of typesetting shortcuts. You just need to know where to look.