I have used both Octave and Scilab as a substitute for MATLAB. Scilab is a much more polished system, but it has multiple gotchas if you are moving back and forth between it and MATLAB. Octave aims for full MATLAB compatibility, but I found the system pretty rough all around.