Assuming you mean Simulink rather than Matlab its not quite a direct alternative. You can use Modelica and Simulink for modeling the same type of problems but they use different modeling approaches. Simulink uses cuasal modeling, where signals flow via connections between individual blocks, transmitting values of individual variables from the output of one block to inputs of another block. Modelica uses acuasal modeling - in Modelica, individual components of the model describe the equations directly and by interconnecting individual components, the systems of equations become connected.

If you are looking for a direct open source causal modeling alternative to Simulink, then SciCos (www.scicos.org) is worth a look. I would recommed playing with both Open Modelica and SciCos to see which is the best fit for you.