This is a nice idea, but I think it needs work. It classified almost every tone that I spoke as 3rd tone. Granted, I'm not native, but I speak well enough that my 4th tone couldn't possibly be interpreted as a 3rd tone.

What's the training data? Is it trained on both male and female voices? Anyways, very nice idea - I hope you pursue it and I'm looking forward to seeing how it improves over time!