funnily enough, you can give the LLM the code and ask it if the function will halt, and for some cases of input, it is able to say that the program does/does not halt.

The halting problem is about being able to answer this question in full generality, though. Being able to answer the question for specific cases is already feasible and always was.