I've tried doing things like this and found that it's often not totally reliable. I've had a hard time getting a consistent output and will randomly get variations I did not expect. I've found it's useful if you're trying to clean up data as a manual task, but not trying to automate a process.
Yes, there is definitely a human-in-the-loop element here.
It would be great if you could share an example of the inconsistent output problem -- we also faced it. GPT-4 was much better than GPT-3.5 in output quality.
