I think current LLMs would be completely lost with the task "scan these documents for information that could be risky for our case": they would either produce tons of false positives, overlook actual risks, or both.

I can personally attest that companies are already using those LLM-based workflows for those exact purposes successfully, performing better than humans, which often isn't a super high bar.

At least you can reasonably use them for translation and in a second pass summarize or just have it detect if a document contains specific keywords / phrases.

Yeah but this is where a human is indispensable.

Suppose the source language has words which could be translated into multiple English words -- look at all the power banks which are advertised as charging treasure for example. One can see how bank vs treasure are close, after all. Further, even in English multiple phrases could be used "don't preserve" "bin it" "throw it away" and countless others. Even worse, it could be a company specific phrase which would only stand out as odd "apply procedure 66 to it".

If given a little thought this is exactly the kind of task where a native speaker would shine and LLM might just miss or if given a wide enough net produce a million false positive.

Sure! But in any case, it's worth a shot to preprocess document dumps with AI first - if it fails to spot anything obvious, you can still go and have humans sift through the pile manually, whereas if it does spot something humans can immediately zero in instead of wasting their time.

hopeless AI cultists everywhere

as I said above: I already asked dang to delete this account, I do not want to be here anymore

AI is a tool, nothing more, nothing less. It is smart to use it when it makes sense, it is dumb to shoehorn it into something it by definition cannot do.

Just tested out the treasure idea and seems to work fine in ChatGPT. I suspect if given context at the task at hand that the LLM would provide a pretty decent first pass.

Y'all are hopeless. I already asked dang to delete this account.

What a weird response. Just providing a counterpoint to your idea that I don't believe is correct. I hope dang does not delete your account because I am not sure whats going on here and I hope you find the help/peace you need.

I listed like half a dozen things that could go wrong and the only reply is "look this particular one in this singular case the bullshit is accidentally correct".

Hopeless, as I said.

Ouch, sorry you are in distress but let me repeat and help flesh it out for you. You would be surprised that when given the context at hand, the level of confusion would be a lot less than your constrained world model envisions. I have been pleasantly surprised with LLMs ability to translate, including legal documents. Should it be the only step in the process? No. Like I originally pointed out, I think a LLM can serve quite well in initial first passes. Its quite naive to hand wave it away with some what-if scenarios and then just become a dismissive immature kid when someone disagrees with you. I can only imagine when you have a corpus to work from of existing legal documents especially for translations, that you get quite close to the spirit of whats written.

"Come delete my account dang". So weird....

You might be caught off guard then because its already happening successfully. There is a lot of overblown hype but systems already exist that do a pretty good job on first passes and other work that paralegals/juniors would be doing. I know this goes against the anti-LLM narrative here but don't be surprised as it happens.

In a discovery you don’t really have a choice.

On the other hand, perhaps the summer intern would have had an easier time finding something (and the defense more time to spot potential gaps in their arguments).

Deffo several startups and incumbents doing this for legal discovery nowadays..