The thought experiment part aside, isn’t the premise a logical contradiction? You may choose to flip a coin about your choice and therefore the machine cannot possibly have a 90% accurate prediction of your guess.
In some variants of the problem, if the machine predicts you'll choose to delegate your choice to something random, it leaves the box empty just as it would if it predicts you'll choose to open both.
Newcomb's paradox obscures the same basic reason you can't have a Halting-Detection TM, only it covers it over with fuzzy terms and human complications.
If you recast the paradox as "There exists a machine scientists made that will determine if any other intelligence will either choose the box or not the choose the box. You are that other intelligence. Do you choose the box or not choose the box?" then you can see that the same principle holds; if you incorporate the logic of the predictor into your own intelligence, you can twist the original machine's logic back on to itself in exactly the same way the Halting problem does.
A subtlety to note about the halting problem is that while it is phrased in terms of that particular machine that can twist back on itself, it is itself a generalized proof of impossibility, and via Rice's theorem and lots of other work over the years it extends out into the impossibility of all sorts of other machines as well. The proof simply provides one machine that unambiguously can not be created, it is not limited to that one machine.
Similarly, while a human brain that encompasses the same logic as the box-prediction machine may have technical problems in the fuzzy real-world land, the fact that such a brain would be fundamentally unpredictable means that the prediction machine can't exist.
If you try to fix up the thought experiment by limiting the size of the predictor and the predictee in various ways, I suspect it isn't that hard to show that the predictor must be exponentially more complicated than the predictee in order to function, and something "exponentially larger" than the human brain doesn't really fit in the universe. And then if you try to escape by allowing arbitrarily large mathematical systems, you're back where I described above. If you try to bound how much larger the predictor has to be than the predictee, you are going to be encountering some really serious mathematical problems doing so (of the "busy beaver" mathematical sequence sorts).
Given that the problem intrinsically encompasses a delay between the scan and the decision, I can simply take an ad-hoc hash function of the experiences of the last day and now the predictor needs to have also had information of my entire last day as well, and a simulation sufficiently detailed to have predicted that, too. Even if it can predict the hash function I would use (itself no guarantee since that is also conditionalized on the intervening day, potentially), it can't predict the input going into it.
I think most people intuitively sense that the predictor can't really exist; I think people's intuitions are correct. It would require the predictor to have physically impossible amounts of additional resources vs. the predictee if you try to embed it in real space and time, and if you put it in math space it's nothing more and nothing less than the halting problem restated.
For anyone interested in a slightly different aspect of this:
A MIRI talk mentioned a hypothetical scenario when an AI gets duplicated. In this case the AI has a very high certainty about the state of its own duplicate. Also Nate argues that many everyday problems humans face are Newcomb-like. (And of course he talks about an AI that knows that humans know its source code.)
That's another way in which embedding it back into reality takes the force out of it... an algorithm that merely does a good job isn't a paradox, nobody has a problem believing that such a thing could exist. The average human might swear up and down the algorithm couldn't predict them, but certainly it won't have a hard time predicting everyone else pretty well. (A variant on "when I misbehave, it's for various good reasons, but when others misbehave it's because they're jerks.")
The paradox arises around the 100.00000000...% accurate algorithm.
> The paradox arises around the 100.00000000...% accurate algorithm.
Is it? As I said, my decision up-front is to flip a perfectly fair coin (or substitute in your favorite CSPRNG or TRNG). So your machine could have whatever claimed accuracy percentage, but I can't see how there's anything to predict beyond 50% and any machine that does so would have to be magical (i.e. outside the rules of science/mathematics).
I used that as a stand-in for my intuitive thought that such a machine's accuracy is time-dependent - the further out from the original scan you are, the more "random coin flips" your brain has experienced & thus you degrade very quickly to somewhere around 50% accuracy. You could try to mitigate that by having a super accurate model of how your brain works & trying to measure all the inputs it has encountered since, but ultimately the machine has to degrade to 50% at some point in time unless "magic". I guess you could do worse than 50%, but given this is a binary choice, any such result I think would mean you could just take the complementary result so the lower bound should always be 50%? Not sure about that last point.
Exactly, but conversely my point is that the machine giving you a 90% rating is evidence that you will not choose to flip a coin.
edit: Oh hang on, this is a machine, not Omega, so I can't say that Omega can just choose to not give you the offer if it can't reach 90%. Idk then. Probably it means that at least 9 out of 10 people don't throw coins.
