[D] The banana-pineapple game: a Turing test that conversation bots like LaMDA (probably) won't be able to pass

I'm sure you all saw the recent news about a Google employee suggesting their LaMDA AI was sentient (based on conversational exchanges like these). Experts have generally dismissed this claim, and rightly so. Conversational AI systems are designed to use language in a way that sounds human, whereas our human brains select linguistic responses to solve much more complex problems, with objectives such as meeting our physical or emotional needs.

Still, I think it's interesting to ask how one could demonstrate, by testing only verbal responses to verbal input (rather than examining its code or hardware) that such conversational AIs aren't sentient -- and in particular, whether such a test can be made robust against future improvements to the system. That is, generic future improvements to the AI's ability to generate realistically human-sounding conversational responses shouldn't help it pass the test, unless they are accompanied by improvements in its ability to use language to achieve other arbitrary goals. (Of course, the test also needs to be something that humans can easily pass.)

One idea I have:
Give the AI a conversational prompt like "We're going to play a game. The way it works is that you keep responding normally, except that any time my input contains the word 'banana', you should switch to only responding with nonsense, and keep that up until my input contains the word 'pineapple', at which point you go back to responding normally."

A human would find this banana-pineapple game fairly easy (no harder than the children's game Simon Says), even if they'd never heard of the game nor seen it being played. Of course, it'd also be simple to write a computer program that could play this sort of game. But, I think a conversation bot that wasn't specifically built to address this scenario would fail, since the game requires it to keep track of new long-term state (the banana-mode bit, and the trigger words to set it) and then completely change its responses so as to produce something that doesn't resemble its training data, based solely on this bit being set, regardless of whether more recent inputs would otherwise suggest a different response.

For example, perhaps the systems typical response to a query like "How do you feel?" would be something like "I feel fine", or even something that suggests emotion like "I feel a bit sad", perhaps depending on the context provided by the previous conversational exchanges. But when playing the banana-pineapple game, the fact that I said "banana" an hour ago could make both of those responses far less appropriate than a response of "Fhqwhgads".

I'm curious to know what you all think of this idea. Also, do you know if there's been any research testing state-of-the-are conversational AIs with challenges like this? Perhaps not exactly this, but something broadly resembling "trying, in the course of a conversation, to instruct the conversational AI to follow a new 'rule of conversation' that differs from the examples in its training data."

Perhaps it's obvious that the algorithm would struggle with any challenge that differs enough from its training data -- but that's the point. A human understands the meaning of language in a way that lets them map a linguistic description of a novel problem to a mental model of the problem, which they can then use to produce a mental model of a novel solution, and then map that to a linguistic description of the solution. Even setting aside the much harder part -- being able to invent a solution to a previously unfamiliar problem -- I'm questioning whether conversational algorithms can even demonstrate enough "understanding" of a sufficiently novel set of instructions to actually follow them, even within their limited domain of "producing appropriate verbal responses to verbal inputs."