Why does the model fail to learn this game of filling up integers

Hi all,

My question: Why does my model fail to learn to play this game of just producing an array of unique elements from 1 to 5 from a partially filled array?

===

I am trying to train a model to perform this task:

Given a fixed array of 5 elements consisting of at most ONE of each element from (1, 2, 3, 4, 5) and ONE OR MORE (0), replace the 0s with appropriate values so that the final array has exactly ONE of each (1, 2, 3, 4, 5).

So, here is how it should be played:

• [1, 2, 3, 4, 0] => [1, 2, 3, 4, 5]
• [4, 3, 0, 5, 1] => [4, 3, 2, 5, 1]
• [0, 3, 5, 4, 0] => [2, 3, 5, 4, 1] OR [1, 3, 5, 4, 2]
• ...

This is not a complicated game (in human sense), but I want to see if a model can identify the rules (replace 0s with 1 to 5, so that final array has only exactly one element from (1, 2, 3, 4, 5)).

The way I did this is:

• Generate N examples of combinations configurations with elements of [1, 2, 3, 4, 5] as answers, and randomly replace some of the elements as 0s.
• For instance, one training example is [(0, 3, 5, 4, 0), (2, 3, 5, 4, 1)].
• There can be multiple same input mapping to different output, i.e. [(0, 3, 5, 4, 0), (2, 3, 5, 4, 1)] and [(0, 3, 5, 4, 0), (1, 3, 5, 4, 2)] can be both present as two separate training instances.
• Separate the training data set 10 fold, shuffled, and train using a RandomForestClassifier from Scikit-Learn.
• A correct output is defined as the final configuration array has exactly ONE element from (1, 2, 3, 4, 5). So (2, 4, 4, 5, 1) is not valid.

===

Surprisingly, using 1000, 10000, 50000, and even 100000 training examples still results in the model only getting ~70% of the test cases right - meaning the model did not learn how to play the game with increasing training examples.

One thing I was thinking is that RandomForestClassifier is just not used for this type of problem, called structured machine learning, where the output is not a single category or a real-valued output, but a vector of output.

More questions:

• Why does the model fail to learn this game?
• Is this the right way to model this problem?
• Is the data not enough to learn this task? But increasing data from 1000 to 100000 does not seem to help at all.

Thank you!