Neural network for ultimate tic tac toe

Like you I have never programmed any of it. My first action would be to look for some explanation of how they work and how to program them at the Internet.

Neural networks good for taking complicated mathematical equations and building an approximation for it. You do this by supplying some inputs, see what outputs they give in some complicated system, then training the network with the given inputs and outputs. Eventually you can use the neural network to make rough approximations of what the complicated system will do given some inputs.

Turn based board games with no random chance can usually have an AI similar to chess. So I would look into existing chess AI implementations that use neural networks and see what you can glean from that.

This link looks promising
http://erikbern.com/2014/11/29/deep-learning-for-chess/

The natural place to use a neural network in that game is in writing an evaluation function. This NN will be a function that takes the current game situation as input and outputs a single real number, which is something like the expected result of the game, where +1 means X wins and -1 means O wins.

You can use a regular feed-forward neural network, which are the easiest type to understand. You can give it something like 81 inputs describing what's on the board, 9 describing what sub-boards are closed and 9 indicating what sub-board the next move should be at. Have a few hidden layers using rectified linear units. The last layer can use a tanh activation function to bring the result to the range [-1,+1].

Write an alpha-beta search that uses the NN plus a small amount of noise for evaluation. Initialize the NN using small weights, so the output for any input will be very close to 0 (so the alpha-beta search will be using a random evaluation function at this point). Now you can play games where the program plays itself. After you have a few thousand games, you can start training the neural network to predict the result of the game. You can alternate generation of games and training of the network for a few iterations (say 4).

I did something like this for Spanish checkers at the beginning of 2015, and it worked great.