April 14th, 2014

I have been using NOMAD global optimization software to tune some of Arasan's evaluation parameters. NOMAD can be linked with Arasan's C++ code and I have built a little tuning program that combines the two.

Instead of using games I am using a short-depth search and tuning for minimal prediction error over a large (3.8M position) training set of FEN positions and their associated game results (the output from the search is run through a sigmoid function and I take average squared diff between that function and the game result: 0, -1, or 1 for draw, loss or win).

Results so far are a bit mixed. I tried tuning a moderately large number of parameters (32) at once, and while the error objective did go down over the optimization run, the tuned parameter set did not perform better in games. However, some of the parameter changes did prove to be good in game testing. So this stuff is not just fire and forget, but it can be helpful.

I also did a little testing with SMAC but did not find that as effective.