how to choose the optimumize values of the training option

Dear all I have a question, I am using Transfer Learning of the pretraining Deep learning networks I would like to know how to choose the optimum values of the training options (Solver, initial learning rate, max epoch, miniBatchsize etc) should I keep trying different values or choosing the default values, please