Neural networks

All neural network models (classes) should inherit from ten::nn::net class, defined in ten/neural/net.hxx.

Learnable parameters should be added to the network by calling add_param(name, value).

Activation functions

Supported activation functions

Description	Activation function
Relu y = max(0, x)	ten::nn::relu
Leaky relu	ten::nn::leaky_relu
Sigmoid	ten::nn::sigmoid

Layers

Supported layers

Description	Layer
Dense/Linear layer (y = w*x + b)	ten::nn::dense<T=float>
Batched dense/linear layer (Y = X*W + B)	ten::nn::batched_dense<T=float>

Optimizers

Optimizers are defined in ten/optim.hxx header file.

The following optimizers are supported.

Supported optimizers

Description	Optimizer
Stochastic Gradient Descent (SGD)	ten::optim::sgd<T=float>

TODO: Activation functions

Activation functions to implement

Description	Activation function
Binary step	ten::nn::binary_step
Hyperbolic tangent	ten::nn::tanh
Parametric rectified linear unit (Prelu)	ten::nn::prelu(alpha)
Gaussian	ten::nn::gaussian
Sinusoid	ten::nn::sinusoid
Softmax	ten::nn::softmax

TODO: Layers

Layers to implement

Description	Layer
Convolution	ten::nn::cnn<T=float>
Recurrent neural network	ten::nn::rnn<T=float>
LSTM	ten::nn::lstm<T=float>
GRU	ten::nn::gru<T=float>

TODO: Optimizers

• The Stochastic gradient descent implementation should be improved, by adding momentum, dampening, and nesterov.

• Here’s a list of optimizers to implement

Optimizers to implement

Description	Optimizer
Adam	ten::optim::adam<T=float>

TODO: Others

Current implementation doesn’t support layers without bias, this should be solved.