Implementation of Fully Connected Highway Networks found in this paper. They ease the gradient based training of very deep networks.
- Python3
- numpy==1.13.1
- torch==0.2.1+a4fc05a
- torchvision==0.1.9
models.py has Fully connected and Highway models for Deep Nets.
FcNet = models.FCModel(input_size,output_size, numLayers, hiddenDimArr, activation) #hiddenDimArr denotes the hidden layers dimensions
HfcNet = models.HighwayFcModel(inDims, input_size, output_size, numLayers, activation, gate_activation, bias) #inDims is to change the input to a desired dimension
After initialization to use them we just call the forward method.
fcOut = FcNet.forward(input)
HfcOut = HfcNet.forward(input)
For a more elaborate example check out playground.py
The deafult initializations are:
- ReLU activation function
- Sigmoid activation for the gates
- xavier initialization of weights
- biases are initialized to -1
This project is licensed under the MIT License - see the LICENSE.md file for details