Basic Deep Learning Framework

Table of Contents

• Introduction
• Dependencies
• Components
  • Layers
  • Normalization
  • Initializers
  • Activation Functions
  • Loss Functions
    • Classification
    • Regression
  • Regularization Techniques
  • Optimizers
• Code Examples
• Implementation notes
• ToDo

Introduction

A from-scratch basic deep learning framework implementation in Python with NumPy, with syntax similar to TensorFlow and implementation similar to the lectures given by professor Andrew Ng in the course The Deep Learning Specialization.

Dependencies

• Numpy for computations.
• Dill for saving and loading deep learning models.

Components

Layers

• Dense

Normalization

• Batch Normalization
• Layer Normalization

Initializers

• Random Normal
• Random Uniform
• He Normal
• He Uniform
• Xavier Normal
• Xavier Uniform

Activation Functions

• Rectified Linear Unit
• Leaky Rectified Linear Unit
• Tangent Hyperbolic
• Sigmoid
• Exponential Linear Unit

Loss Functions

Classification

• Binary Cross-Entropy
• Categorical Cross-Entropy

Regression

• Mean Squared Error
• Mean Absolute Error

Regularization Techniques

• Dropout (Inverted Dropout)

Optimizers

• Stochastic Gradient Descent
• Momentum with Gradient Descent
• RMSProp
• Adaptive Moment Estimation

Code Examples

Checkout the code examples in the examples directory.

Implementation notes

• inputs must have the shape (# of samples, # of features).

ToDo

• Implement Callbacks class.
  • Solve the problem of passing data