Create Sample Data

I will create reproducible synthetic data to simulate a real-world dataset. This should allow you to have the same data on your compute as you follow along.

Notice how in this data I have centered and scaled the continuous covariates. This is a must for your neural network.

library(tidyverse)

set.seed(0)  # For reproducibility

n_samples <- 10000

# The bias term here is used to adjust the distribution of the binary outcome.
bias = -log(.001)  

# Generate data
data <- tibble(
  factor_1 = rnorm(n_samples),  # Normally distributed data
  factor_2 = rnorm(n_samples, mean = 5, sd = 2) + 0.5 * factor_1,  # Correlated with factor_1
  factor_3 = runif(n_samples, min = -1, max = 1),  # Uniformly distributed data
  factor_4 = rbinom(n_samples, size = 10, prob = 0.5),  # Binomially distributed data
  factor_5 = factor_3 * factor_4,  # Interaction term
  factor_6 = sample(0:1, n_samples, replace = TRUE)  # Binary data
) |> 
  # The outcome is a binary variable created using a logistic function (plogis)
  # of the factors, which simulates a complex relationship.
  mutate(outcome = rbinom(n(), 1, 
                          prob = plogis(factor_1 - 2 * factor_2 + factor_3 * factor_4 - 
                                          factor_5 + factor_6 + bias)
                          )
         ) |> 
  mutate(across(.cols = c(factor_4, factor_6, outcome), .fns = as.factor)) |> 
  mutate(across(.cols = is.numeric, .fns = ~(.-mean(.)/sd(.)))) |> 
  mutate(across(.cols = everything(), .fns = ~as.numeric(as.character(.))))

Outcome Percentage

We can see a fairly reasonable split of the outcome in this data.

Visualize Data

From the plot below, you can see the the relationship a select number of factors has with the outcome.

Define the Model Architecture

model <- keras_model_sequential() |>
  layer_dense(units = 100, activation = 'relu', input_shape = ncol(train_data) - 1) |>
  layer_dense(units = 100, activation = 'relu') |>
  layer_dense(units = 1, activation = 'sigmoid')

Model Architecture Details

• Hidden Layers: The model includes two hidden layers, each with 100 neurons and using the ‘relu’ (Rectified Linear Unit) activation function. Alternatives to ‘relu’ include ‘tanh’ and others, each having its own merits depending on the specific use case.

• Output Layer: The final dense layer makes the predictions. I use a ‘sigmoid’ activation function here, which is ideal for binary classification tasks as it returns values between 0 and 1, indicating the probability of the target class.

• Input Shape: The input_shape in the first layer is set to match the number of predictor columns in the data. This ensures that the model correctly processes the input features.

• Sequential Model: I’ve chosen a sequential model for its simplicity and effectiveness in stacking layers linearly. However, other options like functional API models exist, which allow for more complex architectures.

Selecting the Best Options

Choosing the optimal structure and functions for your neural network involves balancing several factors:

• Data Characteristics: Understanding the shape, size, and type of your data is fundamental. Different data types might require different network architectures.

• Modeling Objective: What you are trying to predict or classify directly influences your choice of architecture and activation functions.

• Complexity of Relationships: The complexity in your data’s relationships can dictate the depth and complexity of the neural network required.

• Experience and Experimentation: Selecting the best model is a cross between your own experience and experimentation. Running several models with different configurations and analyzing their performance is the best approach though the cost is the time it takes to run the models.

• Advanced Layer Types: For certain types of data, especially sequential data like time series or text, using advanced layer types such as Long Short-Term Memory (LSTM) (for time series) or Convolutional Neural Networks (CNNs) might be beneficial (text, imagery).

Compile the Model

The next block of code “compiles” the model and configures it for training. It sets up up the optimizer, loss function, and metrics:

model |> compile(
  optimizer = 'adam',
  loss = 'binary_crossentropy',
  metrics = c('accuracy', 
              metric_precision(), 
              metric_recall())
)

Understanding the Model Training Configuration

• x/y:
  • These represent the training and test sets, respectively.
  • The x (features) must be in a format acceptable by the model, typically a matrix or array in numerical format. The number of columns in x should match the number of input features the model expects.
  • The y (target) also needs to be in a numerical format, either as a vector for regression or binary classification, or as a one-hot encoded matrix for multi-class classification.
• Epochs (epochs = 20):
  • An epoch represents one complete pass through the entire training dataset. Here, the model will go through the dataset 20 times.
  • The more epochs, the longer the model has to learn the data. However, there’s a risk of overfitting to the training set. This can be mitigated with techniques such as regularization, dropout between layers, and model callbacks, which are not included in this example.
• Batch Size (batch_size = 100):
  • This defines the number of samples that will be propagated through the network in one pass. A smaller batch size means less memory usage but potentially less accurate estimation of gradients. I recommended you use as large of a batch size as your computer can handle and you have patience for.
• Validation Split (validation_split = 0.3):
  • This parameter reserves a portion of the training data (30% in this case) for validation. The model’s performance on this validation set provides a good indication of its generalization to unseen data.
• Verbose (verbose = FALSE):
  • This controls the level of verbosity of the training process output. It is set to FALSE here to suppress the progress messages for each epoch. This is done to avoid cluttering the description, but I recommended you set it to TRUE to follow its progress.

The fit function tunes the model to your data by adjusting the weights. The history object captures details about the training process, which can be used for further analysis and understanding of the model’s performance over time.