How to Build a Tiny Language Model (TLM) in Ruby: A Step-by-Step Guide

In this article, we will walk through how to create a very simple language model using Ruby. While true Large Language Models (LLMs) require enormous amounts of data and computational resources, we can create a toy model that demonstrates many of the core concepts behind language modeling. In our example, we will build a basic Markov Chain model that “learns” from input text and then generates new text based on the patterns it observed.

Note: This tutorial is meant for educational purposes and illustrates a simplified approach to language modeling. It is not a substitute for modern deep learning LLMs like GPT-4 but rather an introduction to the underlying ideas.

Table of Contents

1. Understanding the Basics of Language Models
2. Setting Up Your Ruby Environment
3. Data Collection and Preprocessing
4. Building the Markov Chain Model
5. Training the Model
6. Generating and Testing Text
7. Conclusion

Understanding the Basics of Language Models

A Language Model is a system that assigns probabilities to sequences of words. At its core, it is designed to capture the statistical structure of language by learning the likelihood of a particular sequence occurring in a given context. This means that the model analyzes large bodies of text to understand how words typically follow one another, thereby allowing it to predict what word or phrase might come next in a sequence. Such capabilities are central not only to tasks like text generation and auto-completion but also to a variety of natural language processing (NLP) applications, including translation, summarization, and sentiment analysis.

Modern large-scale language models (LLMs) such as GPT-4 use deep learning techniques and massive datasets to capture complex patterns in language. They operate by processing input text through numerous layers of artificial neurons, enabling them to understand and generate human-like text with remarkable fluency. However, behind these sophisticated systems lies the same fundamental idea: understanding and predicting sequences of words based on learned probabilities.

One of the simplest methods to model language is through a Markov Chain. A Markov Chain is a statistical model that operates on the assumption that the probability of a word occurring depends only on a limited set of preceding words, rather than the entire history of the text. This concept is known as the Markov property. In practical terms, the model assumes that the next word in a sequence can be predicted solely by looking at the most recent word(s) — a simplification that makes the problem computationally more tractable while still capturing useful patterns in the data.

In a Markov Chain-based language model:

• The future state (next word) depends only on the current state (previous words): This means that once we know the last few words (determined by the model's order), we have enough context to predict what might come next. The entire history of the conversation or text does not need to be considered, which reduces complexity.
• We build a probability distribution of what word comes next given the preceding word(s): As the model is trained on a corpus of text, it learns the likelihood of various words following a given sequence. This probability distribution is then used during the generation phase to select the next word in the sequence, typically using a random sampling process that respects the learned probabilities.

In our implementation, we’ll use a configurable "order" to determine how many previous words should be considered when making predictions. A higher order provides more context, potentially resulting in more coherent and contextually relevant text, as the model has more information about what came before. Conversely, a lower order introduces more randomness and can lead to more creative, albeit less predictable, sequences of words. This trade-off between coherence and creativity is a central consideration in language modeling.

By understanding these basic principles, we can appreciate both the simplicity of Markov Chain models and the foundational ideas that underpin more complex neural language models. This extended view not only helps in grasping the statistical mechanics behind language prediction but also lays the groundwork for experimenting with more advanced techniques in natural language processing.

Setting Up Your Ruby Environment

Before getting started, make sure you have Ruby installed on your system. You can check your Ruby version by running:

ruby -v

If Ruby is not installed, you can download it from ruby-lang.org.

For our project, you may want to create a dedicated directory and file:

mkdir tiny_llm
cd tiny_llm
touch llm.rb

Now you are ready to write your Ruby code.

Data Collection and Preprocessing

Collecting Training Data

For a language model, you need a text corpus. You can use any text file for training. For our simple example, you might use a small sample of text, for instance:

sample_text = <<~TEXT
  Once upon a time in a land far, far away, there was a small village.
  In this village, everyone knew each other, and tales of wonder were told by the elders.
  The wind whispered secrets through the trees and carried the scent of adventure.
TEXT

Preprocessing the Data

Before training, it’s useful to preprocess the text:

• Tokenization: Split text into words.
• Normalization: Optionally convert text to lowercase, remove punctuation, etc.

For our purposes, Ruby’s String#split method works well enough for tokenization.

Building the Markov Chain Model

We’ll create a Ruby class named MarkovChain to encapsulate the model’s behavior. The class will include:

• An initializer to set the order (number of preceding words) for the chain.
• A train method that builds the chain from input text.
• A generate method that produces new text by sampling from the chain.

Below is the complete code for the model:

class MarkovChain
  def initialize(order = 2)
    @order = order
    # The chain is a hash that maps a sequence of words (key) to an array of possible next words.
    @chain = Hash.new { |hash, key| hash[key] = [] }
  end

  # Train the model using the provided text.
  def train(text)
    # Optionally normalize the text (e.g., downcase)
    processed_text = text.downcase.strip
    words = processed_text.split

    # Iterate over the words using sliding window technique.
    words.each_cons(@order + 1) do |words_group|
      key = words_group[0...@order].join(" ")
      next_word = words_group.last
      @chain[key] << next_word
    end
  end

  # Generate new text using the Markov chain.
  def generate(max_words = 50, seed = nil)
    # Choose a random seed from the available keys if none is provided or if the seed is invalid.
    if seed.nil? || [email protected]?(seed)
      seed = @chain.keys.sample
    end

    generated = seed.split
    while generated.size < max_words
      # Form the key from the last 'order' words.
      key = generated.last(@order).join(" ")
      possible_next_words = @chain[key]
      break if possible_next_words.nil? || possible_next_words.empty?

      # Randomly choose the next word from the possibilities.
      next_word = possible_next_words.sample
      generated << next_word
    end
    generated.join(" ")
  end
end

Explanation of the Code

• 
  

  **Initialization:**The constructor initialize sets the order (default is 2) and creates an empty hash for our chain. The hash is given a default block so that every new key starts as an empty array.

• 
  

  **Training the Model:**The train method takes a string of text, normalizes it, and splits it into words. Using each_cons, it creates consecutive groups of words of length order + 1. The first order words serve as the key, and the last word is appended to the array of possible continuations for that key.

• 
  

  **Generating Text:**The generate method starts with a seed key. If none is provided, a random key is chosen. It then iteratively builds a sequence by looking up the last order words and sampling the next word until the maximum word count is reached.

Training the Model

Now that we have our MarkovChain class, let’s train it on some text data.

# Sample text data for training
sample_text = <<~TEXT
  Once upon a time in a land far, far away, there was a small village.
  In this village, everyone knew each other, and tales of wonder were told by the elders.
  The wind whispered secrets through the trees and carried the scent of adventure.
TEXT

# Create a new MarkovChain instance with order 2
model = MarkovChain.new(2)
model.train(sample_text)

puts "Training complete!"

When you run the above code (for example, by saving it in llm.rb and executing ruby llm.rb), the model will be trained using the provided sample text.

Generating and Testing Text

Once the model is trained, you can generate new text. Let’s add some code to generate and print a sample text:

# Generate new text using the trained model.
generated_text = model.generate(50)
puts "Generated Text:"
puts generated_text

You can also try providing a seed for text generation. For example, if you know one of the keys in the model (like "once upon"), you can do:

seed = "once upon"
generated_text_with_seed = model.generate(50, seed)
puts "\nGenerated Text with seed '#{seed}':"
puts generated_text_with_seed

By experimenting with different seeds and parameters (like the order and maximum number of words), you can see how the output varies.

Complete Example: Training and Testing a Tiny LLM

Here is the complete Ruby script combining all the above steps:

#!/usr/bin/env ruby
# llm.rb

# Define the MarkovChain class
class MarkovChain
  def initialize(order = 2)
    @order = order
    @chain = Hash.new { |hash, key| hash[key] = [] }
  end

  def train(text)
    processed_text = text.downcase.strip
    words = processed_text.split
    words.each_cons(@order + 1) do |words_group|
      key = words_group[0...@order].join(" ")
      next_word = words_group.last
      @chain[key] << next_word
    end
  end

  def generate(max_words = 50, seed = nil)
    if seed.nil? || [email protected]?(seed)
      seed = @chain.keys.sample
    end

    generated = seed.split
    while generated.size < max_words
      key = generated.last(@order).join(" ")
      possible_next_words = @chain[key]
      break if possible_next_words.nil? || possible_next_words.empty?
      next_word = possible_next_words.sample
      generated << next_word
    end
    generated.join(" ")
  end
end

# Sample text data for training
sample_text = <<~TEXT
  Once upon a time in a land far, far away, there was a small village.
  In this village, everyone knew each other, and tales of wonder were told by the elders.
  The wind whispered secrets through the trees and carried the scent of adventure.
TEXT

# Create and train the model
model = MarkovChain.new(2)
model.train(sample_text)
puts "Training complete!"

# Generate text without a seed
generated_text = model.generate(50)
puts "\nGenerated Text:"
puts generated_text

# Generate text with a specific seed
seed = "once upon"
generated_text_with_seed = model.generate(50, seed)
puts "\nGenerated Text with seed '#{seed}':"
puts generated_text_with_seed

Running the Script

1. Save the script as llm.rb.
2. Open your terminal and navigate to the directory containing llm.rb.
3. Run the script using:

ruby llm.rb

You should see output indicating that the model has been trained and then two examples of generated text.

Benchmark

The following table summarizes some benchmark metrics for different versions of our Tiny LLM implementations. Each metric is explained below:

• Model: The name or version identifier of the language model.
• Order: The number of previous words used in the Markov Chain to predict the next word. A higher order generally means more context is used, potentially increasing coherence.
• Training Time (ms): The approximate time taken to train the model on the provided text data, measured in milliseconds.
• Generation Time (ms): The time required to generate a sample text output, measured in milliseconds.
• Memory Usage (MB): The amount of memory consumed by the model during training and generation.
• Coherence Rating: A subjective rating (out of 5) indicating how coherent or contextually appropriate the generated text is.

Below is the markdown table with the benchmark data:

These benchmarks provide a quick overview of the trade-offs between different model configurations. As the order increases, the model tends to take slightly longer to train and generate text, and it uses more memory. However, these increases in resource consumption are often accompanied by improvements in the coherence of the generated text.

Conclusion

In this tutorial, we demonstrated how to create a very simple language model using Ruby. By leveraging the Markov Chain technique, we built a system that:

• Trains on sample text by learning word transitions.
• Generates new text based on learned patterns.

While this toy model is a far cry from production-level LLMs, it serves as a stepping stone for understanding how language models work at a fundamental level. You can expand on this idea by incorporating more advanced techniques, handling punctuation better, or even integrating Ruby with machine learning libraries for more sophisticated models.

Happy coding!