Explainable AI and Prompting a Black Box in the Era of Gen AI

In today’s world, for many people, conversing with AI has become as routine as discussing one’s coffee preferences with a barista (or it soon will be!). Yet, here lies an irony: the more we interact with AI, the more elusive our understanding of these conversations becomes.

This irony essentially represents a modern twist on the “black box” dilemma, which has perplexed the ML community for years.

The “black box” problem refers to the opaque decision-making processes of ML models (large language models (LLMs) including), where the rationale behind any given response is shrouded in complexity. Despite advances in technology, the inner workings of these models, governed by billions or soon trillions of parameters, remain largely inscrutable.

Their decision-making is a puzzle, complicated by nonlinear interactions that defy straightforward interpretation.

Prompting — the buzzing word of 2023 — doesn’t make it any better: we now have obscured layers of communication activated with every prompt. What we see — the prompt we type — is merely the surface.

Beneath lies a hidden dialogue, an augmented system prompt, which is a complex, coded conversation the model conducts with itself, away from our understanding. And who knows what a model whispers to itself?

So, if you were confused about prompting amid the avalanche of articles, blogs, and tutorials about it — you should be.

As Ethan Mollick’s research reveals, contrary to intuition, the most effective prompts involve imaginative scenarios, such as pretending to navigate a Star Trek episode or a political thriller, demonstrating that traditional logical or direct prompts may not always yield the best responses from AI.

But it also reveals that it’s not coherent and might change with a new version of the model. He mentions the futility of seeking a universal “magic phrase” for AI interaction, the effectiveness of specific prompting techniques like adding context, few-shot learning, and Chain of Thought, and the significant impact that prompts can have on AI performance.

But for me — and I’ve been using AI a lot — many times, the most straightforward prompts, or “magic words,” can be surprisingly effective.

How to explain it? A few years back, Explainable AI (XAI) was heralded as a solution to the “black box” issue, with entities like DARPA leading the charge (they created the XAI toolkit, which has not been updated since 2021). However, the buzz around XAI seems to have dimmed, overtaken by a broader focus on Responsible AI. Is Responsible AI the solution?

So, that’s what we end up with:

How do machines make decisions? — We don’t know!

How to talk (prompt) to them? — We don’t know as well!

But, please, keep shipping to us new, larger (though we will also take smaller) models! Why? — We don’t know! But we can’t stop.

Great Article Covering the Topic

An explanatory article about the concept of a black box:

🎁 Bonus: The Freshest Research Papers, Categorized for Your Convenience

Special Category: Definitely Worth Reading

• The Era of 1-bit LLMs: Discusses the development and advantages of 1-bit LLMs, promising significant cost reductions and efficiency improvements. Read the paper

• Beyond Language Models: Introduces bGPT, a model that simulates the digital world beyond traditional modalities, predicting and diagnosing algorithms or hardware behavior. Read the paper

Language Models in Specialized Domains

• ChatMusician: Integrates music understanding and generation capabilities into LLMs, demonstrating LLMs’ potential in music composition. Read the paper

• StructLM: Aims to bridge LLMs’ gap in interpreting structured data, enhancing their ability to ground knowledge in tables, graphs, and databases. Read the paper

• StarCoder2 and The Stack v2: Focuses on responsibly creating Code LLMs, contributing to advancements in coding benchmarks, and emphasizing model openness. Read the paper

• Video as the New Language for Real-World Decision Making: Discusses video generation’s potential as a unified interface for diverse tasks, outlining challenges and future directions. Read the paper

Enhancing and Merging Language Model Capabilities

• FUSECHAT: Proposes a method to fuse knowledge from multiple chat models, improving chat model performance through a novel merging technique. Read the paper

• Nemotron-4 15B Technical Report: Details a multilingual language model that showcases superior performance in coding tasks and multilingual capabilities. Read the paper

• Do Large Language Models Latently Perform Multi-Hop Reasoning?: Explores latent multi-hop reasoning in LLMs, revealing their inherent capabilities and limitations in complex reasoning tasks. Read the paper

Scaling and Efficiency in Model Training

• MegaScale: Discusses a system for training LLMs on over 10,000 GPUs, tackling efficiency and stability challenges in large-scale model training. Read the paper

• Towards Optimal Learning of Language Models: Proposes a theory for optimizing LLM learning, aiming for reduced training steps and improved performance. Read the paper

• Griffin: Introduces a model combining gated linear recurrences with local attention, offering an efficient alternative for language processing tasks. Read the paper

• When scaling meets LLM finetuning: Investigates the effects of scaling on fine-tuning LLMs, providing insights into data, model, and method impacts on bilingual tasks. Read the paper

Improving Robustness and Diversity in AI

• Rainbow Teaming: Generates diverse adversarial prompts to enhance LLM robustness, employing an open-ended search method for prompt discovery. Read the paper

• Priority Sampling of Large Language Models for Compilers: Proposes a deterministic sampling technique for code generation, improving sample diversity and model performance in compiler optimization. Read the paper

Generative Models and Interactive Environments

• Genie: Trains a generative model to create interactive virtual worlds from various inputs, advancing generative AI and simulation capabilities. Read the paper

Enjoyed This Story?

I write a weekly analysis of the AI world in the Turing Post newsletter. Subscribe for free using the button below and be the first to read the latest stories.

The goal of Turing Post is to equip you with comprehensive knowledge and historical insights, so you can make informed decisions about AI and ML. Join over 50,000 readers from the main AI labs, forward-thinking startups, and major universities 👇🏼