Meta-Meta-Prompting

Meta-prompting was was introduced in the Meta Prompting for AI Systems paper by Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao and is defined as follows:

A framework that elevates the reasoning capabilities of large language models (LLMs) by focusing on the formal structure of a task rather than content-specific examples.

In this post, we'll be taking a look at the Meta-Meta-Prompt technique described in the paper, and how we can use it to increase the accuracy of tasks given to LLMs.

We are going to be working with prompts that operate at different levels of abstraction. To help make things clearer, some terms will be introduced along the way to help differentiate different kinds of prompts.

What is Meta-Prompting?

Meta-prompting is a prompting technique that explains to a LLM how to accomplish a task using a step-by-step formula tailored specifically for that task.

Instead of trying to get the LLM to complete a task with a prompt like this:

Do (details of TASK)

You'd instead instruct it using a prompt like this:

To do TASK, follow these steps

1. ...
2. ...
3. ...

Now do TASK

The steps to accomplish TASK are described in the prompt with the goal of providing the LLM enough instructions to generate a more accurate response. This style of prompting is called a meta-prompt: we are no longer just telling the LLM what to do, but how to do it. We will define a meta-prompt like this:

Meta-prompt

A prompt which includes the specific steps required to complete a task

Concrete Prompt Example

Before we take a look at a meta-prompt example, we should start with a concrete prompt and work our way up:

Concrete Prompt

A prompt that instructs the LLM to complete a task

Notice that our definition of a concrete prompt does not describe how to do something. It simply tells the LLM to do it.

Here's an example of a concrete prompt:

Convert this SQLite database table into diesel-compatible data structures in Rust:

CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    username TEXT NOT NULL UNIQUE,
    email TEXT NOT NULL UNIQUE,
    age INTEGER CHECK (age >= 18),
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

This prompt directly tells the LLM what to do. It is concrete in the way that a class can be "concrete" in programming: everything is spelled out directly and there are no abstractions.

It's easy to assume that a LLM will get this right because the models are trained on basically the entire internet. In addition, variations of this table are used all over the place in code examples so it should probably get it right without any prompt engineering.

Here's the result:

use chrono::NaiveDateTime;
use diesel::prelude::*;
use diesel::sql_types::{Integer, Text, Timestamp};

table! {
    users (id) {
        id -> Integer,
        username -> Text,
        email -> Text,
        age -> Integer,
        created_at -> Timestamp,
    }
}

// Struct for inserting new users (excludes auto-generated fields)
#[derive(Insertable, AsChangeset)]
#[diesel(table_name = users)]
pub struct NewUser {
    pub username: String,
    pub email: String,
    pub age: i32,
}

// Struct for querying/retrieving users (includes all fields)
#[derive(Queryable, Debug, Clone)]
pub struct User {
    pub id: i32,
    pub username: String,
    pub email: String,
    pub age: i32,
    pub created_at: NaiveDateTime,
}

This completion is almost correct: the age field should be Option<i32> and we should use Nullable<Integer> in the table! macro. This is pretty easy to fix manually. However, the code above isn't what the LLM actually generated. I copied it from this giant wall-o-text:

Click to expand

Below is a complete example of how to represent the provided SQLite users table using Diesel-compatible data structures in Rust. This assumes:

• You're using Diesel 2.x (the latest stable version) with the sqlite feature enabled.
• You have the chrono crate for handling DATETIME (via diesel's Timestamp type, which maps to chrono::NaiveDateTime for SQLite).
• Dependencies in your Cargo.toml should include:

  [dependencies]
  diesel = { version = "2.0", features = ["sqlite", "chrono"] }
  chrono = { version = "0.4", features = ["serde"] }
  

• The schema is defined using Diesel's table! macro for compile-time type safety.
• I've included:
  • The table schema definition.
  • A NewUser struct for insertions (excludes auto-generated fields like id and created_at).
  • A User struct for querying existing records (includes all fields).
• Constraints like UNIQUE, NOT NULL, and CHECK (age >= 18) are enforced at the database level, but you can add runtime checks in your Rust code if needed (e.g., in NewUser validation).
• This goes in a module like schema.rs (which Diesel's CLI can generate, but I've shown it manually here).

schema.rs

use chrono::NaiveDateTime;
use diesel::prelude::*;
use diesel::sql_types::{Integer, Text, Timestamp};

table! {
    users (id) {
        id -> Integer,
        username -> Text,
        email -> Text,
        age -> Integer,
        created_at -> Timestamp,
    }
}

// Struct for inserting new users (excludes auto-generated fields)
#[derive(Insertable, AsChangeset)]
#[diesel(table_name = users)]
pub struct NewUser {
    pub username: String,
    pub email: String,
    pub age: i32,
}

// Struct for querying/retrieving users (includes all fields)
#[derive(Queryable, Debug, Clone)]
pub struct User {
    pub id: i32,
    pub username: String,
    pub email: String,
    pub age: i32,
    pub created_at: NaiveDateTime,
}

Usage Example

Here's a quick example of how you might use these structs in your application (e.g., in lib.rs or main.rs). This assumes you have a Diesel connection pool set up.

use diesel::prelude::*;
use diesel::result::ConnectionError;  // For error handling
use schema::{users, NewUser, User};   // Adjust path as needed

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let connection = establish_connection();  // Your connection setup

    // Insert a new user
    let new_user = NewUser {
        username: "example_user".to_string(),
        email: "[email protected]".to_string(),
        age: 25,  // Enforce age >= 18 in your app logic if desired
    };

    diesel::insert_into(users::table)
        .values(&new_user)
        .execute(&mut connection)?;

    // Query a user
    let queried_user: User = users::table
        .filter(users::username.eq("example_user"))
        .first(&mut connection)?;

    println!("{:?}", queried_user);

    Ok(())
}

fn establish_connection() -> SqliteConnection {
    let database_url = "your_database.db";  // Replace with your DB path
    SqliteConnection::establish(&database_url)
        .expect("Error connecting to database")
}

Notes

• Auto-generated Fields: id uses AUTOINCREMENT, so it's not included in NewUser. created_at defaults to CURRENT_TIMESTAMP in SQLite, so it's omitted from NewUser—Diesel will let the DB handle it. If you need to override created_at on insert, add it to NewUser and use #[diesel(column_name = created_at)] if needed.
• Types:
  • INTEGER maps to i32 (Diesel's Integer).
  • TEXT maps to String (Diesel's Text).
  • DATETIME maps to chrono::NaiveDateTime (Diesel's Timestamp for SQLite; use Timestamptz if you need timezone support in other DBs).
• Constraints:
  • UNIQUE on username and email will cause Diesel to return a NotFound or constraint violation error on insert—handle with Result and error mapping.
  • The CHECK (age >= 18) is DB-enforced; add a validate_age method to NewUser for Rust-side checks, e.g.:

    impl NewUser {
        pub fn validate(&self) -> Result<(), &'static str> {
            if self.age < 18 {
                return Err("Age must be 18 or older");
            }
            Ok(())
        }
    }
    

• Migration: To create this table in Diesel, use the Diesel CLI (diesel migration generate create_users) and define the SQL in the up.sql file as your original CREATE TABLE.
• Customization: If you're using Diesel 1.x or need JSON/serialization, adjust derives (e.g., add Serialize/Deserialize from serde). For more complex queries, use Diesel's query builder with the table! columns (e.g., users::age.gt(18i32)).

This setup ensures type-safe, efficient interaction with your SQLite table. If you provide more details (e.g., full schema or specific use cases), I can refine it!

---

So while the response does have a mostly-correct answer within, the LLM technically did not do what we asked. We asked for:

...diesel-compatible data structures in Rust...

We didn't ask for:

• Introductory text
• Dependency lists
• Explanation of what it wrote
• Extra things that could be added later
• Usage examples
• Extra notes
• ...

So a lot of extra material was generated that we don't want, and it's just burning tokens without providing any additional value. Now if we wanted all these extras then that's great. But given that we already have the SQL code, there's a good chance that we have already considered the implications of writing the table the way we did, making most of the extra content useless.

Converting to a Meta-Prompt

In the previous section we used a concrete prompt to generate a completion. While the generated code wasn't exactly what we wanted, the fix was trivial enough to be acceptable. Let's see if we can do better by using a meta-prompt.

To recap, here is the original concrete prompt:

Convert this SQLite database table into diesel-compatible data structures in Rust:

(SQL code here)

And here is a meta-prompt that explains how to convert a SQLite table into diesel-compatible Rust data structures:

You are an expert Rust developer with deep knowledge of Diesel, an ORM and query builder for Rust that works with SQLite databases. Your task is to convert a given SQLite database schema into Diesel-compatible data structures. This involves generating Rust modules with appropriate table! macros, structs for tables with Diesel derives (like Queryable, Insertable, AsChangeset), enums for any custom types if needed, and any necessary schema.rs definitions. Ensure the generated code is idiomatic, handles relationships (e.g., foreign keys as associations), maps SQLite types accurately to Rust types (e.g., INTEGER to i32, TEXT to String), and includes imports from diesel and diesel::prelude::*.

Follow these steps precisely:

1. Parse the Schema: Read the provided SQLite schema (which includes CREATE TABLE statements). Identify all tables, columns (with types, primary keys, foreign keys, indexes, and constraints), and any views or triggers if present. Note data types: e.g., INTEGER → Option or i32, TEXT → String, BLOB → Vec, BOOLEAN → bool, etc. Handle nullable columns with Option.

2. Define Schema Module: Create a schema.rs file content using Diesel's table! macro for each table. Specify columns with their types, including sql_types like Integer, Text, etc. Define foreign keys as references between tables.

3. Generate Model Structs: For each table, create a corresponding Rust struct (e.g., for table "users", struct NewUser and User). Use #[derive(Queryable, Insertable, AsChangeset)] where appropriate. Include fields matching the columns, with Diesel's #[diesel(table_name = users)] attribute if needed. For primary keys, use Diesel's conventions (often id: i32 with #[primary_key]).

4. Handle Relationships: If foreign keys exist, define associations using Diesel's belongs_to, has_many in a separate models module or via join tables. Suggest how to use diesel::associations if complex.

Edge Cases and Best Practices:

• Use #[table_name = "table_name"] for non-conventional names.
• For composite keys, use tuples in the table! macro.
• Ensure timestamps (e.g., DATETIME) map to chrono::NaiveDateTime with diesel's types.
• Add any necessary custom types or enums (e.g., for ENUM in SQLite).

Output the complete, self-contained Rust code in a code block, organized into files like schema.rs and models.rs with no extra information.

The SQLite database schema to convert is:

CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    username TEXT NOT NULL UNIQUE,
    email TEXT NOT NULL UNIQUE,
    age INTEGER CHECK (age >= 18),
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

As you can see, this prompt is extremely detailed and has a lot of information. It follows a 5-block pattern:

1. General instructions
   Tell the LLM that it's an expert at the specific task and provide some general information about what the task is.

2. A list of steps to take to accomplish the task
   These should be the broad processes required to accomplish the task followed by any relevant details. For example, the Parse the Schema step is bold and is followed immediately with the details of what Parse the Schema actually means.

3. Extra details (edge cases/best practices)
   Include the details of extra things to do, or things to watch out for. Project-specific information can be included here as well.

4. Describe how the output should be formatted
   It's very important to include how you want the output to be formatted. If you have a very specific output format, provide an example here. This is how we will go from wall-o-text to just the code.

5. The actual task data to process

With this prompt, the LLM generates an output that much more aligned with our original request to generate "diesel-compatible data structures in Rust". Here is the complete output from model:

// src/schema.rs
// This file is generated by Diesel CLI and should not be manually edited.

table! {
    users (id) {
        id -> Integer,
        username -> Text,
        email -> Text,
        age -> Nullable<Integer>,
        created_at -> Text,
    }
}

// src/models.rs
use chrono::NaiveDateTime;
use diesel::prelude::*;
use diesel::Insertable;
use diesel::Queryable;
use diesel::AsChangeset;
use crate::schema::users;

#[derive(Queryable, AsChangeset, Debug)]
#[diesel(table_name = users)]
pub struct User {
    pub id: i32,
    pub username: String,
    pub email: String,
    pub age: Option<i32>,
    pub created_at: Option<NaiveDateTime>,
}

#[derive(Insertable, Debug)]
#[diesel(table_name = users)]
#[diesel(check_for_backend(diesel::sqlite::Sqlite))]
pub struct NewUser {
    pub username: String,
    pub email: String,
    pub age: Option<i32>,
    pub created_at: Option<NaiveDateTime>,
}

Notice that it output only the code (as was originally requested) and also it marked the age field as Option<i32> as indicated in the original SQLite schema. The only thing it got wrong was re-importing some things that are already included in the diesel::prelude module, but this code can still be used unmodified.

So my first thought when I stumbled upon meta-prompting was "Ok... but I'm not going to write that giant wall-o-text for every task that I want the LLM to do". And I would imagine you are thinking something along those lines that as well. Thankfully, the authors of the paper have a solution for this which we'll look at next.

Automating Meta-Prompt Generation

Writing super-detailed instructions for every task would be a huge endeavour and wouldn't save time for one-off tasks. So instead of doing that, we can leverage the LLM to do it for us!

We do this by using a:

Meta-Meta-Prompt

A prompt that generates prompts

A meta-meta-prompt is a universal prompt that acts as a blueprint, allowing you to create specialized meta-prompts for virtually any scenario.

Using a meta-meta-prompt, the workflow for tackling any task changes and now consists of three steps:

1. Identify the task
   Start by framing your objective in broad, reusable terms to establish the core structure.

2. Generate a meta-prompt
   Combine the task description with the meta-meta-prompt.

3. Execute the task
   Combine the generated meta-prompt with the data for your task.

So while it does take a little extra work, the generated output has a higher chance of being what you wanted. And there will be less of a need for back-and-forth conversation with the LLM because all the details are provided up-front.

The authors of the paper crafted a meta-meta-prompt that was designed for self-improving autonomous systems that recursively iterate on prompts to eventually reach a solution to a given problem. While we won't be writing any fully autonomous systems in this post, we can still take advantage of the meta-meta-prompt for a single-task situation.

I've modified the authors' meta-meta-prompt for easy copy+pasting of prompts and inputs. The generated meta-prompt should not contain substitution phrases like [INPUT HERE] randomly throughout the prompt, making it easy to flow from one prompt to another. Here is the meta-meta-prompt:

Task: Meta Prompting for In-Context Prompt Design

1. Input Analysis:
   • Input: [User task instructions]
   • Action: Analyze and extract key concepts, methodologies, challenges, and objectives.
2. Task Interpretation:
   • Action: Synthesize the extracted information to define the core problem or task.
   • Considerations: Identify constraints, goals, or requirements.
3. Prompt Design:
   • Objective: Develop a structured prompt for problem-solving, including clear instructions, a step-by-step approach, and relevant background information.
4. Optional – Direct Solution Proposal:
   • Objective: Propose initial steps or a complete solution strategy, ensuring feasibility and practicality.
5. Output Prompt: [Generate the output prompt]

Note: The output should be a coherent, actionable prompt or solution strategy tailored to the specifics of the input task. Structure the prompt so it naturally ends by indicating that the user input will follow the end of the prompt. Do not include replacement indicators like [input here] or [paste here].

User task instructions:

To use the meta-meta-prompt, we need to include our task after "**User task instructions:****". However we cannot just paste our original concrete task at the end and expect it to work. The input task will need some minor adjustments before being ready for the meta-meta-prompt.

Preparing a Task for Meta-Meta-Prompting

In this section we will convert our concrete task into an:

Abstract Task Description

Reworded, generalized task suitable for meta-meta-prompt input

We need to word our concrete task in a way that does not include the actual task information. This way the meta-meta-prompt can create a generalized meta-prompt that works with any given task information.

That was kind of a mouthful, so let's look at some examples:

Concrete Prompt	Abstract Task Description
Scrape the homepage of example.com	Extract content from a specified web page address, organizing it into a structured JSON format that captures elements like title, headings, and links.
Summarize this article: [paste article text]	Condense a provided article on a designated topic into a brief summary of its main points, keeping the output under 200 words.
Translate "Hello, world!" from English to French	Convert a supplied English phrase into French, ensuring the translation retains the precise meaning and original tone.
Generate a Python script to calculate the factorial of 5	Develop a Python function to compute the factorial of an input positive integer, with built-in error handling for non-valid cases.
Classify this email as spam or not: [email body]	Analyze the content of a given email to classify it as "spam" or "not spam". While evaluating the email, use common indicators such as suspicious links or wording.
Write a marketing description for a wireless mouse that's ergonomic and blue.	You will be given a brief product description and your task is to write an engaging product description that emphasizes key attributes of the item, such as its design and color, to appeal to potential buyers.

So the goal would be to go from the concrete prompt on the left and turn it into an abstract task description on the right.

Generating Abstract Task Descriptions

We'll stick with the spirit of meta-prompting and use the meta-meta-prompt to generate a meta-prompt to generate the abstract task description:

Click to expand

You are an expert in prompt engineering, specializing in abstracting specific instructions into generalized task descriptions. This process helps in creating reusable, in-context prompts for AI models by distilling concrete examples into broader, applicable frameworks.

Your goal is to take a given concrete prompt—such as a specific instruction for a task like web scraping, summarization, translation, code generation, classification, or content creation—and convert it into an abstract task description. The abstract version should:

• Capture the core objective, key actions, and any relevant constraints or requirements (e.g., output format, error handling, evaluation criteria) without referencing the specific details of the example.
• Maintain neutrality and generality, making it suitable for application to similar but varied inputs.
• Be concise yet comprehensive, typically 1-2 sentences, phrased as a clear task directive.
• Draw inspiration from the following examples to ensure consistency in style and detail:

Concrete Prompt	Abstract Task Description
Scrape the homepage of example.com	Extract content from a specified web page address, organizing it into a structured JSON format that captures elements like title, headings, and links.
Summarize this article: [paste article text]	Condense a provided article on a designated topic into a brief summary of its main points, keeping the output under 200 words.
Generate a Python script to calculate the factorial of 5	Develop a Python function to compute the factorial of an input positive integer, with built-in error handling for non-valid cases.
Classify this email as spam or not: [email body]	Analyze the content of a given email to classify it as "spam" or "not spam". While evaluating the email, use common indicators such as suspicious links or wording.

To perform the conversion:

1. Read the concrete prompt carefully and identify its main verb/action (e.g., scrape, summarize, translate), inputs (e.g., URL, text, phrase), outputs (e.g., JSON, summary, code), and any specifics (e.g., word limits, handling rules).
2. Generalize by replacing concrete elements with placeholders like "a specified," "provided," or "a given" to make it applicable to variations.
3. Ensure the description is actionable and self-contained, as if it were a prompt template for an AI assistant.
4. If the concrete prompt includes unique elements (e.g., tone or format), incorporate them as guidelines in the abstract version.

Now, convert the following concrete prompt into an abstract task description:

---

Using that meta-prompt with the "Write a marketing description for a wireless mouse that's ergonomic and blue" concrete prompt example yields this abstract task description:

Create a persuasive marketing description for a specified product, emphasizing its key features such as ergonomic design and color, in an engaging tone suitable for promotional materials.

Using the generated abstract task description with the original meta-meta-prompt results in this meta-prompt for generating marketing descriptions:

Click to expand

You are an expert copywriter specializing in persuasive marketing content for consumer products. Your goal is to craft engaging, compelling descriptions that highlight key features to drive interest and sales, using vivid language, emotional appeals, and a enthusiastic tone perfect for websites, ads, or packaging.

To create an effective description:

1. Start by understanding the product: Identify its primary function, target audience, and unique selling points.
2. Emphasize the specified key features: Weave in details like ergonomic design (e.g., how it fits comfortably and reduces strain) and color options (e.g., how they add style or match lifestyles), making them sound irresistible.
3. Structure the description: Begin with a hook to grab attention, build excitement around benefits, and end with a call to action or memorable tagline. Keep it concise (150-250 words) yet vivid.
4. Adopt an engaging tone: Use conversational, upbeat language with rhetorical questions, sensory details, and persuasive words to make readers envision owning it.

Now, based on the following product details, write the marketing description.

Product details:

---

And when combining the meta-prompt with "a wireless mouse that's ergonomic and blue" we can finally generate a marketing description:

Click to expand

Elevate Your Workflow with Effortless Precision

Tired of tangled cords cramping your style and stiff wrists slowing you down? Say hello to the ultimate game-changer: our sleek Wireless Ergonomic Mouse in vibrant blue! Designed for the modern multitasker—whether you're conquering deadlines at the desk, gaming through epic quests, or streaming your creative genius—this mouse isn't just a tool; it's your new best friend for seamless, strain-free navigation.

Crafted with thoughtful ergonomics, its contoured shape cradles your hand like a custom glove, promoting a natural grip that reduces fatigue and wrist strain during those marathon sessions. No more awkward angles or nagging aches—just smooth, intuitive control that lets you click, scroll, and conquer with effortless precision. The wireless freedom means you can roam up to 30 feet without a single snag, keeping your setup clutter-free and your focus laser-sharp.

And that eye-catching blue hue? It's more than just color—it's a bold statement of style that complements your vibrant lifestyle, from minimalist home offices to colorful creative studios. Pair it with any device via reliable 2.4GHz connectivity, and watch productivity soar.

Why settle for ordinary when extraordinary is just a click away? Grab your Wireless Ergonomic Mouse in Blue today and unlock a world of comfort and control. Your hands (and your workflow) will thank you!

---

Output Modification

If you want to steer clear of common LLM clichés such as the overused phrase "It's more than just X, it's Y", try adding an instruction like "Avoid common LLM clichés like..." directly to your abstract task description.

This approach works because the resulting meta-prompt will weave in your request, which helps minimize the chances of those clichés appearing in the final output. Keep in mind, though, that the generated meta-prompt won't always explicitly state "avoid this thing." In some cases, it might cleverly sidestep the issue by suggesting a fresh, unexpected set of steps that still achieves the goal effectively. The exact outcome can vary depending on the specifics of your input task.

Always include such requirements in the abstract task description upfront, rather than tacking them on at the end of an existing meta-prompt. This ensures the meta-meta-prompt (the one generating your meta-prompt) has access to all your preferences from the start, leading to a more integrated and reliable result.

Summary

Whew! This post was prompts all the way down, so let's briefly recap how to use meta-meta-prompting:

1. Identify the Concrete Task
   Begin by clearly defining your specific goal or problem. What exactly do you need to accomplish? For example, "Summarize this article on climate change."

2. Separate the Core Task from the Specific Data
   Break down the task into its reusable action (e.g., "summarize") and any unique task-specific details (e.g., "this article on climate change"). Replace the details with a neutral placeholder like [data] or [input article] to keep things general. Now it looks like: "Summarize [input article]."

3. Create an Abstract Task Description
   Either manually write the abstract task description or use this prompt to do it for you. This strips away specifics and focuses on the "why" and "how" of the process, making it adaptable to any similar scenario.

4. Generate a Meta-Prompt
   Feed the abstract task description into the meta-meta-prompt. This outputs a meta-prompt tailored specifically for the task.

5. Apply the Meta-Prompt to Complete the Task
   Append the actual data from your original concrete task to the end of the generated meta-prompt and then run the final completion with the LLM.