How to Use an Interrogatory LLM to Extract Expert Knowledge

Introduction

When you need a large language model (LLM) to handle a complex task, it often requires a lot of context – details about the desired feature, implementation guidelines, external systems to consult, and more. Manually writing this context can be time-consuming and error-prone. An alternative is to let the LLM interview you, the human expert, to gather all necessary information. This technique, called an interrogatory LLM, flips the script: instead of you feeding the model context, the model asks you targeted questions and builds the context itself. This How-To guide will walk you through the process, from preparing the LLM to using it for document review or knowledge capture.

How to Use an Interrogatory LLM to Extract Expert Knowledge — Source: martinfowler.com

What You Need

Access to an LLM – Ideally one that can follow multi-turn conversations (e.g., GPT-4, Claude, etc.).
A clear goal – What context do you need the LLM to produce? (e.g., a feature specification, a project plan, a knowledge document).
Willingness to be interviewed – You'll be answering questions one at a time.
Optional: existing documents – If you want the LLM to review or update an existing document, have it ready.
Patience and time – The process may take several rounds of questions.

Step-by-Step Guide

Step 1: Set Up the LLM to Act as an Interviewer

Start by prompting the LLM with clear instructions. Tell it that it will be interviewing you (a human expert) to gather information for a specific purpose. For example:

“You are an interviewer. Your task is to ask me questions one at a time to collect all the details needed to write a design document for a new feature. Do not ask multiple questions at once. After you have enough information, summarize and present the context.”

Make sure to include the single-question rule in your prompt. This prevents overwhelming you with multiple questions and keeps the conversation focused.

Step 2: Instruct the LLM on the Scope and Sources

Tell the LLM what you want the final context to cover. For instance, mention the user interface aspects, implementation guidelines, and external systems. If the LLM needs to consult other sources (like a codebase or external API docs), specify those. You can say something like:

“The context should include: how the feature appears to users, implementation constraints, and any dependencies on third-party services. Refer to the attached API documentation if needed.”

This step helps the LLM ask relevant questions without you having to repeat basic facts.

Step 3: Begin the Interview – Answer One Question at a Time

Now let the LLM start asking questions. It will begin with a single question. Answer it thoroughly but concisely. The LLM will then ask the next question based on your answer. Continue this loop until the LLM signals it has enough information. If the LLM occasionally asks multiple questions, gently remind it to ask only one at a time. This is a common issue, so be prepared to reinforce the rule.

During the interview, you can also provide clarifications or steer the conversation by saying, “Could you also ask about X?” The LLM is flexible and will adapt.

Step 4: Let the LLM Generate the Context Document

After the LLM has gathered enough information, ask it to compile everything into a structured context report. You can prompt:

“Now please summarize all the information you've collected into a well-organized context document. Use headings, bullet points, and clear sections. This document will be used by another LLM (or a human) to proceed with the next steps.”

The LLM will output a formatted document. You can then review it, make edits, or use it directly.

Step 5: Use the Technique for Document Review (Optional)

Instead of building a document from scratch, you can give the LLM an existing document – such as a software specification – and ask it to interview you to verify its accuracy. This is an alternative to you reading and reviewing the document line by line. Prompt the LLM to:

“Here is a document about [topic]. Interview me to determine if the information in this document is correct. Ask me one question at a time. At the end, provide a revised version of the document with any corrections.”

This approach is especially helpful when the document is poorly written or when you find reviewing tedious. The conversation with the LLM can uncover gaps or errors more naturally.

Step 6: Combine Multiple Interrogatory LLMs for Complex Workflows

You can chain multiple interrogatory sessions. For example, use one LLM to build the initial context document (as in Steps 1–4). Then use a different LLM (or the same one) to review that document with another expert. This multi-pass approach ensures thoroughness. Each LLM can be specialized: one for gathering facts, another for quality assurance.

Tips for Success

Remind the LLM frequently: The model may forget the “one question at a time” rule after a few turns. Reiterate it if needed.
Be specific about your constraints: If you only have a limited time, tell the LLM to prioritize the most important questions.
Use context from previous sessions: If you're working on a multi-step project, feed the output of one session as input to the next.
Consider different models for different roles: Some LLMs are better at asking probing questions, others at synthesizing information. Experiment.
Don't worry about AI “taint”: The resulting text may sound like AI writing, but it's better than no documentation. You can always polish it later.
Use for knowledge capture: If you find writing difficult, use this technique to extract your own knowledge. The LLM acts as a smart interviewer, turning your spoken thoughts into structured documents.

By following these steps, you can harness the power of interrogatory LLMs to efficiently create high-quality context, review documents, and capture expertise – all while reducing the manual burden of writing.

Tags: