"What prompt engineering techniques do you use in production, and when does each one help?"
As AI engineering matures, interviewers want evidence you've moved past "write a good prompt" into specific techniques with known tradeoffs. The answer separates AI-tourists from engineers who ship real LLM systems.
Zero-shot: just describe the task. Works well for capable models and straightforward tasks. Default starting point.
Few-shot (in-context learning): include 2–5 input-output examples in the prompt. Helps when the task format is non-obvious, like extracting structured data from text or applying a domain-specific style. Cost: more tokens per request.
Chain-of-thought (CoT): ask the model to "think step by step" or show reasoning before the final answer. Significantly helps multi-step reasoning and math. Caveat: newer reasoning models (o1, o3) do this internally — explicit CoT can hurt them.
Structured output: require JSON output conforming to a schema. Use the provider's native JSON mode or function-calling API rather than parsing freeform text. This is the single biggest reliability improvement for production pipelines.
Role / system prompt: set persona, tone, and hard constraints in the system message. Separate from the user message so downstream user input can't override the instructions. Put security-critical guardrails here.
Prompt engineering in production is less about clever phrasing and more about systematic patterns with known tradeoffs. Zero-shot is the default — describe the task plainly. For capable models this handles most cases. Few-shot — showing 2 to 5 input-output examples — helps when the output format is non-obvious or domain-specific. For example, extracting structured fields from messy text, or matching a particular writing style. The cost is more tokens per request, and the examples need to be representative; biased examples create biased outputs. Chain-of-thought prompting adds "think step by step before the final answer" or similar instructions. It dramatically helps multi-step reasoning, math, and code analysis because the model has room to work through the problem in its output tokens. Newer reasoning models like o1 and o3 do this internally in hidden reasoning tokens — adding explicit CoT to those actually hurts. Know which model family you're using. Structured output is probably the single highest-leverage technique for production. Instead of asking for "a JSON-formatted response" in freeform text and parsing it, use the provider's JSON mode or function-calling API — OpenAI's response_format={"type": "json_schema", "schema": ...}, Anthropic's tool use, or models that support constrained decoding. The model is guaranteed to emit valid JSON matching the schema. This eliminates parsing failures and makes downstream pipelines robust. Role prompting — using the system message to establish persona, tone, and hard constraints — separates static instructions from user input. Security-critical guardrails go in the system prompt; user input goes in the user message. A common pattern I use: system prompt with role ("You are a careful data analyst...") + output format ("Respond as JSON matching schema X") + constraints ("If the question is out of scope, say 'out of scope'"). User prompt is just the actual user query, sanitized. For long-context RAG, prompt position matters: models pay more attention to the start and end of long contexts (the "lost in the middle" effect), so critical instructions go in the system prompt, and the most important retrieved chunks go at the end of the context. The real discipline is having evals. A prompt that works on three test cases can silently regress on production traffic. I keep a versioned golden set of inputs with expected outputs and score every prompt change against it.
Mentioning structured output / JSON mode is the strongest signal of production experience — most candidates skip this.
Knowing that reasoning models (o1, o3) invalidate traditional CoT shows you've kept up with 2024–2026 changes.
The "lost in the middle" effect for long contexts is a specific, memorable talking point.
If you freeze on provider-specific JSON mode syntax during the interview, HireMe AI can surface response_format / tool use examples.
No — it increases cost and sometimes overconstrains the model. For tasks where the model already performs well zero-shot, few-shot can hurt.
It helps non-reasoning models but adds tokens and latency. For reasoning models it's counterproductive. And for genuinely hard problems beyond the model's reasoning ceiling, CoT just produces longer wrong answers.
Never put user input inside system-level instructions. Use structured inputs, validate outputs, and for high-stakes outputs, have a separate LLM act as a safety check.
HireMe AI listens to your interview in real-time, detects questions, and generates instant answer suggestions — so you never freeze up again.
Try HireMe AI Free