What are the best LLM engineer interview questions?
The best LLM engineer interview questions test model-layer depth. That means LoRA and QLoRA fine-tuning tradeoffs, quantization formats and their accuracy-versus-speed cost, and inference serving decisions across vLLM, TGI, and TensorRT-LLM. It also means evaluation discipline using tools like promptfoo, RAGAS, or Arize Phoenix. A candidate who can only describe calling an API is not an LLM engineer. That holds true regardless of how fluently they talk about GenAI.
LLM engineers work at the model layer: fine-tuning, quantization, and serving open-weight models, not just calling hosted APIs
Quantization tradeoff questions are the fastest way to separate candidates who have actually deployed models from those who have only read about it
Evaluation-harness fluency (not just “we checked the outputs manually”) is one of the strongest predictors of production readiness
Why “AI Engineer” Interview Questions Don’t Work for LLM Engineers
Job titles in this space have blurred together faster than hiring processes have caught up. An “AI engineer” today usually means someone who builds application-layer GenAI features. Think RAG pipelines, agent orchestration, calling a hosted model API, and wiring the response into a product. An LLM engineer is a different animal entirely. This is the person who fine-tunes the model itself and decides which quantization format ships to production. They also pick the inference server that keeps latency under budget at scale. And they build the evaluation harness that catches regressions before users do.
Interview panels frequently reuse the same question bank for both roles. It shows up in bad hires within the first sprint. A candidate who can talk fluently about LangChain agents and prompt engineering may have never touched a training run. They may have never opened a GGUF conversion script. And they may have never had to explain to a VP of Engineering why P95 latency doubled after a model swap. None of that is a knock on application-layer skill. It is simply not the same job.
The 24 questions below are organized by the actual work an LLM engineer does day to day. That includes fine-tuning decisions and quantization and serving tradeoffs. It also includes evaluation and debugging discipline. And it includes the safety work that keeps an open-weight model from embarrassing the company in production. For each question, we describe what a strong answer sounds like. Just as important, we describe what a weak answer reveals about the candidate’s actual depth.
Fine-Tuning Judgment (Questions 1-6)
Question 1: LoRA, QLoRA, or Full Fine-Tuning
Question 01
Walk me through how you decide between LoRA, QLoRA, and full fine-tuning for a given adaptation task.
Tests: whether the candidate reasons from constraints or recites definitions.
Strong answer: Explains that LoRA freezes the base weights and trains small low-rank adapter matrices injected into attention and MLP layers. This keeps GPU memory and training time low. QLoRA adds 4-bit quantization of the frozen base model during training. That is the difference between needing an 80GB card and fitting a 70B-parameter fine-tune on a single consumer-class GPU. Full fine-tuning updates every parameter and gives the strongest adaptation depth. But it carries real risk of catastrophic forgetting and costs an order of magnitude more compute. A strong candidate picks based on the size of the adaptation gap and the available hardware budget. They also weigh whether preserving general capability matters. Weak candidates default to “LoRA is always better” without acknowledging when full fine-tuning is actually the right call.
Question 2: Setting LoRA Rank and Alpha
Question 02
What rank and alpha values would you start with for a LoRA fine-tune, and how do you know when to adjust them?
Tests: hands-on training experience versus memorized hyperparameter tables.
Strong answer: Describes starting somewhere in the r=8 to r=32 range depending on task complexity. Alpha is typically set at 1x to 2x the rank. Explains that rank controls the capacity of the adapter to represent new behavior. A narrow stylistic adaptation needs less rank than a task requiring new reasoning patterns. Describes watching training loss curves and validation performance to detect when rank is too low, where loss plateaus above an acceptable floor. The same monitoring catches when rank is too high, where validation loss diverges from training loss and signals overfitting on a small dataset. Cites specific runs and what changing rank actually did to output quality. Weak answers just repeat “8 is standard” without connecting the value to the task.
Question 3: Fine-Tuning With a Few Hundred Examples
Question 03
How do you build a fine-tuning dataset when you only have a few hundred labeled examples of the target behavior?
Tests: practical data engineering for low-resource fine-tuning.
Strong answer: Describes LLM-assisted synthetic data generation to expand a small seed set. Human review of a sample catches drift or repetitive patterns in the generated examples. Discusses held-out validation set sizing appropriate to a small training set. This often means a strict 80/20 or 90/10 split rather than a large held-out slice that starves training. Notes that with very limited data, few-shot prompting is sometimes the more defensible choice than fine-tuning at all. A few hundred examples risks overfitting badly. A strong candidate treats data scarcity as a decision point, not just an obstacle to push through.
Question 4: PEFT Methods and Why LoRA Won
Question 04
Explain PEFT as a category. Where does LoRA sit relative to prefix tuning, adapters, and prompt tuning, and why has LoRA become the default?
Tests: breadth of parameter-efficient fine-tuning knowledge beyond the one method everyone has heard of.
Strong answer: Places LoRA within the broader parameter-efficient fine-tuning family. Adapter layers insert new trainable modules between existing layers, which adds inference latency. Prefix and prompt tuning prepend trainable vectors to the input without touching model weights. That approach is cheap but caps out on adaptation depth for complex tasks. LoRA’s advantage is that its adapters merge back into the base weights at inference time, so there is zero added latency once merged. It can also be trained with far less overhead than adapter layers. Explains this is why LoRA displaced most of the earlier PEFT methods in production use. Weak answers can name PEFT but cannot explain why one method won over the others.
Question 5: Diagnosing Catastrophic Forgetting
Question 05
You fine-tuned a Llama or Qwen open-weight model and it now performs better on your target task, but a colleague reports it seems to have gotten worse at general reasoning and instruction following. How do you confirm and fix this?
Tests: catastrophic forgetting awareness and a real remediation process.
Strong answer: Describes running the base model’s original evaluation suite, or a representative sample of general capability benchmarks, against the fine-tuned checkpoint. This quantifies the regression rather than relying on anecdotal reports. If forgetting is confirmed, discusses mitigation options. These include lowering the learning rate, reducing the number of training epochs, and mixing a portion of general-instruction data into the fine-tuning set to anchor original behavior. Reducing LoRA rank to limit how much the adapter can shift model behavior is another option. A strong candidate treats this as an expected risk they actively guard against, not a surprise.
Question 6: Choosing a Base Model to Fine-Tune
Question 06
How do you decide which open-weight base model (Llama, Mistral, Qwen, or another family) to fine-tune for a given production use case?
Tests: model selection judgment grounded in real constraints, not brand preference.
Strong answer: Weighs license terms, since some Llama licenses carry usage restrictions above certain user counts that matter for commercial deployment. Also weighs context window needs, tokenizer efficiency for the target language or domain, and community fine-tuning support and available quantized checkpoints on Hugging Face. Benchmark performance on tasks similar to the target use case matters more than generic leaderboard rank alone. Describes running a small bake-off across two or three candidate base models on a representative eval set. This happens before committing engineering time to a full fine-tune. Weak answers default to “whatever is newest” without evaluating fit.
Quantization and Inference Serving (Questions 7-12)
Question 7: GGUF vs AWQ vs GPTQ
Question 07
Explain the difference between GGUF, AWQ, and GPTQ quantization, and when you would choose each for a production deployment.
Tests: real quantization deployment experience versus name recognition.
Strong answer: Explains GGUF as a file format built around llama.cpp, designed for CPU and mixed CPU/GPU inference. It is popular for local and edge deployment where a dedicated GPU server is not available. AWQ, or Activation-aware Weight Quantization, preserves the weights most important to activation outputs at higher precision. It generally gives better accuracy retention at 4-bit than naive quantization and is well suited to GPU-served production workloads. GPTQ is a post-training quantization method that also targets GPU inference with strong compression. It was historically an early production standard before AWQ’s activation-aware approach improved on accuracy retention. A strong candidate picks based on deployment target: CPU and edge favors GGUF, while GPU-served production favors AWQ or GPTQ. They can also cite an actual accuracy delta they measured between formats, not just a name they’ve read about.
Question 8: Quantizing a 70B Model for a 24GB GPU
Question 08
You need to quantize a 70B parameter model down to fit on a single 24GB GPU. Walk me through your approach and what you would be worried about.
Tests: practical math and risk awareness around aggressive quantization.
Strong answer: Does rough memory math out loud: a 70B model at FP16 needs roughly 140GB. Getting under 24GB requires 4-bit quantization, which brings weights to roughly 35-40GB before further optimization. It also requires techniques like GGUF’s k-quant mixed precision or AWQ 4-bit, combined with KV cache management and possibly offloading some layers to CPU. Flags the real risk: aggressive 4-bit quantization on a 70B model can degrade performance meaningfully on tasks requiring precise reasoning or numerical accuracy. So the plan includes running the target eval suite before and after quantization, rather than assuming quality holds. Weak candidates give a format name without doing the memory arithmetic or naming what could go wrong.
Question 9: vLLM vs TGI vs TensorRT-LLM
Question 09
Compare vLLM, TGI (Text Generation Inference), and TensorRT-LLM as serving frameworks. When would you pick each?
Tests: real inference serving deployment experience across the major frameworks.
Strong answer: Describes vLLM’s PagedAttention for efficient KV cache memory management and continuous batching. This makes it a strong default for high-throughput multi-tenant serving with broad model family support. TGI, from Hugging Face, offers similar continuous batching with tight integration into the Hugging Face ecosystem. It is often chosen when a team is already standardized on Hugging Face tooling for the rest of the pipeline. TensorRT-LLM compiles models into highly optimized engines specifically for NVIDIA hardware. It delivers the lowest latency and highest raw throughput per GPU, but with a steeper build and iteration cycle and less flexibility when swapping models frequently. A strong candidate picks based on the tradeoff between raw performance and engineering velocity for their specific team.
Question 10: Controlling Latency vs Throughput
Question 10
A production model serving system needs to hit a strict P95 latency target under variable load. What levers do you have to control latency versus throughput?
Tests: systems-level thinking about the throughput-latency tradeoff in serving infrastructure.
Strong answer: Discusses continuous batching parameters, where larger batches improve throughput but can increase tail latency for individual requests. Covers speculative decoding to reduce time-to-first-token and overall generation latency, plus quantization to reduce compute per token. Also covers GPU instance sizing and horizontal scaling with load balancing, and request prioritization or queueing strategies when load spikes exceed capacity. Notes that P95, not average, latency requires specifically monitoring and managing tail behavior. Batching decisions that help average latency can hurt the slowest requests. Mentions streaming partial tokens to the client to make perceived latency better even when total generation time is unchanged.
Question 11: Sizing GPU Infrastructure Without Production Data
Question 11
How do you size GPU infrastructure for a new model deployment before you have real production traffic data?
Tests: capacity planning discipline for a genuinely hard estimation problem.
Strong answer: Describes load testing with synthetic traffic patterns modeled on expected usage: concurrent users, average input/output token length, and request rate distribution. This means calculating tokens-per-second throughput needed at target concurrency and matching that against benchmarked throughput numbers for the chosen model, quantization level, and GPU type. Builds in headroom for traffic spikes and plans for horizontal autoscaling rather than over-provisioning a single large instance. Explains that this estimate should be validated against real traffic within the first weeks of launch, and adjusted as needed. Synthetic load testing rarely captures the real distribution of production query patterns.
Question 12: KV Cache and Serving Efficiency
Question 12
What is KV cache and why does it matter for both memory planning and multi-user serving efficiency?
Tests: understanding of a core inference mechanic that directly drives serving cost.
Strong answer: Explains that the key-value cache stores intermediate attention computations from previous tokens, so the model does not recompute them for every new token generated. This is what makes autoregressive generation tractable at reasonable speed. Notes that KV cache size scales with context length, batch size, and model dimension. It is frequently the actual memory bottleneck in production serving, not the model weights themselves, for long-context workloads. Connects this to why vLLM’s PagedAttention was a meaningful efficiency breakthrough. It allocates KV cache in non-contiguous blocks, similar to virtual memory paging, which lets it serve many concurrent users with varying context lengths without wasting memory on over-allocation.
Evaluation and Debugging (Questions 13-18)
Question 13: Building an Automated Evaluation Harness
Question 13
Walk me through how you would set up an automated evaluation harness for a fine-tuned model before it ships to production.
Tests: whether evaluation is a real engineering practice or an afterthought.
Strong answer: Describes building a golden test set covering the target task plus a general-capability regression set. This runs through a framework like promptfoo for structured test-case comparison across model versions, or a custom harness logging pass/fail against defined rubrics. Mentions RAGAS-style metrics when retrieval is involved, and LLM-as-judge scoring for open-ended generation quality. A human-reviewed sample validates that the judge model’s scoring is trustworthy. Notes that evaluation needs to run on every checkpoint during training, not just the final model. This way regressions are caught early rather than after a full training run completes.
Question 14: Using Observability Tools Day to Day
Question 14
How do you use a tool like Arize Phoenix or LangSmith in your day-to-day debugging workflow, versus just reading raw logs?
Tests: real tool fluency and whether observability is integrated into daily practice.
Strong answer: Describes using trace-level observability tools to inspect the full chain of a request, rather than grepping through unstructured logs. This includes prompt construction, retrieved context if applicable, token-level generation, and latency breakdown by stage. Explains how they use these tools to spot patterns across many requests. One example is a specific prompt template consistently producing longer generations that drive up cost. Another is a cluster of failures correlated with a specific input length or language. A strong candidate has a specific debugging story. That includes a metric that flagged an issue, the trace that let them isolate root cause, and the fix. Weak answers describe only “checking the outputs manually.”
Question 15: Diagnosing an Offline-to-Production Quality Gap
Question 15
A model that scored well on your offline evaluation set is producing noticeably worse outputs in production. What do you check first?
Tests: diagnostic process for the offline-online evaluation gap, a common and costly failure mode.
Strong answer: Checks whether production traffic distribution matches the offline eval set. Real user queries are often messier, longer, or cover edge cases the eval set never included. Checks for a serving-side issue separate from the model itself. This might be a quantization format applied at deploy time that was not present during offline evaluation, a truncated context window in production, or a prompt template mismatch between the training-time format and the serving-time format. Describes pulling a sample of real failing production queries, adding them to the eval set, and re-running to close the gap. This is treated as an ongoing feedback loop, not a one-time fix.
Question 16: Judging “Good Enough” Beyond One Accuracy Score
Question 16
How do you evaluate whether a smaller or more heavily quantized model is “good enough” to replace a larger one, beyond a single aggregate accuracy score?
Tests: nuanced evaluation methodology beyond a single top-line number.
Strong answer: Describes breaking evaluation down by task category and difficulty tier rather than trusting one blended score. Aggregate accuracy can hide a smaller model failing badly on the hardest 10% of cases that matter most in production. Discusses measuring latency and cost improvement against the accuracy delta to make an explicit tradeoff decision. When the quantitative gap is small but the qualitative difference in output quality is unclear from metrics alone, runs a human preference comparison on a sample of outputs. Notes that the “good enough” bar should be set by the business risk of the specific use case, not a generic threshold.
Question 17: Building a Regression Test Suite
Question 17
How do you build a regression test suite that catches quality drops when you update a prompt template or swap an underlying model version?
Tests: engineering discipline treating prompts and models as versioned artifacts.
Strong answer: Describes maintaining a fixed test set with expected characteristics. These are not necessarily exact-match answers, but rubric-scored properties like faithfulness, format compliance, and tone, and the set runs automatically in CI whenever a prompt or model version changes. Uses this to block a deploy if scores drop below a defined threshold. That removes reliance on a human noticing quality decline after the fact. Mentions logging prompt and model version alongside every evaluation run. This lets regressions be traced to the specific change that caused them.
Question 18: Validating an LLM-as-Judge Setup
Question 18
What is your process for using an LLM as a judge to score outputs at scale, and how do you know if the judge itself is reliable?
Tests: meta-evaluation rigor, since LLM-as-judge is only useful if it is validated.
Strong answer: Describes writing a clear rubric for the judge model with specific scoring criteria, rather than a vague “rate this response” prompt. Then runs the judge against a human-labeled sample set to measure agreement rate before trusting it at scale. If agreement is low, iterates on the judge prompt or falls back to human review for that category of output. Notes the risk of judge models having systematic biases, such as favoring longer responses or a particular writing style. Describes controlling for that in the rubric design.
Want Pre-Screened LLM Engineers?
Skip the 24-question interview process. We deliver candidates who already pass our model-layer technical screen.
GET STARTED
Safety, Guardrails, and Deployment (Questions 19-21)
Question 19: Jailbreak-Testing Before Launch
Question 19
How do you jailbreak-test a fine-tuned or newly deployed open-weight model before it goes live?
Tests: whether adversarial testing is a real practice or an assumption that “the base model already handles it.”
Strong answer: Describes running a structured red-team set of known jailbreak patterns against the deployed model. These include roleplay framing, encoding tricks, instruction override attempts, and multi-turn escalation. Fine-tuning can weaken safety behavior the base model originally had, even unintentionally. Mentions using or adapting existing adversarial test suites rather than improvising a handful of prompts. Re-runs this test set after every fine-tune or quantization change, since both can shift refusal behavior in either direction. Notes tracking a specific jailbreak success rate metric over time rather than treating this as a one-time pass/fail gate.
Question 20: Implementing PII Redaction
Question 20
Describe how you implement PII redaction in a pipeline where user input or retrieved context might contain sensitive personal data.
Tests: production-grade privacy engineering, not just awareness that PII is a concern.
Strong answer: Describes a layered approach. The first layer is pattern and named-entity detection on inbound text before it reaches the model. This catches structured PII like SSNs, phone numbers, and emails reliably, and uses an NER model for less structured cases like names and addresses. The second layer is redaction or tokenization of detected PII before logging or storing any request data. A third layer separates PII that must legitimately reach the model for the task to work from PII that should never leave the redaction layer. Discusses the tradeoff between redaction aggressiveness and task quality. Also describes how they validated the redaction pipeline against a labeled test set, rather than assuming a regex list is sufficient.
Question 21: Deployment Safeguards for Public-Facing Models
Question 21
You are deploying an open-weight model that will be publicly accessible through a customer-facing feature. What deployment safeguards do you put in place beyond the model’s own training-time safety behavior?
Tests: defense-in-depth thinking for production safety, not reliance on a single layer.
Strong answer: Describes input classification to catch clearly malicious or off-topic requests before they reach the model. Output classification is a second layer that catches anything that slips past training-time safety behavior. Rate limiting prevents automated abuse and probing. Logging and alerting on flagged interactions supports human review, backed by a documented incident response path if a harmful output does reach a user. Notes this layered approach exists precisely because fine-tuning and quantization can degrade a base model’s original safety alignment. Deployment-time controls cannot assume the model alone is sufficient.
Behavioral Questions (Questions 22-23)
Question 22: Owning a Production Incident
Question 22
Tell me about a fine-tuning or quantization change you shipped that caused a production incident. How did you catch it, and what changed afterward?
Tests: real incident ownership at the model layer, with enough specificity to confirm it happened.
Strong answer: Describes a specific change, such as a new fine-tune, a more aggressive quantization level, or a serving framework swap. Also describes the signal that surfaced the problem, whether that was a monitoring alert on latency or output quality, or user reports. Walks through the diagnostic process to isolate the root cause. The regression could have come from the model change itself, a serving configuration mismatch, or a data distribution shift. Describes the concrete fix. More importantly, describes what evaluation or monitoring gap they closed afterward so the same class of issue would be caught before shipping next time. Candidates without real production ownership give vague, generic answers here.
Question 23: Staying Current Without Chasing Every Release
Question 23
How do you stay current with the rapid pace of open-weight model releases and serving technique changes, and how do you decide what is worth adopting versus skipping?
Tests: genuine technical engagement versus name-dropping every recent release.
Strong answer: Names specific sources followed, such as Hugging Face model releases and leaderboards, specific research groups, and arXiv categories relevant to efficient inference and fine-tuning. Describes a real decision process for adoption. The key question is whether the new technique or model solves a measured problem in their current stack, or is merely interesting without a clear application. Gives a specific recent example of something evaluated, adopted, or deliberately passed on, with the reasoning behind that call. Weak answers list every framework release from the last quarter without describing how any of it changed their actual work.
Scenario Questions (Questions 24)
Question 24: An 8-Week Migration Plan
Question 24
Your team needs to replace a hosted, closed-source LLM API with a self-hosted open-weight model to cut costs, with a hard deadline of eight weeks and a requirement that output quality cannot visibly degrade for end users. Walk me through your plan.
Tests: end-to-end system design judgment combining fine-tuning, evaluation, quantization, and serving decisions under a real constraint.
Strong answer: Structures the plan in phases. First, building a comprehensive evaluation set from real production traffic against the current API’s outputs, since quality parity cannot be judged without a real baseline (weeks 1-2). Next, selecting and benchmarking two or three candidate open-weight base models against that eval set (weeks 2-5). A fine-tuning pass follows if the gap is close but not fully closed. Then, choosing a quantization level and serving framework based on the latency and cost target (weeks 5-6). Load testing under realistic traffic follows. Finally, running a staged rollout, starting with a small percentage of traffic with full monitoring and an instant rollback path to the original API (weeks 6-8). Traffic then scales up gradually to full volume.
How the Plan Handles a Quality Shortfall
Explicitly flags what happens if evaluation reveals a quality gap that cannot be closed in the available time. In that case, the honest recommendation is to extend the timeline or accept a hybrid approach for a subset of harder queries, rather than shipping a quietly degraded experience to hit the deadline.
Frequently Asked Questions
How Long Should the Interview Take?
How long should an LLM engineer technical interview take?
75 to 90 minutes for the live technical interview works well in practice. You do not need to cover all 24 questions in one session. If the role is fine-tuning heavy, lean on questions 1 through 6 plus a scenario question. If the role is serving and infrastructure focused, use questions 7 through 12. Add two or three evaluation questions regardless of specialization, since evaluation discipline is relevant to every LLM engineering role. Pair the live interview with an async take-home covering a quantization or serving tradeoff decision.
LLM Engineer vs AI Engineer vs ML Engineer
How is an LLM engineer different from an AI engineer or a machine learning engineer?
An AI engineer typically works at the application layer, building RAG pipelines and agentic workflows on top of hosted model APIs. A machine learning engineer often works on traditional ML pipelines, feature engineering, and classical model training separate from large language models entirely. An LLM engineer sits at the model layer specifically. That means fine-tuning open-weight models, choosing quantization formats, running inference serving infrastructure, and building evaluation harnesses for generative model behavior. These roles overlap in some teams but require distinctly different interview questions.
Should You Run a Live Coding Exercise?
Should I ask candidates to do a live quantization or fine-tuning exercise during the interview?
A live coding exercise for a full training or quantization run is impractical within an interview window given GPU and time constraints. A more predictive format is an async take-home. The candidate is given a specific tradeoff scenario, for example a target latency and accuracy budget, and asked to write up their approach with reasoning. Alternatively, they are given a small existing fine-tuning script with an intentional issue to review and debug. This produces more signal about real judgment than a whiteboard discussion alone.
What Pass Rate Should You Expect?
What is a realistic pass rate for LLM engineers who reach the technical interview?
This is a narrower specialization than general AI engineering. The candidate pool that can genuinely discuss quantization tradeoffs and inference serving in depth is smaller. With a strong resume screen and an async technical pre-assessment, expect 25 to 40% of candidates who reach the live interview to pass at the senior bar. Without a pre-screen, expect the pass rate to drop into the single digits to low teens. The title is applied loosely across job postings, and many applicants have only worked with hosted APIs, not the model layer itself.
Does the Process Change for Nearshore Candidates?
Do these questions work for evaluating nearshore LLM engineers, or is a different process needed?
The technical bar does not change based on location. A nearshore LLM engineer in Argentina or Costa Rica should be evaluated against the exact same fine-tuning, quantization, and serving questions. The bar is identical to a candidate in the United States. The one process adjustment worth making is confirming timezone overlap and communication clarity early, before investing in a deep technical interview. Run the same async pre-screen, the same live technical questions, and the same scoring rubric regardless of where the candidate is based.
