how I build with AI
Using an LLM is not a skill. Scoping, building, and measuring an AI system is.
Anyone can call an API. The work that matters is deciding what the model is allowed to say, proving a change made answers better instead of worse, and knowing what a query costs before the invoice arrives. Everything below is drawn from projects that are shipped or in active build, not from a course.
(01)
The 8 AI engineering fundamentals, and where I have shipped each one
LLM APIs & prompt design
Prompts are versioned specs with an explicit contract, not strings you tweak in place.
Hive: qa_prompt.yaml, versioned v1/v2, version logged per query
RAG
Retrieval is a product decision. What gets chunked, what gets retrieved, and what the model is allowed to say.
Hive: full ingest → embed → retrieve → generate pipeline, shipped
Embeddings & vector DBs
Knowing what a vector is, what a similarity score means, and why the store choice matters.
Hive: OpenAI text-embedding-3-small into pgvector on Postgres
Agent orchestration
Decomposing a task across model calls, and knowing when a state machine beats a single prompt.
Hani: 7-skill conversational state machine. Hive: query decomposition on the roadmap
Evaluation & observability
If you cannot measure whether a change made answers better or worse, you are not engineering, you are guessing.
Hive: live RAGAs faithfulness and relevance scoring on every query, logged with cost
Guardrails & output validation
The refusal path is a designed, tested feature. Fabrication is a requirements failure.
Hive: answer-only-from-context contract. Hani: never quiz, never correct, written before the prompt
Context-window management
The window the model sees is a known quantity set in config, not whatever the last change produced.
Hive: chunk_size 1000, K=4, enforced. ask-brian-skills: token budget as a product metric
Fine-tuning vs prompting
Most problems are prompt and retrieval problems. Knowing when they are not is the skill.
Hive: prompt-only now, decision framework with exit gates on the roadmap
Fundamentals 2, 3, 5, 6, and 7 all show up in one pipeline. Hive, drawn end to end:
(02)
The stack, named exactly
Specific model IDs and versions, because “uses LLMs” and “shipped a RAG pipeline on pgvector with a cheap judge model on every query” are not the same claim.
Models
- Claude claude-sonnet-4-6 (generation)
- claude-haiku-4-5 (eval judge)
- OpenAI text-embedding-3-small (embeddings)
- OpenAI Realtime API (voice-to-voice)
Orchestration & data
- LangChain 0.3+
- pgvector / Postgres
- Supabase
- Firecrawl
- WebRTC
Eval & delivery
- RAGAs
- Streamlit
- Docker
- Next.js
- Jupyter (as a design-record layer)
(03)
How I evaluate
Two scores on every answer. Faithfulness asks whether the answer stayed inside the retrieved context. Relevance asks whether it actually addressed the question. An answer can be perfectly grounded and still useless, so both matter.
A judge model does the scoring. I use claude-haiku-4-5 because a fast, cheap judge that runs on every single query is worth more than a perfect judge that never gets wired in.
Cost is logged next to the score, not separately. Every query records its token counts and computed USD cost in Postgres, so answer quality and answer cost are always read together.
Prompt versions are logged too. When I test a terser prompt against the current one, I can compare faithfulness and cost across versions after the fact, with real query data, instead of arguing about it up front.
(04)
How I set guardrails
Guardrails are requirements, written before the prompt. In Hive the contract is: answer only from retrieved context, and if the context does not support an answer, decline. In Hani it is: never quiz, never correct, silence is allowed.
Prompt-level first, hard validation second. The prompt spec carries the contract; a validation layer enforces it where the stakes justify the cost. Both, not one.
The refusal path is a feature. “I don’t have enough information in the knowledge base to answer that” is a tested, designed response, not an edge case the system fell into.
(05)
See it in the work
Hani applies the same discipline to a real-time voice product: a state machine instead of a script, guardrails written first, the hard limits enforced server-side.
Hive
RAG pipeline with live eval, cost logging, and a designed refusal path.
Hani
Voice-to-voice companion on the Realtime API, guardrails written first.
The rest of this site is the same work from the customer’s side. The account. The renewal. The room.