Portfolio / Personal · Product Manager · AI Engineering · 2026
Hive
A production-grade RAG knowledge assistant, built to prove the AI engineering, not just use it
“Every RAG system is a product decision, not just a technical one.”
outcomes
- Full ingest → embed → retrieve → generate pipeline shipped (Phase 2), running on pgvector + LangChain 0.3+
- Every query scored live for faithfulness and relevance with RAGAs, using claude-haiku-4-5 as the judge model
- query_log in Postgres records token counts and computed USD cost per query, next to the eval score
- Prompt versions (v1 vs v2) logged per query, so answer quality can be compared across prompts after the fact
- Context budget enforced in config: chunk_size 1000 chars, K=4 retrieval, not left to chance
- 6 teaching notebooks, one per pipeline stage, each ending in a PM takeaway
pm skills
Problem
Non-technical teams drown in scattered docs across PDFs, Notion, and Confluence. Every time someone cannot find an answer, they interrupt a subject-matter expert. That is an invisible tax on institutional knowledge. Ask an LLM directly and it hallucinates, confidently, in the exact situations where a wrong answer costs the most.
The harder problem sits underneath: "we built a RAG system" is not a finish line. Without an eval strategy you cannot tell whether a change made answers better or worse. Without guardrails the system fabricates. Without cost instrumentation you find out what it costs from the invoice.
What I Built
Hive is a RAG pipeline that ingests documents, embeds them into a vector database, and answers questions in plain language with source citations. If the knowledge base does not contain the answer, it says so instead of guessing.
What makes it a portfolio artifact rather than a demo is everything around the pipeline. Every answer is scored the moment it is produced. Every query logs what it cost. Every architectural decision lives in a notebook that explains the tradeoff and closes with what a PM should take away from it.
I designed the eval strategy, the guardrail philosophy, and the phased roadmap before writing a line of pipeline code.
Approach
Eval before features. Faithfulness asks whether the answer stuck to the retrieved context. Relevance asks whether it actually addressed the question. Both are scored live per query with RAGAs, judged by claude-haiku-4-5 because a fast, cheap judge that runs on every query beats a perfect judge that runs never.
Guardrails as requirements, not polish. The prompt spec (prompts/qa_prompt.yaml, versioned) carries an explicit contract: answer only from retrieved context, and if the context does not support an answer, decline. The refusal path is a designed feature, tested, not an edge case.
Context budget as a config, not a hope. Chunk size and retrieval K are set explicitly and enforced, so the window the model sees is a known quantity rather than whatever the last change happened to produce.
Cost is a first-class metric. query_log records token counts and computed USD cost per query, stored right next to the faithfulness and relevance scores, so quality and cost are read together.
Prompt versioning built in. PROMPT_VERSION selects which prompt spec loads, and every query records the version it ran. v2 is a terser prompt tested against v1 to see whether brevity hurts faithfulness. The data to answer that is being collected by design.
The 8 AI Engineering Fundamentals
| # | Fundamental | Where it lives in Hive | Status |
|---|---|---|---|
| 1 | LLM APIs & prompt design | Versioned prompt spec in prompts/qa_prompt.yaml | Done |
| 2 | RAG | Core pipeline: ingest → embed → retrieve → generate | Done |
| 3 | Embeddings & vector DBs | OpenAI text-embedding-3-small into pgvector | Done |
| 4 | Agent orchestration | Multi-agent query decomposition | Roadmap |
| 5 | Evaluation & observability | Live RAGAs scoring plus query_log cost tracking | Done |
| 6 | Guardrails & output validation | Prompt-level guardrail now, hard validation next | Done (Phase 1 scope) |
| 7 | Context-window management | Config-enforced chunk size and retrieval K | Done |
| 8 | Fine-tuning vs prompting | Prompt-only now, decision framework later | Roadmap |
Being explicit about what is not built yet is the credibility. A roadmap with honest exit gates is a PM artifact. A pile of half-finished features is not.
Stack
LangChain 0.3+ for orchestration, Claude claude-sonnet-4-6 for generation, OpenAI text-embedding-3-small for embeddings, pgvector on Postgres for the vector store, RAGAs for evaluation, Streamlit for the UI, Docker for the dev environment, Jupyter notebooks as the learning layer.
Outcome
Phase 2 is shipped and running: the full pipeline, live eval on every query, and cost logging. Phases 3 and 4 (hard guardrail validation, multi-agent decomposition, the fine-tune decision) are scoped with exit gates. Built to demonstrate the difference between "I use AI" and "I can scope, build, and measure an AI system."
