Category: Field Notes

I have been building a prompt optimization framework for about three months. It started as a quick utility and grew into something I now treat as core infrastructure.

The observation that drove it: prompts are code. They have bugs, regressions, performance characteristics, and upgrade paths. But most teams treat them like documentation — edit in place, no version control, no testing, no rollback plan.

What my framework does:

Stores prompts in version-controlled files. Every change is diffable. Every change can be rolled back. This alone prevents the most common prompt-engineering mistake: losing a working version when experimenting with changes.

Runs prompt changes against a frozen evaluation set. Every candidate prompt is evaluated on the same inputs as the current production prompt. Regressions are caught before deployment, not after.

Tracks cost and latency per prompt. Some prompts produce equivalent outputs at wildly different costs. Without measurement, cheaper-but-equivalent prompts never get found.

Supports structural compression. A prompt that achieves the same output with 40% fewer tokens is a real win. The framework can automatically suggest compressions and verify they preserve output quality.

What I learned: the returns on prompt optimization compound. A 20% cost reduction on every call adds up across millions of calls. The discipline of treating prompts as real engineering artifacts pays back many times over.

The hardest part has been cultural. Engineers who would never commit code without review will happily push untested prompt changes to production. Treating prompts with the same discipline as code is the prerequisite for everything else.

I have been running a three-node Kubernetes cluster at home for about a year. The unglamorous take: it has been worth it, but not for the reasons I expected going in.

What I thought I would get: scalability, high availability, orchestration for my personal workloads.

What I actually got: a testbed for production-equivalent infrastructure patterns, practical familiarity with the Kubernetes operational surface, and a place to run experiments where the orchestration layer matches what I see in work contexts.

What I did not need: high availability. My personal workloads fail occasionally, and that is fine. I do not run an SLA-bound service at home. The HA story was oversold for my use case.

What surprised me: the operational overhead is non-trivial. Upgrades, certificate rotation, storage provisioning, network debugging. I spend about 2 hours a month on cluster maintenance. For a bigger lab this would pay for itself; for a small one it is a tax you pay to live in K8s-land.

Would I do it again? Yes, but I would probably start with k3s rather than full k8s. The resource overhead of a full control plane is heavier than I needed. k3s gives you most of the operational pattern with a fraction of the overhead.

The broader point: running a home cluster is about learning, not about the cluster. If the learning maps to your professional context, it pays for itself. If it does not, it is a hobby project, and that is also fine.

A question that keeps coming up in my work: when should a system use RAG (Retrieval-Augmented Generation), and when should it just put the relevant content directly in the prompt?

After about six months of building both approaches into different systems, here is the rough decision framework I use:

Direct context wins when the relevant information is small and stable. If the content you need in the prompt fits in 4-8K tokens and does not change often, put it in a system prompt and call it a day. You will have faster responses, simpler architecture, no retrieval quality issues, and no vector database to maintain.

RAG wins when the information is large, changing, or user-specific. A large knowledge base, a per-customer dataset, or content that updates daily — these are RAG territory. The retrieval latency and infrastructure complexity are worth it because the alternative is unmaintainable.

Hybrid wins more often than either pure approach. In practice, many of my production systems use a small stable system prompt (identity, rules, common patterns) combined with RAG for dynamic content. Both techniques have different strengths; using them together is not a compromise, it is a better architecture.

The failure mode I see most often: teams defaulting to RAG for everything because it is the sophisticated-sounding choice. For small, stable knowledge bases, RAG adds complexity without adding value. Start simple. Add retrieval only when the simple approach genuinely fails.

Everyone posts about their home-lab GPU setup. Almost nobody posts about the operational reality of running one. A few notes from my own, in the spirit of transparency.

Heat is the constraint. A single workstation GPU under load produces meaningful heat in a residential room. Two of them can make a small room uncomfortable in summer. I ended up segregating the compute into a separate space with its own HVAC. This is the hidden cost nobody mentions in the build posts.

Power is the other constraint. The GPU draws power. So does everything adjacent to it. My lab is on a dedicated 20A circuit, which is enough for current use but would need upgrading if I added another GPU. Home electrical is not built for always-on compute loads.

Remote access is essential. I quickly learned that I did not want to be physically in the lab room when it was running. VPN into a bastion, SSH into the compute node, run experiments remotely. The lab is infrastructure; I do not want to sit next to it.

Cloud still wins for bursty work. I own the lab because I do a lot of steady-state experimentation. If my workload were a few 10-hour training runs per month, I would rent spot instances and save money. The home lab is worth it because I am running things most of the time, not because it is cheaper per hour.

Would I do it again? Yes. The iteration speed of running experiments on hardware I own, whenever I want, with no billing anxiety — that is worth the operational cost. But the operational cost is real, and the build-post aesthetic systematically understates it.

I have been running multi-agent orchestration experiments for about four months. The short list of observations, in order of usefulness:

Specialization beats generality, almost always. A single agent with a broad prompt and many tools is harder to reason about and less reliable than three agents with narrow prompts and fewer tools each. Breaking a task into specialized agents forces the system designer to articulate boundaries clearly, which is where most of the reliability gains come from.

The orchestrator is the hard part. The individual agents are comparatively easy. Coordinating them — deciding who runs when, what state gets passed between them, how to handle partial failures — is where the complexity lives. Most of my time has gone into orchestration logic, not agent prompts.

Determinism at the boundaries; non-determinism inside. The inputs and outputs of each agent should be schema-validated, deterministic, and logged. The reasoning inside the agent can be non-deterministic. If you let non-determinism leak into the boundaries, debugging becomes impossible.

Budget every call. Multi-agent systems generate cost at a multiple of single-agent systems. Without per-call and per-run budgets, a buggy orchestration can burn through a month of spend in an hour. I learned this by burning through a month of spend in an hour.

The counterintuitive thing: multi-agent systems are not about intelligence. They are about decomposition. The intelligence is in how you break the problem apart.

I have been running n8n for internal workflow automation for about six months. The short version: it is the least hyped and most useful piece of my personal tooling stack.

The setup: self-hosted on a modest container, connected to internal APIs, email, calendar, Slack-equivalent, and a few LLM endpoints. Nothing exotic.

What I learned:

The highest-value workflows are the ones that turn ambiguity into structure. The automation that forwards an email to a channel is fine. The workflow that ingests a messy inbound message, runs it through an LLM to extract structured fields, writes those fields to a database, and triggers a downstream decision is transformative. One of those workflows saved a colleague 6 hours a week.

LLM nodes should be used sparingly. Every workflow I have built with 3+ LLM nodes ended up flaky and expensive. Every workflow with 1 LLM node in a specific place, surrounded by deterministic logic, worked well. The LLM is a classifier or a summarizer; the rest is if-then.

Monitoring is non-negotiable. I learned this the expensive way. A workflow silently failed for three weeks before I noticed. Now every workflow writes success/failure signals to a dashboard, and I get pinged on failure patterns. Without this, you are building technical debt and calling it automation.

The biggest insight: workflow automation is a research practice, not just a productivity practice. The act of building a workflow forces me to articulate what a process actually does, which surfaces assumptions I did not know I was making. Half the value of automating something is the clarity you gain during the automation.

I have been prototyping a zero-trust identity model for AI agents for about two months. The question: how do you give an agent access to what it legitimately needs without exposing what it should not see?

The architectural principles are not new — information silos, opaque identifiers, field-level encryption — but applying them to agent systems surfaces specific design choices that matter.

Opaque identifiers prevent cross-context reasoning. If agent A refers to a customer as cust_7a3f and agent B refers to the same customer as cx_b91d, neither agent can correlate information across boundaries unless a trusted broker authorizes the correlation. Information silos enforced at the identifier layer are stronger than silos enforced at the access-control layer, because there is nothing to join on.

Field-level encryption beats row-level access control. Row-level access control protects against unauthorized row reads. It does not protect against an authorized agent that leaks the row content downstream. Field-level encryption ensures an agent can only decrypt the fields it specifically has keys for, even if it can read the whole row.

Audit everything, especially the boring stuff. The interesting attacks on agent systems are usually boring: an agent has been given wider access than it needs, and something inside the agent\\u2019s logic exploits that. Comprehensive audit logs — every request, every field accessed, every tool called — are the only way to reconstruct what went wrong after the fact.

Still early in the work. What I can say so far: this is one of the areas where standard web-security patterns need adaptation for AI, not just adoption. The threat model is different because the actor is non-deterministic.

Running a LoRA fine-tune on a consumer GPU at home turns out to be more forgiving than the AI Twitter discourse would suggest — and more informative about what actually matters for production than a cloud run ever is.

The setup: a single-GPU box with 24GB VRAM, a 7B-parameter open-source base model, and a target corpus of about 30K domain-specific instruction pairs. Total training time for a meaningful LoRA adapter: roughly 4 hours. Total cost: electricity.

Three things surprised me, in order:

1. Data quality dominates everything else. I spent two weeks preparing training pairs and 4 hours running the fine-tune. If I had it to do over, I would spend three weeks on data and 4 hours on compute. The returns on cleaner, more specific, more consistent training data are enormous. The returns on another hyperparameter sweep are marginal.

2. Evaluation is harder than training. The fine-tune finished in an afternoon. Evaluating whether it was actually better than the base model on realistic tasks took a week. Most public benchmarks do not test what I actually wanted the model to do, so I ended up hand-building a task-specific eval. That work was more valuable than the training itself.

3. Local fine-tuning changes the economics of experimentation. When each run costs nothing but time and a few kilowatt-hours, you iterate more. I ran 14 experiments in a month. That would have cost thousands in cloud compute. The fact that I could try genuinely bad ideas quickly is why I eventually found the good ones.

Next: move to a larger base model and a more rigorous eval harness. The research practice is catching up to the infrastructure.