Databricks exists to close gaps like this one. Not by making the old setup faster, but by questioning why transactional systems (the ones handling checkouts) and analytical systems (the ones tracking trends) need to be separate in the first place.

What Databricks actually is

Databricks sells one core bet: your operational data and your analytical data should live in one place, governed by one set of rules, queryable by both humans and now autonomous agents, without anyone building a pipeline to shuttle data between them.

It started as a lakehouse company. The pitch was that you could get warehouse-style structure, schemas, ACID transactions, fast SQL, on top of cheap object storage instead of paying warehouse prices for warehouse lock-in. That was already a real improvement over the old split between data lakes (cheap, messy, slow) and data warehouses (expensive, structured, fast). But a lakehouse still assumed your transactional database lived somewhere else and got copied in on a schedule.

The current platform goes further. It folds the transactional database itself into the same storage, governs every model and every tool call through one gateway, builds a live map of what your business terms actually mean, and gives autonomous coding agents a shared, policed place to operate. This is the thesis: fragmentation is the problem, not the volume of data.

The Problem Databricks Solves

For four decades, systems that need instant reads and writes, like charging a credit card (OLTP - Online Transaction Processing ), and systems that need to scan millions of rows for a trend, like this month’s fraud rate by region (OLAP - Online Analytical Processing), have needed different physical storage.

• Row-based storage is fast for grabbing one record.

• Column-based storage is fast for scanning one field across millions of records.

You can’t optimize a single copy of data for both access patterns at once, so companies kept two copies and wrote pipelines to keep them in sync. To understand the problem and how Databricks solves for it with a sketch, keep reading!

Read more

A front desk fixes this by becoming the one place every request passes through. It checks your identity once. It knows your room charges. It routes you to the right department. It keeps a single log the whole hotel can be audited against.

Every application that wants to call GPT-4, Claude, or a self-hosted Llama model checks in there first. The gateway verifies who’s asking, checks whether they’ve got budget left, decides which “room” actually fits the request, and writes it all down. Nothing reaches an external provider without going through that desk. This matters because the alternative is failure mode at nearly every company scaling past its first LLM prototype. An autonomous agent left running over a long weekend, spawning sub-agents that spawn more sub-agents, each one re-reading the same document and asking a reasoning model to summarize it again, can burn through a quarter’s compute budget in three days. Nobody did anything malicious. There was simply no gate between “an agent wants to call a model” and “the model gets called.”

It counts how many people walk in per minute and blocks obvious troublemakers, but has no idea what a room costs or what you’re planning once you’re inside. LLM traffic needs a front desk, not a doorman, because the “room charges” here are token costs that swing by orders of magnitude between a one-line completion and a fifty-turn agent loop.

The Two-Tier AI Gateway: Four Design pattern

The simplest setup is a single, centralized gateway sitting at the perimeter of the whole company. Every request, from every team, funnels through one cluster. It’s easy to reason about and gives you one audit trail, but it shares the weakness of a hotel with a single front desk serving fifty floors: everyone queues behind the same counter, and if that counter goes down, the whole building stops checking anyone in.

Large organizations tend to split the job into two tiers instead.

Read more

The failure mode is dramatic, slow, and wrong. The agent forgets what it found three tool calls ago. It runs out of context budget halfway through. Nobody catches the mistake because there’s no second agent checking the first one’s work.

This is where multi-agent systems come in, and where a lot of teams either overcorrect into unnecessary complexity or undercorrect and keep duct-taping a single agent that was never built for the job.

We will go through each architecture one-by-one and at the end I will give you a complete visual of the difference!

What a single agent actually is

Picture a solo cook running a small food truck. One person takes the order, checks the fridge, chops the vegetables, cooks the dish, plates it, and hands it over. There’s no handoff, no division of labor. It works beautifully for a short menu.

A single-agent system works the same way. One reasoning loop, usually built around an LLM, sits at the center. It receives a query, decides what it needs, calls a tool such as web search or a database, reads the result, and decides again whether it has enough information or needs another round. This loop repeats until the agent is confident enough to answer.

The architecture is simple: one control loop, one model doing the reasoning, one place where state lives. You can trace exactly why it made a decision because there’s only one decision-maker.

Single agent flow: User query → Agent (reason, act, repeat) ↔ Tool (web search, database) → Response. The agent and tool loop back and forth until the agent has enough information to answer.

That loop is also exactly where things fall apart. The agent has one context window, one memory, one thread of reasoning trying to hold research, analysis, and writing all at once. Ask it to do three unrelated things and it does them one after another, dragging all the baggage from step one into step three.

What multi-agent actually is

Now picture a real kitchen during service. A head chef doesn’t cook every dish personally. They look at the order, decide which station handles what, and let the grill cook and the pastry chef work in parallel. Someone expedites at the pass, checking every plate before it goes out. Nobody does everything; everyone does one thing well, and the head chef keeps it all moving in the right order.

Read more

And I keep watching the same thing happen.

A smart, capable engineer walks into a room with a customer. The customer says something like “we need faster deployments.” And the engineer because they’re smart, because they’re capable, because that’s what they’ve been trained to do their whole career jumps straight to CI/CD.

The engagement dies slowly after that. Nobody says anything. Everybody stays polite. But it’s dead.

The engineers who win do something different

They have a process for the part before the toolbox/demo/product. Before the architecture diagram. Before the sprint plan. Before any of it.

I call it the SCOPE Framework.

S — Study
Before you talk to anyone, research the company. Industry, business model, what their customers are actually complaining about. You should walk in already knowing something not sitting there waiting to be briefed.

C — Connect
Push for the executive conversation. Everything below that level comes pre-filtered. Skip it, and you’ll get handed a project someone already decided on. You’ll build it well. It still won’t be the right thing.

O — Observe
Walk the entire process with them the tools, the teams, where things actually slow down. That “we need faster deployments” ask? Half the time the real problem is regression testing eating 40% of the cycle. Nobody says that in the first conversation. You only find it by watching.

P — People
The skeptical CTO. The engineer who doesn’t want you anywhere near their system. The VP whose timeline doesn’t add up. Same project, completely different conversations, completely different objections — and you have to hold all of them at once.

E — Execute
Scope to real outcomes, not deliverables. Hand off in a way that doesn’t fall apart the moment you leave the room.

Most engineers skip straight to E. That’s the whole problem.

Come do the reps

On September 27th, I’m teaching this live — a full day, real discovery reps, real enterprise scenarios. Not a slide deck you could’ve read at home.

If FDE is a role you’re working toward, this is where you build the muscle.

→ Save your seat with code EARLYBIRD200: https://maven.com/pvergadia/storytelling-for-fde?promoCode=EARLYBIRD200

Register $200 off

Questions first? Grab 15 minutes with me: https://calendar.app.google/eGQ3gEdSnPTnAm3c6

Here is my FDE video series:

— Priyanka

P.S. Have a specific scenario you’re stuck on a stakeholder situation, a scoping problem, a career transition question? Bring it. There’s an open Q&A at the end and I’ll work through real ones, live.

What you own is a building shaped like a hotel. A hotel is what happens when someone answers the phone at 11pm, cleans a room between guests, runs a reservation system that does not double-book the same suite, and sends a bill that matches what the guest actually used. None of that comes bundled with the real estate.

This is exactly the situation power owners are walking into right now, except the building is a data center and the guests are GPU workloads.

Bare Metal Is Not a Product

Picture a family energy company in West Texas. Three generations, mostly gas peakers, a wind stake from a decade back. They have 40 megawatts that never gets fully monetized because the grid does not always want what they can produce, when they can produce it.

Then the calls start. A broker wants to lease the pad. A reseller wants to talk GPU orders. Somebody’s cousin in private equity sends a model showing what an H200 rents for per hour, times 8,000 hours a year, and the number at the bottom is large enough that the room goes quiet.

So they run the math. Power, check. Land, check. Shell and cooling, solvable. Every physical line item has a vendor and a quote.

Then someone asks the only question that matters. What are we selling, exactly?

The building is not the business. Nobody pays to stand in the lobby. The customer never touches your substation, they touch an API, a console, a login, and an invoice, and everything between the power and that invoice is the part nobody quotes you for.

GPU Capacity Constraints Are Real

Demand for accelerated compute is running ahead of what can be brought online, and the good large-scale capacity is mostly spoken for. That gap is pulling capital toward whoever controls power, which is why power-site developers and well-capitalized individuals are suddenly being courted as future cloud operators.

They are right about the scarcity. Power is the binding constraint.

The wrong conclusion is the arithmetic that follows: power plus hardware equals cloud platform. It gets you a depreciating asset with a clock running on it and no mechanism for turning it into revenue. It gets you infrastructure, not a service.

Six Systems You Have to Build Before You Can Bill Anyone

1. Provisioning by hand doesn’t scale past customer ten.

If a human touches a machine every time a customer signs up, you have built colocation with worse margins. The eleventh customer, on a Saturday, will find out.

2. Tenant isolation on a GPU fabric is a distributed systems problem.

RDMA fabrics and shared storage mean two tenants sharing hardware is not like two VMs on a hypervisor. Getting it wrong is not a performance issue, it’s a disclosure event.

3. Reclamation is a security boundary.

When a tenant releases a node, something has to guarantee GPU memory and firmware state are clean before the next tenant lands. This is the likeliest way one customer’s model weights leak into another’s environment.

4. Utilization is a scheduling problem with balance-sheet consequences.

An idle GPU depreciates on schedule regardless of use. Fractional allocation (MIG, time-slicing) is what lets you sell two GPUs to a customer instead of turning them away because your smallest unit is a full node.

5. Telemetry and billing are different systems.

Graphing GPU usage gets you a fifth of the way there. Between a metric and an invoice sit rating rules, SKUs, quotas, and a number that survives a finance team’s scrutiny.

6. Multi-site is the default topology.

You likely have three sites of 8-15 megawatts, not one campus of 300. Customers won’t tolerate three consoles and three support paths.

These aren’t exotic problems. They’re the ordinary cost of being an operator, and hyperscalers spent a decade and thousands of engineers learning them.

The Service Stack: Every Layer Above Bare Metal Is Software

The physical asset at the bottom of this stack and the top is identical. Same power, same GPUs. What changes as you climb is how much software sits between your hardware and your customer, and with it, who owns the brand, the price, and the relationship.

Rung one is a real business. Leasing power is fast, low risk, and produces revenue while you’re still learning what a tenant even is. Some owners should stop there.

But understand the trade. At rung one you’re infrastructure supply for someone else’s platform. Your upside is capped at the lease terms, and the customer belongs to them.

Build vs. Buy: The Trade-offs

Lease the power, host someone else’s gear

✅ Fast revenue, no operational risk.

❌ Fails if your upside stays capped while compute services get more valuable around you.

⚖️ You give up the customer relationship and any path upward without starting over.

Rent raw GPU capacity yourself

✅ Fastest way to be a real operator.

❌ Fails when a competitor with cheaper power does the same thing and price is the only lever left.

⚖️ No differentiation. GPU hours are a commodity.

Build the control plane in house

✅ Wins with a real platform team and a multi-year horizon.

❌ Fails on timing: 12-24 months of building is 12-24 months of GPUs depreciating without revenue.

⚖️ You give up time-to-first-dollar, the expensive thing in this market.

Buy the control plane, operate the platform yourself

✅ Wins when you want the customer and would rather spend engineering time on service design than rebuilding tenant isolation.

❌ Fails if you treat the platform as magic and skip building real operations around it.

⚖️ Some architectural freedom, one vendor dependency.

My Recommendation: Buy the Control Plane, Own the Operations.

Rafay is your stack

Two clocks start the moment your hardware is racked. One is depreciation, running from day zero. The other is revenue, which doesn’t start until a customer can self-serve something. Every month spent building provisioning and metering from scratch is a month where only the first clock runs.

This is where we need a platform layer. Rafay is one of the products that packages the middle column above: bare metal provisioning, GPU lifecycle and reclamation, multi-tenancy, MIG and time-slicing, metering that connects to billing, and a single control point across sites that aren’t physically together. What matters is that this layer now exists as a purchasable product, which wasn’t true at all five years ago. That’s the difference between first revenue in a quarter versus a fiscal year.

Want to learn more about the solutions? Attend the Rafay AI Infrastructure Leadership Summit in Barcelona Sept 8-10 for free.

Request to Join

Decision Guide

If you have power but no site or capital, stay at rung one and revisit in 18 months. If you have power and financing but zero technical staff, aim for rung two or three on a bought control plane. If you have a platform team with genuinely unusual requirements, build only the pieces that are actually unusual and buy the rest. If you run three or more small sites, treat federation as a hard requirement, not a nice-to-have, since nothing that treats each site as an island will work. If you want to reach inference or token-metered services eventually, choose your foundation for that now, because retrofitting metering and multi-tenancy later is a rebuild, not an upgrade. And if you want to stay a landlord, that’s a real answer too, as long as it’s a decision and not a default.

The opportunity isn’t to supply capacity to someone else’s cloud. It’s to decide, deliberately, how far up that stack you intend to own. Most people won’t decide. They’ll drift into rung one because it’s the option that arrives with a contract already drafted.

This blog post is sponsored by Rafay, thanks for partnering with me on this post

AI agents can spin up a working feature in minutes. But someone still has to look at what got shipped and decide: is this safe, is this right, does this actually do what we meant? That job hasn’t gotten any faster. If anything it’s gotten harder, because the code AI writes doesn’t fail the way human code fails. No missing try/catch blocks or sloppy null checks. The mistakes are architectural: unnecessary complexity, quiet duplication, changes that technically work but drift from what the system actually needed. You can’t catch that by skimming a diff.

So the bottleneck didn’t disappear when AI got good at writing code. It just moved. Straight into code review.

I sat down with Harjot Gill, co-founder and CEO of CodeRabbit, to dig into exactly this. Harjot’s been building in this gap since before “agent” was even a word people used, and the conversation ended up covering everything from pricing strategy to why GitHub isn’t going anywhere. Here’s the short version.

Watch the full episode on Youtube

Q: Why did you start CodeRabbit? Same rhyme, different verse. In his last startup (Netsil), the bottleneck moved from infrastructure to observability once cloud/Kubernetes made deploying easy. Now that AI has made writing code easy, the bottleneck has moved downstream to review and explainability.

Q: What does CodeRabbit actually do? It’s a background agent, not a chatbot. It fires on a GitHub webhook the moment a PR opens, no one has to remember to invoke it. It clones the repo, builds a live code graph, runs static analysis to ground itself, then fans out across an ensemble of models (small ones for context, expensive ones for reasoning) before returning a review in 10-15 minutes.

Q: Why not let users pick their own model? CodeRabbit never exposed a model picker, even back in 2023 when everyone did. Instead they invested in the harness: semantic code graphs, blast-radius analysis, and tight context pruning, so the expensive reasoning model only sees what actually matters.

Q: How did they get early traction? Free for open-source maintainers on public repos. That gave visibility (enterprise engineers watched it work on repos they already used), fed a feedback loop from real public data, and built trust before asking anyone to pay.

Q: How do they keep pricing flat despite expensive models? Heavy engineering investment specifically to make a flat, affordable price point possible at agentic-system scale, deliberately undercutting a market used to premium, per-seat security tooling.

Q: What’s changing in engineering teams? Roles are blending. PMs prototype and touch code directly. “Taste” and systems-thinking now matter more than raw coding speed. CodeRabbit runs cross-functional “war rooms” where product, design, and engineering all work in the same session.

Q: Will GitHub stay the center of gravity? Gill thinks yes. Every competitor trying to unseat it “ends up looking like GitHub.” He compares displacing it to Twitter vs. Bluesky: technically possible, but the ecosystem gravity is enormous.

Q: What’s the bigger threat than competition? Uncertainty about whether the whole review workflow survives disruption at all. Competing on quality is easy. The real question circulating the industry is whether some future model collapses the need for review entirely.

Q: Where’s CodeRabbit expanding next? Beyond reviews into Jira, Linear, and Slack; treating “collaborative AI for teams” as the bigger category, with code review as just one activity inside it.

What stuck with me most from this one: everyone’s chasing the coding agent, but Harjot’s betting the real value sits one layer downstream, in the choke point where humans still have to decide what’s actually safe to ship. Given how fast the space is moving, that might be the most durable bet on the board right now.

Full Transcript

Priyanka: Welcome to this episode of the Cloud Girl podcast. Today I have Harjot Gill with me. Harjot, welcome to the show.

Harjot: Yeah, thanks for having me here, Priyanka. Thank you for sharing the CodeRabbit space and inviting me to the studio.

Priyanka: Thank you. But why don’t we start with just a little bit of an introduction of who you are and what is CodeRabbit?

Harjot: So I’m co-founder CEO of CodeRabbit, which as you know is a leading provider of AI code reviews, essentially providing automatic validation and explainability for all this agentic software development stack that everyone’s adopting. Company is roughly two and a half years old, but we have several hundred thousand developers who use the product, love the product on a daily, monthly basis.

Priyanka: Yeah. So tell me a little bit about your background because it’s very interesting. You’ve had multiple exits. Can you take us through some of that and then come to CodeRabbit?

Harjot: Yeah. So CodeRabbit has been dear to my heart, being an engineering leader as well as an entrepreneur in the space. This is my third startup. My first startup was a spin out from University of Pennsylvania, where you and I first met. I was a TA back almost a decade back. That company was called Netzil, and that was in the observability space. Netzil was “listen” spelled backwards, and at that time it was a very similar situation where cloud computing was taking off, all these orchestration systems like Kubernetes, microservices, Docker, and there was a lot of chaos. Humans were finding it hard to understand these fast-moving environments and a lot of the bottleneck shifted.

Priyanka: Yeah, like the second order effect, right?

Harjot: And now fast forward to now, right, where we are seeing the same situation. There’s a lot of chaos. The agents are creating massive amounts of code, and as a second order effect, explainability is becoming a bottleneck. As we say, history doesn’t repeat but it rhymes. For me it certainly does. When I saw the opportunity here I could just relate it back to my first startup.

Priyanka: Very interesting, from cloud to here in the AI space. And you’re so right, the customers, consumers I speak with, everybody is battling the same problem. Now we have 90% of the code written by AI, but my senior engineers, all of their time is spent in reviews. So how did the idea come about, and from there to today, the journey of CodeRabbit and the evolution?

Harjot: Yeah, I mean this happened in my second startup, which I kind of missed talking about. So the first startup exited to Nutanix, which is a big infrastructure company, and I was there for a few years. Started another company in the load management, reliability management space, because at that time some of this webscale computing was taking off, but right around, it was during COVID, and right after that GPT-3.5 arrived, GPT-4 arrived. And the killer app at that point was GitHub Copilot, and there was no Cursor by the way. So GitHub Copilot type predates even ChatGPT. And as soon as I saw that, it was very clear to me that this is an inflection point, the entire world is going to change.

The bottleneck had been code reviews. Coding now the coding is getting automated, the reviews would certainly become the next bottleneck. And running the small team I could clearly see, we were a remote team because we started during COVID, so we had people in Eastern Europe, India, the US, Canada, bunch of places, and they were moving at a startup pace, shipping multiple times a day, and code reviews were already a pain. So we had automated a lot of our DevOps processes, static analysis tools, people were trying to open small PRs, doing stacked pull requests. We even had an internal tool we wrote, this was even before this Graphite company, Graphite had released a CLI, we got early access, but we also had our own tool doing similar stuff. That tool was helping with code reviews but it wasn’t fundamentally solving the problem, humans still had to understand the code changes. Having small changes makes sense but then you sometimes lose the bigger picture, so there were always trade-offs. But when AI arrived it was very clear everything is going to change. And that’s where CodeRabbit started.

It’s my other co-founder who started the company. I came on board after the second startup struggled to raise more capital in that space, where all the VCs lost interest. And we were lucky, we kind of attacked and saw this pain point before others realized it would hit them. Now two and a half years forward, it’s very clear that the biggest bottleneck in SDLC is now code reviews. Code generation is automatic.

Priyanka: It’s realizations now for a lot of the enterprises because the adoption has increased with AI now, but you saw the space before it even emerged, right, and it’s very likely a lot of the value in the agentic coding space is going to accrue in this bottleneck.

Harjot: Right. So code generation more and more is going to become commoditized, and even the models, you’re going to see open source models are inevitable. And even beyond a certain intelligence level, code generation doesn’t benefit. The two things that benefit are planning and reviews, which are more reasoning-heavy. But if you’re talking about just raw writing lines of code, even simpler open source models are doing a great job, as you’ve seen with Cursor Composer 2, may not be as great as Opus, but it’s really great at writing code. And you’re going to see more of that. So very likely the bottlenecks are going to be around two things. One is how do I automate some of these validation steps, and the second is explainability, because you still need to build trust in these outputs very quickly, to say hey is this all garbage fluff I could just close a PR, or is this something worth merging.

Priyanka: I’ve described this to people as, so we had a linear process of creation with planning and test development and testing, and now we are in the process, you beautifully said just now, where it seems like an hourglass, where coding itself is the smallest part of this hourglass, and then the planning and requirements gathering, there are tools needed and innovation there, and then there’s the lower part of the hourglass which is the review side of things. As I talk to more and more CTOs, CIOs that are on this journey of yes our engineers are now starting to use AI coding tools, what do you see as the problem beyond the reviews? Where do you see the human element continuing to play a huge factor?

Harjot: It’s kind of a good balance. Some of the things AI is already better than humans at, when it comes to writing quality code or even finding issues in the code, but you still need explainability in terms of whether these changes make sense or not. Are they going to increase the entropy of my codebase? You want to understand the blast radius, whether core flows or some invariants have been rolled back, because you sometimes don’t know how your prompts ended up. The other bottleneck, you’re right, is upstream, where the nature of human review is going to shift from line-by-line code reviews to actually reviewing the specification and deciding what to build as a team. So I kind of see this more like a hamburger. The inner loop is automated, and that inner loop is also very much single player right now. Most of these coding agents are running on your local workstation. If you’re an IDE person you’re using Cursor, if you’re in the terminal you’re probably using Claude Code or Open Code, and if you’re using desktop apps, Codex desktop app is getting pretty popular. Different people have different preferences for these surfaces, but increasingly these tools are still local. Background agents still haven’t taken off in a meaningful way. Then what’s left as the bottleneck is on the outer loops, which are team-based activities. Upstream you have Jira, Linear, Slack, Notion, where you need to collaborate on what to build, and then downstream you have version control systems like GitHub, which is under too much tremendous stress right now, as you at Microsoft probably know all about. So the human review is now moving from a lower-level understanding of code, the tank battle, to an air battle, you have to understand higher abstractions, at the same time shift left a little. You have to decide as a team what to build, describe your destination more accurately, then these agents can run for longer. In fact the longer they run, if you don’t describe the destination, they’re almost always going to go off track.

Priyanka: And those PRs never get merged. So many pull requests get opened and they don’t get merged.

Harjot: So there has to be some choke point upstream. There is no choke point right now. The only choke point we have is GitHub, which by the way I feel is late. Once you bring in automations and triggers, even agents opening PRs on each support ticket, then the problem is not just automated reviews, you still have to triage and validate them. So there has to be some choke point upstream, some sort of repository of plans or prompts, has to exist. No one has got it right yet.

Priyanka: No, and everybody’s experimenting, because generating code is so easy and fast, my experimentation earlier used to be, let’s plan more and we will build what we really want to, but now it can be, we’ll build three things and then we’ll decide which one we should ship. What do you see or think with your customers in terms of shifting into that type of thinking, where building is so easy that let’s just experiment with a lot more and then decide?

Harjot: I do think there’s some problems with that approach, because you still need, you’ve owned businesses, you know, you need to decide what you are going to build. There are different degrees of AI automation in these codebases. There are always going to be codebases touching real customers that are mature, and you won’t be reckless, we’ve seen how AWS is now preventing engineers from being reckless with those agents, because outages can be crazy, and if you can’t understand the changes you shouldn’t be introducing AI. So there’s always going to be more of that. But then there are lower-risk areas, especially around prototyping, which is kind of a golden age. Now PMs themselves are becoming designers, they can quickly show a working prototype, and you can always optimize the backend and scale it later. But getting to a quick prototype has removed so many barriers that you can actually see the ideas very quickly and even experiment with them in the real world, whether it’s going to stick or not. That’s how the model labs are shipping, they have a very newsroom-style marketing on Twitter, they’re building public and shipping every day. So yes, the velocity on shipping has changed, and the ideas that stick, that’s the main thing, the learning, and then you can double down. The code itself has zero value now. The other revolution we’ve seen is internal tools, there were so many things people wanted to do internally, never could find resources, now people are stepping up and building amazing internal tools, support teams are building their own tools, and sometimes those UIs look even better than your main product.

It has generally removed the barrier where now multiple personas can touch the code. Earlier they were all on the periphery guiding the engineers, now everyone can contribute their own taste to the vision, to the idea. And how you run an engineering team is getting evolved. Even at CodeRabbit we were running teams in a very different way two years back, when it was still tab completion. In the last 12 months it’s just been amazing, going from a high-level prompt to large-scale changes. Even I personally, I’m not in a code editor anymore, for the first time in my life. I’ve been coding roughly 30 years now, since ‘94, ‘95, and I would say this is the first time I’m not in a code editor.

Priyanka: Really? What is your tool of choice?

Harjot: I love the Codex desktop app. This is the first time I’ve been able to stop using Neovim. I was on Neovim earlier, then I just jumped to the Codex desktop app and I haven’t looked back since. I’ve been keeping it busy 24/7 pretty much, and it’s so fun to prototype.

Priyanka: You talked about the engineering teams, I talk to large organizations where they’re worried about how we’re going to structure our engineering teams over time, because they’re still early in the thinking. Would you share a little bit about how you’ve evolved the engineering team in the last, because you’re at the very top of seeing these things very early, how have you evolved the CodeRabbit engineering team with the use of AI?

Harjot: Yeah, it’s evolved a lot. You’re looking for a different kind of talent as well, talent which has good taste. Of course you still need some basic principles, systems design and so on, so you can validate and ask the right questions to the agent. I talk to the agent a lot, sometimes to validate the work you just talk to it. But now more people can be part of the process. Now we’re creating pods, we call them “war rooms,” we put people from product mindset, from design, engineering, even some DevRel in the same room, so multiple people can touch the product. When the PM is opening a pull request they’re looking at their things, they want to add a banner here or something, and designers can go and change icons without talking to any engineer. Otherwise earlier there used to be Figma mockups getting implemented, now you can do it live. So multiple people are able to go and touch the artifact now, which is the code. And some people are good with systems design, will do great distributed systems coding, and we do a lot of distributed systems as well, hard stuff. But also the style is evolving, things in the past that were too hard, people were not investing a lot in simulations or formal methods, we for example recently launched a Slack bot, we explored using TLA, temporal logic of actions. We modeled everything using AI and then the code came, wrote the code based on that, once we could prove it’s going to be correct. So the style of engineering has evolved, the things that used to be hard in the past are so easy, so why not do them, in fact you need all those guardrails for AI to prove it has done the right thing. And the bottlenecks have also shifted outside engineering, our product velocity has exceeded our ability to talk about these products. That’s why we started to embed devrel into the same room as well, because they need to go out and talk about this the day it comes out.

Priyanka: When you think about the work and the bottlenecks shifting, do you still think in the boundaries of, I’ve heard very different views from different companies and different people, have you just allowed more things to be done by the same individual, because now they can do different things beyond the scope of what they were able to do, like the PMs you described, they can prototype and actually build a feature? Has that led to changing the team structure at all, or is it fluid? I’m curious what you’ve seen.

Harjot: Some things have changed. Earlier, ops, people would rely on another team to set up services or Terraform, now developers are able to take things end to end much further, even PMs are able to take stuff much further along. So there are still some handoffs, but each individual is becoming more and more full stack in things they were also not doing in the past. Yes. So that clearly is happening, ops is another area where we’re seeing a lot more people enabled with these agents, they can go set things up. Of course you have to put all the guardrails so these agents don’t delete a database or something. But still it’s great, how quickly the single person can now go and do things.

Priyanka: So when you’re hiring, you’re looking at the ability to expand beyond what they have currently done. How are you hiring?

Harjot: One thing is, in the past even location, let’s talk about that, earlier coding was a bottleneck, so you would go to all kinds of locations, my second startup even Eastern Europe, India, because you wouldn’t find all the talent you need in SF given the competition, the craziness. Now some of these constraints have gone away, now it’s about product being the bottleneck, knowing what to build, so we’re increasingly hiring more local talent for instance. The other thing has been the taste, which is hard to measure. Earlier I’d interview people and look at how they work in a code editor, whether they’re comfortable with the tools, that gives you a good hint on whether they’ve done work in large-scale codebases or not. Sometimes we’d do a one-day session where they build something and we see it, but now we don’t know if that’s the skill we really need. Some of the system knowledge, they still have to piece together a puzzle, solve a murder mystery, because they’re debugging code they haven’t seen, they need to be able to ask the right questions, have the right hypothesis, that’s still needed. At the same time, taste, which is hard to describe, you have to have people who are really passionate about something like dev tools, you need to know that workflow to build it in the right way. Code review is one of them, developers are a very opinionated lot, so if you aren’t a developer yourself you have no business building a product in this space. And a lot of PMs we’re hiring are also coming from developer backgrounds. Outside of that, you still need people who can go and talk to the world, because that’s not something that’s automated, feedback is oxygen, all that has to come in too, bringing the people skills, the relationships with customers, those are still the real bottlenecks. Now even a small, highly skilled engineering team with breadth of knowledge and taste can keep a large sales force or go-to-market busy, unlike in the past.

Priyanka: We didn’t talk much about this, so I want to highlight a little bit of how CodeRabbit became CodeRabbit, or got the early customers. I think it was a phenomenal DevRel mindset, with open source as one of the front-and-center repos. Can you talk about that? I get a lot of questions from founders of AI startups, how do we get customers, and this is a really classic and amazing way of building a trusted base of people who want to talk about you. What has CodeRabbit done in this journey?

Harjot: There were many things, and we continue to do many of them. One of the things that was very clear in the early days of AI, not many people had experienced advanced models, tab completion was one way they were consuming, but we were always a reasoning-heavy system, so you had to see it to believe it. And the idea at that time was how do we get it into the hands of as many people as possible. It was a habit change we wanted to bring to the world, this was a new, serious workflow, no one was using AI in code reviews before CodeRabbit, so it needed a new habit. The biggest thing was, when would people even accept it, because at that time people were trying to shove AI into everything and there was a lot of pushback. So one of the things we did really well is we made the product accessible to a lot of open source maintainers. We said, these people have a genuine pain point, they’re doing a thankless job, and even as an engineering leader I consume so many of these libraries, I wanted a way of giving back. So we made the product free for open source maintainers on public repositories. That did two things really well. One, you could now see the product in action, so a lot of developers at bigger companies, not maintainers themselves, but looking at the pull requests, could see the product in action, see what it does, and then want to bring it into their organization, that’s visibility, distribution. The other thing it did for us was improve the product very rapidly, because public data you could see, unlike private data. So we set up a feedback loop where we could see how people were using the product in open source and quickly improve the harness, the context assembly, and understand the effectiveness of the system. The other thing we continue to do is pricing innovation. We’re one of the few products trying to make this very affordable, so anyone in any kind of developing nation can afford the product at the price point. We worked hard, there’s a lot of engineering involved in keeping a flat price for an agentic system, it’s not a trivial problem. It took a lot of engineering to make sure we’re not making it a very premium product, we’re not selling at hundreds of thousands of dollars per seat only to larger enterprise, because traditionally security scanning tools were very expensive, going top-down to the high-end market, and we wanted to be pervasive everywhere. Now installed on five to six million repositories, we recently crossed Snyk in number of installs. Pricing-wise, we made it so affordable that we are everywhere.

Priyanka: This is a masterclass in how you get the product to a good level of adoption. You set the ceiling, even the competitors had to meet you at this price point.

Harjot: Right, we kind of gave a very hard problem to even the competition and ourselves, to make sure we could meet the market at that same price point.

Priyanka: How do you sustain that though? Because as I understand it, you’re using the foundational models and building on top, and those aren’t cheap. Talk to me about how you sustain that cost, that price, and the systems built around it.

Harjot: The magic is in the harness. If you’re going to do a brute force way of just throwing agents without a lot of bearing, they’re going to spend a lot of tokens just understanding which repository they’re in and finding their way. So we invested a lot, first of all we’re one of the very few companies that never let the user choose a model. Think about back in 2023, every product had a chat interface with a dropdown to choose a model, we never had that. We were always using an ensemble of models from day one, small models for context assembly, expensive models for reasoning. The other thing we did really well is around code graphs and semantic analysis, how do you make sure you’re giving the model enough context, so they’re not flying blind, you’re giving them the means to get to the answer quickly. We spent a lot of time on semantic analysis of code understanding, the blast radius, if you’re going to change a few files, how do I bring in the right definitions or references, expand the fog of war a little to bring in other context, what could be impacted. The other thing we did really well, we were one of the very first agentic tools, there was no term “agent,” no tool calling, we predate function calling. We were doing sandboxes and running shell scripts in the sandbox to do code review, generating code to do code review, getting evidence on whether the problem we’re hypothesizing is even real. Back in 2023 we started doing sub-agents, GPT-4 I think, GPT-3.5 was a sub-agent at that time. So we’ve always had this reasoning model with a lot of sub-agents underneath, an architecture that’s grown a lot, eight or ten different models now. We did it very effectively and cost effectively. A lot of engineering goes into making something fit into a flat price point that’s predictable while being so powerful, it wasn’t trivial.

Priyanka: I can imagine. So when you do a code review, let’s walk through the journey. Let’s say I tell CodeRabbit, look at this particular PR and do a review on it. What happens first? Are you taking the codebase, creating a sandbox? Walk me through the behind the scenes.

Harjot: CodeRabbit is a background agent, one of the very few products that doesn’t need a human to remember to use it. Most products are chat-based, you go talk to the agent and give it work. We were one of the first agents that just worked as soon as you open a pull request, there’s no other human input, it’s a webhook that triggers an agent. Chat is more of a secondary feature, there’s chat as well, but that’s not the primary interface into CodeRabbit. As soon as we get a webhook, it’s a background agent system that runs a sandbox, clones your code, builds a code graph, a live code graph, because staleness is a real problem, so we do it live, and it’s also very efficient the way we built it, all Rust code. Then we run a bunch of static analysis tools to ground the agent a little, some low-hanging stuff like secret keys injected. The other thing we’re doing is a lot of context preparation, it’s not a single model call, there’s a variety of things like map-reduce style workflows, scatter-gather patterns, parallelizing and then bringing information together to get a bigger picture. We do a lot of that, a variety of models, a lot of people working on that project now. There’s a lot of context preparation, and then the reasoning models kick in. We try to make it as efficient as possible, every token has a very high cost when it goes to the reasoning model, so we want that context to be really relevant. Prune as much as we can, because the more context you throw at the reasoning model, the more off track they go. So you have to be sure this is the right context needed for the code review, and after that it’s never going to be enough, because the codebase is massive, so the reasoning models have an escape hatch to go explore the code further. Surface-level issues they can flag, but then there are issues they feel could be a problem but need more evidence, and they can go run a bunch of sub-agents, navigate the code, fan out. It takes a while for this whole pipeline to run, 10 to 15 minutes typically, which is why it’s running in the background, almost like how you run CI/CD, waiting for unit tests to run and complete. So you open up a PR, you can go get a coffee, and by the time you’re back you have a code review from AI, very high quality, it understands the intent and gives you very high-quality feedback. That feedback has been improving, the models get better, but our harness has also been getting really good in terms of providing stronger evidence, the context assembly getting more sophisticated.

Priyanka: You talked about open source and how it helped improve the product. What about the enterprise context, because that’s always a challenge, “our code is unique,” which is true and not true in different scenarios. How has the evolution of the product been to solve the uniqueness of the enterprise codebase problem?

Harjot: We have a lot of enterprise customers, especially in the last year, Fortune 500 companies, tens of thousands of developers, and some of them have very legacy codebases, sometimes not even on Git platforms, Perforce for example. One of the things we’ve done really well is how the system tunes over time. One of the advantages we always had over every other tool, we were always a team-based product. We built a global memory layer back in 2023 itself, at that time it was Pinecone, now we use something else, where we’re able to create learnings, a global memory, so the more developers who work in the product or talk to it, the knowledge compounds over time. The system learns very quickly all the tribal knowledge, the quirks for the enterprise code. So in complex codebases, it’s already a step function improvement when you adopt it, but over time you can actually tune it a lot just by talking to it, very natural. The second thing we do is you can also explicitly provide instructions, there are many ways to tailor the system. It gets smart over time, almost becomes like a virtual teammate. So a lot of tuning happens just by talking to it, the memory layer takes care of that. The nice thing is each developer who contributes a memory helps everyone else, unlike a lot of single-player tools that only remember your local conversations and get smarter only for you.

Priyanka: That’s exactly where I was going, because this is such a critical piece of the puzzle in an enterprise, where there are a few developers doing an amazing job asking really great questions, and that gets fed back into the enterprise context.

Harjot: That’s where we had an advantage given the central nature of the tool. Think about MCP servers, in every tool, every developer has to go manually connect MCPs in their local Cloud Code or Cursor, everyone’s doing it, you have a thousand developers, they have to make a thousand connections to let’s say Datadog. In our case, admins create MCP connections centrally once, and all your thousand developers who open a PR benefit right away. It’s much easier to tailor the system for the entire org and the context.

Priyanka: Is that also one of the reasons why adoption has been easier?

Harjot: That’s right. This is not a product you can buy and then forget to use, or put on a shelf. If you’re buying and launching it, everyone is made to go through it, so the adoption is always high. Every developer, every PR is reviewed by CodeRabbit. In fact it’s become so mission critical that people cannot ship software if we ever have even a small outage, we’re as critical as GitHub at this point for a lot of these workflows. An individual developer doesn’t have to have a learning curve for it, they don’t even know it’s there, maybe if you’re a first-time developer you just open a PR and voila, you have a code review by AI. But developers in general love the feedback, it’s generally helped them uplevel their skills, interestingly AI is sometimes also a great mentor, some tricks they never knew about the code, but interestingly now no one’s reading the code, that’s the other sad part. It also helps when it goes to the senior developer for the actual human review, if you need that for a critical piece of software going out, the time they spend reviewing is also reduced because they know parts of it have already been handled.

Priyanka: We do two things for them, one is they have some certainty AI has done a good job finding low-hanging faults, so they can focus on higher-level architectural questions. The other thing we think is a really big deal, and becoming an even bigger deal, is explainability. Right now the outputs of AI have exceeded a point that humans are able to understand and comprehend. People are not reading the code, let alone expect a reviewer to come and understand. So that explainability gap is widening day by day, now you need to bring in tools that present information in higher-order abstraction, more diagrammatic, more visual, so you can quickly build trust. Because the nature of issues has evolved, AI is not making the same mistakes a human would, it’s not like they’re silly, missing error handling or try/catch blocks, they’re doing a good job, but they’re making mistakes at a larger level, like whether this thing is even needed, or is it the right way, or is it too much repetition increasing the entropy of the codebase. So we’ve built a UI that’s way better than what GitHub offers on code reviews, where you can understand things layer by layer. First layer, understand your schema changes, this is what it looks like, next layer you’re writing the business logic around it, then the API routes, then you built a UI. Lots of diagrams, sequence diagrams and flowcharts to understand and build trust quickly.

Priyanka: You have a lot of competition, what’s your view on the evolution of this space? Let’s start with the different types of companies. There are the no-code players, the Lovables of the world, the Bolts, and others, where you really do not want to see the code, you’re just building something and need to ship it. Then there are the pro-code developers, Cloud Code, GitHub Copilot, Codex, and those interfaces are also evolving. All these companies are working towards getting to the end to end. What is keeping you up at night in terms of where you want to take CodeRabbit?

Harjot: That’s a great question. One of the nice things is I’m happy we have competition, given I’ve done startups with no competition, and that’s the worst place to be, because nobody cares. That was my second startup, we had no competition, actually one company started because we got funded, and I feel bad for them, wish they knew we were also struggling. If you have a space with a lot of opportunity, everyone wants a piece of it. I’m happy to see larger players come in, smaller startups, some of them are growing, but we’re much further along, kind of seen as the gold standard, everyone compares to us and thinks about us when they want to do a bake-off. Competition is not a bad thing, it’s great for business. The other thing we see is how the space is going to evolve, we’re seeing more than competition, everyone when we go out always has a question, what does the future look like, is this workflow even going to be there, that’s the bigger question people have than the competition. Because competition is easy, best product wins, there’s fragmentation, it’s not going to be one coding agent, you can’t create your own homework, you need adversarial review, you need a clean context, just ask the coding agent to go and check its work, it trusts its own diffs too much. And secondly, reviews have to be central, that’s also very clear. But the bigger question is, are we going to see complete disruption, singularity, how quickly, do we even need it in two years? Dario comes up with a statement every other month and investors are just so uncertain. I believe that’s a bigger question more than competition. We’re not even worried about that given the products are just further along on all the dimensions that matter. The other thing about competition, the space, some things are not future-proof, code editors are not future-proof, people are now going into CLI, first they moved out of code editors, now desktop apps, so even VS Code was not future-proof. It’s like the self-driving, the steering wheel on a car is not future-proof, but the four wheels are, you still have to propel it somehow, drive. So code review, we believe some validation has to happen somewhere, now the question is is it GitHub or somewhere else, but the choke point has to exist. If you’re sending software to real customers you will need to look at it once and validate it before it goes out. I think this is the point where, once the dust settles, I believe this is where the value is going to accrue, it’s not very clear right now to a lot of people, but everything else will keep changing, this layer itself is going to be more and more valuable, in fact the most valuable layer in SDLC. Because this is a very clear choke point, it’s very hard to even do a choke point upstream, in fact we are thinking about planning, but that workflow doesn’t exist, even Linear says issue systems are dead, they’ve declared that’s not durable, which I would disagree with, of course it’s their own company, they could say anything, but I think some choke point has to exist upstream, but this one is definitely going to remain, this is the final choke point, we code, merges, and goes to production.

Priyanka: And we can argue that the choke point earlier in the cycle could be plan and requirements, what are we building, but even then, since the cost of building is so low, maybe we build a lot more and that choke point isn’t necessary.

Harjot: That’s going to be a new workflow, it’s a lot of garbage in garbage out. Once you have some of the background agents take off, then you have hundreds of pull requests open, that’s too much noise. So we’ll see a lot of changes in the space, but I agree with one key factor, which is anything that would go out needs to be looked at by a human at some point, and that point is somewhere closer to that code review stage. Even AI, it’s risk management, some things humans will look at, some things they can say this is low risk, let the AI decide whether it ships, but that validation quality gate has to be central. Because just because people have different preferences, they’re all picking different agents, it’s very hard to build consistency, and everyone wants to do coding, Datadog wants to do coding, Postgres wants to do coding, everyone’s building coding agents. So the world we’re living in is going to be more like, just like you trust your developers with, let’s say unit tests is one example, you could run unit tests idly as a pre-merge hook, but you’ll again run them in CI/CD, that never goes away. Datadog is another example, Kubernetes makes sure you don’t crash, but it doesn’t mean you take down your monitoring and guardrails. So guardrails never go away, very likely this layer becomes more and more important, even with a lot of autonomy, that validation layer will be standardized centrally, no matter how you run or generate the code, it will all go through this layer. The question is whether GitHub remains that central choke point, which I believe is true, I think those issues they’re facing are more temporary, everyone’s trying to build a better GitHub but it all actually looks the same as GitHub, it’s almost going to be a Twitter versus Bluesky moment, very hard to move off a platform when every open source project is there, all the ecosystem is there, you can’t just displace GitHub because you have slightly better reliability or performance, it has to be something like a new workflow. But assuming GitHub remains that choke point.

Priyanka: The workflow definitely seems like there’ll be evolutions, more harness, the loop engineering we’re seeing come around, a lot more of, all of that means there’s just more code to manage.

Harjot: Yeah, it has to live somewhere, that one thing is very clear, there’s going to be a lot more code to manage. None of these earlier techniques like stacked pull requests, I mean you have to bring in another AI, the layer I’m seeing is kind of like stacked pull requests without going through that workflow, so you’re going to naturally have PRs that are bigger and bigger, and then humans will need to move away from understanding lines of code to something else, structure of the application, architecture. But the code definitely is growing, especially some use cases where you don’t need to provide a lot of input, for instance translating something from an older language to a newer one, taking your business application, key factors, or adding another integration, if you already have a GitHub integration and want to add GitLab, it’s so easy for these agents. We recently had a Slack bot, I ran a goal for 40 hours to build a Discord version of it, and it worked. It’s amazing, some of these things you can just fire and forget, let the agent run in a loop validated against some goal. Other things you have to steer, be on the steering wheel sometimes, and those experiences are still very local, on the developer’s machine, people are finding ways to not let the laptop go to sleep, buying a Mac or something.

Priyanka: Exactly what I’ve done, it keeps running in the background. Is there anything else you want to talk about?

Harjot: No, it’s great. I think this has been, at CodeRabbit, a ride of a lifetime. I’ve done a few startups as you know, this one’s clearly been a very different animal, I’ve done fundraising multiple times for this startup, and each time the story was so different because the space moved so fast. Two years back we were talking about different kinds of concerns, where the value is going to accrue. Now people understand the word “harness,” at that time we were trying to explain that guys, the models are going to be a commodity, end of the day, this was even before Anthropic was, when Sonnet 3 wasn’t there.

Priyanka: We’re talking about the first fundraise back in ‘23, ‘24, right?

Harjot: Yeah. And the main thing was to explain what’s going to be the model’s value versus the applications on top. You were already building a harness for this, and we didn’t know what to call it at that time. It’s very interesting, and also the habits have evolved, some of the things haven’t been durable, but at least we’ve been in a space where it’s been durable for us, the tech we’ve built has been compounding over time. It’s not like we had to go remove everything and find PMF again in some other surface area. For the last two years we’ve been building in this direction, we built a lot of tech in this direction, but it’s not disrupted yet, it is about to, and is at the cusp of it, everybody’s realizing this is where they need to focus.

Priyanka: Perfect, in that space, is there any other product evolution you’re thinking about or focusing on right now?

Harjot: In fact, reviews as a space we believe we’ve cornered in a big way with the best product, almost like a Datadog in this space, it’s the best product, if you don’t want to waste your time DIY-ing, this is the best product you can just pick up, or you can fool around and come back, that’s also happening, that’s also fine, we’ve seen that so many times, people try but then give up and say okay let’s just pick this up. So now we’re starting to think about the larger SDLC, from a central vantage point, which is a unique position to be in, very hard to get into, let’s say customers’ GitHub, very few use cases get you access to that. If you think about it, Codex for example launched as a background agent a year and a half back, did not see success, because people don’t want to connect GitHub unless it’s mission critical, like code reviews or security or CI/CD, so they launched as a CLI and then desktop app. We already have these many GitHub installs, which is crazy, and now the question is how can we make a jump from there to other collaborative services like Jira and Linear, we don’t think issue systems are dead, so we want to be in Jira and Linear, we also feel we probably should be in Slack, we launched a Slack bot, so we’re now extending our arms into other surfaces. The whole idea of CodeRabbit is going to evolve from code reviews to being more collaborative AI for teams, where reviews is one collaborative activity you’re doing, planning is another one, incident response maybe is the third one. We’re going after those complex, team-based use cases, because that’s been our strength, background agent sandboxes, very efficient harnesses for those kinds of use cases, team-based memories, and we want to bring that into as many surfaces as we can.

Priyanka: Yeah, it’s pretty exciting. Thank you so much for being with me, spending time and sharing all the things CodeRabbit is doing.

Harjot: Thanks. Thanks, Priyanka.

Priyanka: All right, thank you for watching. I am very excited for you all to drop your questions in the comments, and if there’s any other guests you would like to hear from, let me know as well.

1. Start & Create

• /new — Start a new chat from scratch.

• /project — Create a project to organize related work.

• /upload — Upload files for Claude to reference.

• /paste — Paste content straight from your clipboard.

• /template — Use a template to skip the blank page.

• /import — Import content from an existing file.

• /scan — Scan and digitize documents.

• /voice — Use voice input instead of typing.

2. Focus & Context

• /focus — Set the main objective for the conversation.

• /context — Add background information Claude should know.

• /details — Provide more specifics on the task.

• /examples — Give examples to guide the output.

• /clarify — Ask Claude to pose follow-up questions.

• /define — Define key terms up front.

• /assumptions — List the assumptions in play.

• /priorities — Set what matters most.

• /constraints — Set limits or boundaries for the response.

3. Think & Solve

• /analyze — Break a problem down into its parts.

• /compare — Compare two or more options.

• /pros-cons — List the pros and cons.

• /recommend — Get a suggested course of action.

• /brainstorm — Generate a batch of ideas.

• /challenge — Stress-test an idea or plan.

• /solve — Work through and solve the problem directly.

4. Write & Edit

• /write — Generate new content from a prompt.

• /improve — Sharpen existing writing.

• /edit — Edit for clarity.

• /summarize — Condense text into a summary.

• /rewrite — Rework a piece to read better.

• /paraphrase — Restate the same idea in different words.

• /shorten — Make it more concise.

• /proofread — Catch grammar and typo issues.

• /expand — Add more depth and detail.

5. Organize & Structure

• /outline — Build a structured outline.

• /table — Turn information into a table.

• /structure — Organize loose content into a clear shape.

• /summary — Summarize the content of a document or chat.

• /bullet — Convert content into bullet points.

• /key-points — Pull out the key takeaways.

• /numbered — Format as a numbered list.

• /mindmap — Create a mind map of related ideas.

• /flowchart — Create a flowchart of a process.

6. Code & Tech

• /code — Write code for a given task.

• /debug — Find and fix bugs.

• /explain — Explain what a piece of code is doing.

• /optimize — Improve performance.

• /refactor — Restructure code without changing behavior.

• /test — Write tests for existing code.

• /convert — Convert between formats or languages.

• /docs — Write documentation.

• /review — Review code for quality and issues.

7. Data & Analysis

• /analyze-data — Analyze a dataset for patterns.

• /visualize — Turn data into charts.

• /insights — Extract insights from data.

• /forecast — Make predictions based on trends.

• /report — Generate a data report.

• /stats — Calculate key statistics.

• /clean — Clean and tidy messy data.

8. Automate & Integrate

• /workflow — Design a multi-step workflow.

• /automate — Automate a repetitive task.

• /api — Work with an API.

• /integrate — Set up reminders or connections between tools.

• /trigger — Set a trigger for an action.

• /tasklist — Create a task list.

• /checklist — Create a checklist.

9. Personalize & Control

• /preferences — Set your working preferences.

• /memory — Manage what Claude remembers.

• /tone — Adjust the tone of the response.

• /style — Change the writing style.

• /length — Change how long the response is.

• /format — Change the output format.

• /reset — Reset the conversation.

• /clear — Clear the current context.

10. Learn & Research

• /search — Search the web for current information.

• /learn — Get an explanation of a topic.

• /sources — Find supporting sources.

• /research — Run a deep research pass.

• /tldr — Get a TL;DR summary.

• /fact-check — Verify a claim.

• /explore — Explore a topic more broadly.

11. Collate & Share

• /share — Share the conversation.

• /export — Export content out of the chat.

• /download — Download a file.

• /feedback — Send feedback.

• /copy — Copy content to your clipboard.

• /email — Turn content into an email.

• /publish — Publish content externally.

Bonus Power Shortcuts

• /iterate — Improve output through multiple revision passes.

• /critique — Critically review the output.

• /simulate — Role-play a scenario.

• /debate — Present opposing viewpoints on a topic.

• /extract — Pull structured information out of raw text.

• /classify — Categorize information into groups.

• /translate — Translate content into another language.

• /generate-tests — Create test cases.

• /plan — Build a step-by-step execution plan.

• /check — Verify consistency or correctness.

• /help — Show help and tips.

Wrapping Up

Try chaining these commands together and magic starts happening. Set your /context, run a /brainstorm, tighten it with /edit, and close the loop with /iterate. That’s the whole game: context first, then structure, then refine. Master this handful of moves and you’ll spend less time prompting and more time getting the answer you actually came for.

This article my attempt at showing you the actual progression, and take you through one real use case where we use all five stages, from prompt to graph. By the end, you'll be able to explain every one of these terms to anyone who throws them at you like buzzwords.

Imagine you’ve built a travel-planning assistant. Someone types “plan a 3-day trip to Paris” and your model spits back a generic itinerary: Day 1, Eiffel Tower, Day 2, Louvre, Day 3, shopping. It works in the demo. Then it ships, and within a week your Slack fills up with complaints. The user who told the bot they’d already been to Paris three times gets the same beginner itinerary. The user with a food allergy gets a walking tour that ends at a seafood restaurant. The user who wanted the AI to book something is confused why it just talks.

None of this is a model problem. It’s an architecture problem. Your AI system stopped growing at stage one (prompt), and stage one was never built to carry this much weight.

The five stages, and why each one exists

Think of building an AI system the way you’d think about growing something from seed to orchard, not as a cute metaphor but because the constraint at each stage is the same one gardeners deal with: what you planted last season determines what you can grow this one.

Stage 1: Prompt engineering

This is the seed, the instruction itself: “plan a 3-day trip to Paris.” The question here is simple, what’s the core request. Your AI reads it, does its best, and replies with a generic plan. This works exactly once, for exactly one kind of user: someone with no history, no preferences, no constraints. Which is to say, nobody real.

Stage 2: Context engineering

The seed goes into a pot. Now there’s soil around it, budget, interests, past trips. The question shifts from what’s being asked to what else the AI needs to know right now. Tell it the user already visited London last year, prefers art and food over shopping, and has a moderate budget, and the itinerary stops being generic. It becomes theirs. This is the stage most teams stop at, and for a lot of chatbot use cases that’s fine. But once your AI needs to do something instead of just talk about it, context alone won’t carry you.

Here’s the whole progression, top to bottom is here explained visually with a sample use case. Each stage doesn’t replace the one before it. It adds a layer around it, the way a greenhouse doesn’t replace the pot, it wraps around it.

Next let’s discuss Stage 3, 4 and 5 from harness to loop to graph engineering.

Read more

“Mango didn’t finish dinner tonight, is that ok?”
“Biscuit’s left eye is goopy again”
“the orange one keeps going in and out of the litter box and nothing’s happening”

She sorts these in about four seconds each. The first is probably nothing. The second needs eye drops by Thursday. The third is a cat that might die tonight, because a male cat straining with no output is a urinary blockage until proven otherwise.

The answers aren’t hard. There is just one Meera, she sleeps sometimes, and forty of these arrive a night.

That’s the shape of problem AI agents are good at, and it’s also where most first agents go wrong. People get excited, they structure multi-agent logic, build a little swarm of planner and researcher agents passing notes to each other, and never ship. Define role and goal, design structured input and output, tune behavior and add protocol, add reasoning and tool use: those four are just the prototype. Everything after that is an amplifier you add to ship a product for real user like Meera.

Let’s break all the 10 steps down to build a legit AI Agent for production.

Step 1: What is the ONE thing your AI Agent needs to do?

Write the AI Agents role in one sentence, then write what it refuses to do

The agent is a triage assistant for foster volunteers. It reads a description of what a cat is doing, decides how urgent it is, and tells the foster what to do next.

That’s the ONE sentence. If you can’t write yours, stop and don’t write any code yet.

The more important half is the refusal list. This agent does not diagnose. It does not recommend medication or dosages. It does not say “wait and see” for anything involving breathing, straining to urinate, or a kitten under twelve weeks. Those go straight to a human no matter how confident the model feels.

People skip this part because it isn’t fun. It is the single highest leverage step in the entire build. Every ambiguity you leave here becomes a bug you cannot reproduce later.

Let’s visually draw all the 10 steps to show you how they progress and then complete this example with those steps.

Read more

If you haven’t built an AI app yet, it’s not too late and no, you’re not behind. Only 0.04% of the global population has used AI to build an app or workflow. I’m on a mission to increase AI literacy and make more people builders. I teach this one!! Come join my next cohort, you don’t need to be a coder, I’ve got both a coding track and a no-code track.

Over 5 weeks, you’ll build not just 1 but 2 multi-agent AI applications, while learning how AI actually works under the hood, in plain English, so you can participate in the decision-making at your company. If you already have a coding background, treat this course as your starting point for learning to build AI apps through the coding track.

For my substack family I am offering 30% discount to join early with the code EARLYBIRD, click on the button and get in!

Build Your First AI App with Me!

Step 2: For coders who want to become AI Engineers

If you know how to code and want to become an AI Engineer, this course by ByteByteGo is for you. It’s focused on skill building — the goal is for every participant to walk away with a strong foundation for building AI systems.

• Learn by doing: Build real world AI applications, not just watch videos.

• Structured, systematic learning path: Follow a carefully designed curriculum that takes you step by step, from fundamentals to advanced topics.

• Live feedback and mentorship: Get direct feedback from instructors and peers.

• Community driven: Learning alone is hard. Learning with a community is easy!

  

  ByteByteGo AI Engineer Course

Step 3: For AI Engineers ready to take their AI agents to production with Eval skills

Your next best step is learning how to evaluate the AI agents you build for production that’s AI Evals. This course by Parlance Labs teaches a practical, end-to-end approach to building and evaluating LLM-powered agents from construction all the way to production monitoring.

It’s hands-on: you build a support agent, instrument it so its behavior is measurable, find its failures through a disciplined error-analysis process, put it under a CI/CD regression suite, red-team it, and then improve both accuracy and cost. Across five modules, you’ll go from building agents (foundations, designing for evaluability, synthetic data & scenarios) to error analysis (open and axial coding to build a failure taxonomy and validated evaluators), to CI/CD (turning failures into regression test cases with automated gates), to security and adversarial evaluation (prompt injection, governance frameworks), and finally optimization (balancing accuracy against cost through prompt caching, model routing, and distillation).

Throughout, the course emphasizes a human-owned, evidence-based methodology grounded in frameworks like the Three Gulfs (comprehension, specification, generalization) rather than letting agents evaluate themselves.

AI Evals Course

Step 4: For engineers targeting the Forward Deployed Engineer FDE path

Want to become a Full Stack Forward Deployed Engineer? It’s the hottest job in tech right now. I ran a FREE session you can grab the recording for here. At Microsoft, I built a forward deployed engineering team before “FDE” was even a term, and in this session I share exactly what hiring managers are looking for in this role. If you’re a hardcore software engineer already working toward the FDE path and you understand AI Engineering fundamentals, I’d suggest this comprehensive course put together by Hamza.

Become FDE

This is the most common failure mode in enterprise AI, and it has nothing to do with model quality. The model was fine. What Northwind was missing was everything around the model: a way to get clean data into it, a way to turn its output into something a human could recognize as “this truck” or “that customer,” and a way to route a decision to the person who actually needed to make it.

That gap is what a proper AI platform architecture is supposed to close. And the answer to “do we need to buy into someone’s whole ecosystem to get this” is no. You can build it standalone, and you should understand what “standalone” actually requires before anyone sells you a shortcut.

The five things nobody puts on the roadmap

You are at an airport. Somewhere below the tarmac, baggage handlers move thousands of items from planes to belts to customs to the right owner, hour after hour, without most passengers ever thinking about it. That’s data integration. It’s unglamorous by design. Connectors that pull from a hundred different source systems, quality checks that catch a corrupted feed before it poisons everything downstream, versioning so you can tell what changed and when. Nobody praises the baggage system when it works. Everyone notices when it doesn’t.

Above that sits the fleet itself, different aircraft from different manufacturers, each with its own maintenance quirks, all needing to operate on the same runway. That’s model integration. A platform that forces every model into one proprietary format is an airport that only accepts one plane manufacturer. What you actually want is support for the runtimes people already use, PySpark and R showing up as first-class citizens instead of afterthoughts, standard package managers, model formats that don’t trap you.

Then there’s air traffic control, and this is the layer most teams skip entirely. ATC doesn’t see planes. It sees a radar blip and turns that blip into “Flight 448, a Boeing 737, currently at 12,000 feet, cleared to descend.” That translation from raw signal to meaningful object is the whole job. That’s the ontology layer: the thing that takes your tables and your model scores and turns them into “this customer,” “this shipment,” “this at-risk delivery,” with the relationships between them intact. Without it, every team downstream reinvents its own private interpretation of what a row in a database actually means, and those interpretations quietly drift apart.

Above ATC, gate agents and ops staff work the actual operation. Someone reviews a delayed connection and rebooks a passenger. Someone escalates a mechanical issue. That’s workflows, the layer where a person filters, reviews, and takes action on live data through an interface built for the job, not a spreadsheet somebody exports every Monday.

And at the very top, the control tower makes the calls that everything else has been feeding into: hold this flight, clear that runway, divert around weather. That’s decision orchestration, structured and auditable, so when someone asks “why did we do that” six months later, there’s an actual trail to point to.

Here’s the visual represention of stack, bottom to top, data flowing up into decisions:

Read more

Chemistry got a periodic table in 1869 and hasn’t needed to renegotiate what an element is since. AI engineering doesn’t have that yet, so here’s an attempt: the full set of building blocks, organized by what they do rather than when they became trendy, with an actual definition attached to each one.

Think of the whole stack as a restaurant. A chef with trained instincts, a kitchen moving fast during service, a pantry stocked with fresh ingredients, a maître d’ deciding who gets seated where, a health inspector who shows up unannounced. Six floors, twenty terms each.

Here are all 120 elements visually on a periodic table and explained with the restaurant analogy.

Read more

That someone is now called a Forward Deployed Engineer. Postings are up something like 800%. Anthropic, OpenAI, Databricks, Palantir, all hiring, with total comp at the AI labs running $350K to $750K. And nobody in tech is telling you the most useful fact about this job: it isn’t new. I started my career doing exactly this work. It just had a worse name and no equity. Once you see that, everything about how to get the role becomes obvious.

We’ve Had This Job for Thirty Years

Enterprise software has always needed a person who ships inside the customer’s mess. Around 2005 to 2015, that person showed up twice in the deal cycle.

Before the contract, they were a Solutions Architect. Free, because the vendor wants to win your business. Sitting with customers, sometimes for weeks, whiteboarding what the deal would actually take. That person was me. I have vivid memories of translating between a CTO who hadn’t said out loud that he didn’t believe any of it, and engineers who were quietly doing the math on their own jobs.

After the contract, they were Professional Services. Same work, except now it costs money. Embedded with the customer’s engineering team, on-site for months, moving fast through environments that were never as clean as the sales deck promised.

Both got measured on exactly one thing. Did the customer’s problem get solved in production? Not features. Not lines of code.

Embed. Build. Own the outcome. That’s the whole job, and it’s been the job since before some of the people now interviewing for it could drive.

So What Changed? Three Things

AI collapsed implementation time. Six months became six weeks, sometimes days, because Claude Code and the modern stack write the scaffolding for you. The typing got easier. And the economics flipped with it: one FDE now covers what took a team of three a few years ago. That’s why AWS put a billion dollars into a dedicated FDE org, and why Databricks unified Pro Serv across 1,900+ customer engagements. The math finally works at AI speed.

The hard problems moved. Coding shrank; thinking got harder. Pro Serv never had to check for hallucinations. I never built an eval pipeline as an SA, not once. Your logs and metrics and traces will not catch LLM failures, which is a genuinely uncomfortable thing to explain to a customer who just spent two decades trusting their monitoring stack. Verification is the new discipline: rubric-graded test suites, LLM-as-judge, golden datasets.

And the role got a name that made VCs write checks. I know that sounds cynical. It’s also just true. “Solutions Architect” meant decades of work, no equity, less prestige. “Forward Deployed Engineer” commands attention and compensation for the same work plus an AI layer. The rebrand matters. Use it.

So what are the skills you need and how do you lands this FDE role anyway? Let’s look at that now and also a video detailing all this.

Read more

When we first started building agentic systems, the instinct was to write a massive system prompt and trust the model to figure it out. The results were predictable: infinite loops, hallucinated tool calls, context windows that silently degraded until the agent started confusing itself. The model wasn’t broken. The architecture was just not there, its been evolving.

Twelve design patterns have come out of those early wrecks. They map onto five functional subsystems: Reasoning, Perception, Action Execution, Learning, and Inter-Agent Communication. Understanding how they fit together is important to build production AI systems.

The Physical Constraints Of Running AI Apps

Think of a hospital emergency room. Walk in with a sprained ankle versus a cardiac arrest and the triage system routes you very differently. This is because finite resources need to match urgency and complexity. Every agentic AI system faces the same allocation problem, except the finite resource is the context window and the currency is tokens.

Transformer inference splits into two phases: the prefill (reading your prompt) and the decode (generating the response). You want Time-To-First-Token under 400ms. Incremental token generation under 50ms. And you need the model’s attention to stay sharp, which means keeping context window utilization below 40% capacity. Pack in too much and the model starts losing track of what mattered. This is the “lost in the middle” phenomenon, empirically documented, not theoretical.

The 12 patterns are strategies for managing this constrained allocation problem. Let’s cover them now.

If you want to learn to build your first AI Agent in 5 weeks join our next cohort starting in September, offering EARLYBIRD 30% discount now.

Check It Out Now

Read more

Two blocks away, a stadium is feeding 80,000 people at halftime. Nobody back there is improvising. There are four thousand line cooks, every single one executing the same instruction: assemble the burger, wrap it, pass it down. No creativity. No memory of previous customers. No adjusting for allergies. Just: same operation, performed in parallel, at scale.

Your CPU is Chef Ramsey. Your GPU is the halftime kitchen.

And once you really understand that, a lot of things about AI performance, training costs, and why your PyTorch script runs ten times faster on GPU start to make sense.

Ramsey Can Do Anything. His Kitchen Has 24 Cooks.

A modern high-end CPU has maybe 24 cores. Each one is capable. Out-of-order execution, branch prediction, multiple levels of cache, clock speeds above 5 GHz. Ramsey’s kitchen can handle a reservation change mid-service, run the books afterward, and call suppliers in the morning. When you boot Python, parse a JSON file, or run a Flask server, Ramsey’s kitchen is exactly what you want.

But imagine you need to transform the coordinates of 8 million vertices in a 3D game scene. Every vertex needs the same math applied to it: add the object’s world position to the local position. Vertex 4 has zero interest in what vertex 3 is doing. They’re completely independent.

Ramsey could do this. He could process each vertex one at a time. But that’s not what he’s built for.

The GPU has 10,752 CUDA cores on a chip called the GA102. Each core is a stripped-down calculator. It does one thing: multiply two numbers and add a third. a * b + c. That’s it. It does not do anything else: Nothing related to OS or memory. But there are ten thousand of them, and they all fire at once.

That’s 36 trillion arithmetic operations per second. The halftime kitchen doesn’t care that it can’t plate a seven-course menu.

The Stadium Has a Franchise Structure. 28 Billion Employees.

Walk into the halftime kitchen and you don’t see chaos. You see an org chart.

The whole operation is the GA102 chip, 28.3 billion transistors. It splits into 7 wings, each called a Graphics Processing Cluster. Each wing has 12 cooking stations, the Streaming Multiprocessors. Each station has several teams of 32 cooks working in lockstep, called warps.

But not all line cooks do the same job. There are three types in the kitchen and they do very specific jobs. Let’s look at those now.

The CUDA cores are the burger flippers. 10,752 of them. They do a * b + c and nothing else. Most of what a game or a Python training loop needs goes through them.

The Tensor Cores are the prep cooks who specialize in bulk assembly. 336 of them. Instead of one a * b + c, they take an entire tray of ingredients (a matrix), multiply it by another tray (the weights), add a third tray (bias), and produce a finished output tray. The whole thing in one clock cycle. This is what neural networks live on.

The Ray Tracing Cores are the lighting designers. 84 of them, each complex. They simulate how light physically bounces around a scene. Fewer, but specialized.

One other thing worth knowing: that 3090 Ti you’re eyeing and the 3080 sitting next to it at a lower price are the same physical chip. Manufacturing defects happen. A chip with 16 broken cooking stations gets disabled down to 8,704 working CUDA cores and sold as a 3080. Nvidia calls this binning. It’s efficient engineering and better margins. Both chips, same kitchen floor plan.

The Ingredients Are Stuck in a Warehouse. 10,000 Cooks Are Waiting.

Here’s where a lot of developers get surprised.

The stadium kitchen is ready. 10,752 cooks, arms crossed, staring at empty prep surfaces. The ingredients are in a warehouse. How fast can you get them here?

That’s the memory bandwidth problem. A GPU’s processing cores are so fast that data delivery is usually the actual bottleneck, not compute. The RTX 3090 has 24 GB of GDDR6X memory sitting directly on the graphics card, connected via a 384-bit bus. Think of 384 parallel conveyor belts, all running simultaneously, delivering 1.15 terabytes of ingredients per second.

Your CPU’s RAM? 64-bit bus. About 64 GB per second. The stadium kitchen’s delivery infrastructure is 18 times faster than Ramsey’s.

Now imagine the model you’re training has 175 billion parameters. That’s GPT-3. At 2 bytes per parameter, just holding the weights requires 350 GB. A single H100 has 80 GB of HBM3 memory. You can’t even fit the model on one card, let alone stream all the weights through it on every forward pass.

This is why AI infrastructure companies spend so much money on memory. The compute is almost a solved problem. Getting data to the compute fast enough is the hard part.

The H100 uses HBM3, High Bandwidth Memory, stacked in physical cubes directly on top of the chip using microscopic vertical connectors. The distance data travels is measured in microns. One HBM3 stack delivers 3.35 TB/sec. Compared to the 3090’s GDDR6X at 1.15 TB/sec, it’s not a small upgrade.

The Chef Announces One Order. Every Cook Executes It.

Back in the stadium kitchen. The head chef picks up the mic: “Everyone, sear the protein for exactly 90 seconds.” Every single line cook executes that instruction on whatever is in front of them. Different proteins. Different weights. Same instruction.

That’s SIMD. Single Instruction, Multiple Data.

It’s why GPUs are so good at video game rendering. A scene has thousands of objects, each with thousands of vertices. Every vertex needs the same transformation: take its local position, add the world offset, output the result. Vertex 4 doesn’t care what vertex 3 is doing. They’re independent.

In Python terms, this is the difference between:

python

for vertex in vertices:
    result.append(transform(vertex))  # Ramsey doing it one at a time

and:

python

result = transform_matrix @ vertices  # stadium kitchen, all at once

That second line dispatches to thousands of CUDA cores simultaneously. The first line runs on one CPU core, sequentially. For 8 million vertices, the difference is not 10%. It’s orders of magnitude.

The technical term for this class of problem is “embarrassingly parallel.” Nothing embarrassing about it. It just means the work splits perfectly with zero coordination overhead. No cook needs to know what another cook is doing. Video game rendering is the textbook case. So is training a neural network, which is why the same hardware runs both.

The Prep Cooks Who Changed AI

When Nvidia added Tensor Cores to the Volta architecture in 2017, the AI community noticed immediately.

A standard CUDA core does one multiply-add: a * b + c. One output per clock cycle. A Tensor Core takes a small matrix A, multiplies it by matrix B, adds matrix C, produces output matrix D. The entire matrix operation completes in one clock cycle. Not element by element. The whole thing at once.

Why does this matter for AI? Because a neural network layer is literally a matrix multiplication. Your input batch is a matrix. The learned weights are a matrix. You multiply them, add bias, pass through activation. Repeat hundreds of times through the layers. GPT-4 reportedly took training runs requiring tens of thousands of H100s running for months. That scale is only possible because Tensor Cores collapse what would be millions of sequential CUDA operations into a handful of matrix operations.

Training and inference are the same hardware, different jobs. Training is the expensive loop: forward pass, compute how wrong the answer was (loss), run math backwards through all layers to adjust every weight (backpropagation), repeat millions of times. GPT-3 needed approximately 355 GPU-years of compute.

Inference is the cheaper loop: tokenize your prompt, run a forward pass through every layer, predict the next token, append it, repeat until done. No backward pass. Still needs a GPU, because each forward pass through a large model is still hundreds of matrix multiplications across billions of parameters.

When to Send Ramsey Home and Call the Stadium

The GPU wins when your work is large, uniform, and parallel. Training any neural network. Running inference with a batch size above 1. Rendering. Matrix operations. Anything you can express as tensor @ tensor in PyTorch or NumPy.

The GPU loses when your work is sequential, branchy, or small. A Python loop where each step depends on the last one. Business logic with conditionals. Anything where the cost of shipping data to the GPU exceeds the compute benefit. Batch size of 1 on a tiny model is often faster on CPU just from avoiding the transfer overhead.

For Python specifically, the rule is simple. If you’re writing a for loop over individual elements and calling GPU operations inside it, you’re sending Ramsey to the stadium one burger at a time. The GPU wants its full batch upfront. Give it that, and 10,752 cooks go to work simultaneously. Starve it with sequential calls, and you’re paying for a stadium kitchen to flip one burger.

Ramsey’s fine-dining kitchen hasn’t gone anywhere. Your operating system, your file system, your network stack, the Python interpreter itself: all of that runs on Ramsey’s side. The stadium kitchen can’t run Python. It can only run the math you explicitly send it.

They work together. Ramsey coordinates. The stadium executes. The trick is knowing which job belongs to which kitchen, and making sure you’re not accidentally asking one to do the other’s work.

Ramsey is still at Ramsey’s Kitchen on Saturday nights. He’s not going anywhere. But when 80,000 people want a burger at halftime, you don’t call him.

You know which kitchen to call now. The trick, in Python or in production, is making sure you’re not accidentally sending a seven-course tasting menu order to the stadium. Give the line cooks their full batch. Let them work in parallel. That’s what 36 trillion operations per second actually means.

You need both CPUs and GPUs for an AI workload but for different things!

If this struck a cord, follow for more blogs explaining Cloud and AI concepts visually.

If you’re honest, it was mostly the first one and the last one. The twelve statements sandwiched in the middle? You processed them. They technically passed in front of your eyes. But if pressed, you’d struggle to recall a single detail from statement nine. The detail that would have cracked the case — the one that put the killer at the wrong address at the wrong time — was in statement ten. You missed it.

That’s not a failure of intelligence. It’s a failure of architecture. Your brain, under pressure, with a lot of input, naturally weights the first thing it encounters and the last thing it read. Everything in the middle blurs.

Your LLM does the exact same thing. The research calls it “lost in the middle,” and it will silently destroy your RAG pipeline’s accuracy without a single error message to warn you.

The evidence you’re already ignoring

Liu et al. (2024) ran a version of the detective experiment at scale. They gave models a set of documents, varied the position of the one document that actually contained the answer, and measured whether the model got the question right. At position 1: high accuracy. At position 20: high accuracy. At position 10 out of 20: accuracy dropped by more than 30 percentage points. Same model, same documents, same question. The only variable was the shelf the answer was on.

That 30% gap is the gap between a RAG system your users trust and one they quietly stop using.

Chroma’s 2025 study tested 18 frontier models on contexts ranging from 10,000 to 500,000 tokens. Every single one showed the same pattern. Not most of them. Every one. The RULER benchmark tested 17 long-context models and found that claimed context window sizes “dramatically overstate effective context.” A model advertised at 1 million tokens does not give you 1 million tokens of equal attention. It gives you reliable recall at the edges and a cold valley in the middle.

Let’s understand this visually

Why the LLM architecture does this

The detective analogy maps cleanly to what’s happening in the transformer weights. When a detective reads twenty statements back to back, primacy and recency effects in human memory cause the first and last items to stick. The cause is cognitive, rooted in how short-term memory consolidates. Let’s unpack that now

Read more

Registrations close in about 48 hours! Secure your spot here.

Register

Here is wha you will learn:

• AI fundamentals you can actually use: distinguish traditional ML from GenAI, and deploy your first production AI system using a 4-step framework used at Fortune 100 companies

• A decision matrix you keep forever: know when to use RAG vs fine-tuning, how to weigh cost vs performance, and how to pick the right model for any use case

• How to build and ship a real multi-agent system: design human-in-the-loop workflows, prototype without code, and deploy agents that work at scale

• How to lead AI projects that get buy-in: frame cost vs performance for non-technical stakeholders, manage AI like a product, and define ROI for real business outcomes

• Responsible AI that your org can actually implement: why traditional testing fails for agents, how to evaluate AI apps properly, and how to build an ethics policy that isn’t just a slide deck

Wanna join?

Check it out now

AI tools are the new kitchen. They are extraordinary. But they produce, by design, the statistically average output of everything they were trained on. They optimize for “plausible.” They cannot optimize for “yours.”

When production costs nothing and everyone has the same kitchen, the only remaining variable is the cook’s judgment. That judgement has a name, and it is TASTE.

Taste Gap

The diagram below is worth staring at for a moment.

There is a well-known phenomenon in creative work called the taste gap. Ira Glass described it for radio, but it applies everywhere: when you start, your taste already knows what good sounds like. Your skills cannot reproduce it yet. The gap between those two things is where most people quit in frustration.

Generative AI violently collapses the skill side of that gap. You can produce a polished UI mockup on your first week as a designer. You can ship working code without knowing how it works. The floor jumps up overnight.

But collapsing the skill gap does not collapse the taste gap. It widens it. Because now you can generate a hundred outputs before your eye can evaluate a single one. You have more production than judgment. You have a kitchen bigger than your palate.

The space between “what AI can produce” and “what actually matters to a human being” is what researchers call the Curation Zone. It is the territory where every dollar of value in a post-execution economy lives. And you can only operate in it with taste.

What Taste Actually Is (Not What People Think)

Taste gets dismissed as subjective. That is a cop-out, and Paul Graham has a useful argument against it: if taste is entirely subjective, then there is no such thing as good design. But a well-architected distributed database is objectively better than a fragile monolith. A sentence that lands is objectively better than one that does not. Therefore the faculty that discerns quality has to have objective validity, even if it is difficult to measure.

Taste is not aesthetic preference. It is the accumulated internal library of what works and why.

It operates in two modes.

Additive taste is what designers use. You start with something functional and you layer in the choices that make it feel right: the typographic weight, the whitespace, the color temperature, the detail that makes someone pause. It requires saturation in the domain, thousands of hours of exposure, enough context to know that a slightly heavier font weight changes the emotional valence of the entire page.

Subtractive taste is what product managers and engineers use. You face a thousand possibilities and you cut. You figure out what not to build. You find the one constraint that makes everything else obvious. Apple deciding the first iPhone would have no physical keyboard was not an aesthetic choice. It was an act of subtractive taste so aggressive that it looked insane to everyone who had not yet built the mental model.

Both modes share something: they cannot be faked, and they cannot be delegated to a model. Let’s understand this visually now!

Quick note: My course is starting next week, if you want to build your AI Engineering chops come join us

Register Now

What Anthropic’s Head of Product Actually Does All Day

Cat Wu, who leads product for Claude Code at Anthropic, has described her job as being almost entirely about taste. Not roadmap prioritization in the traditional sense. Not user story grooming. The actual work is: reading hundreds of pieces of raw feedback and developing the intuition for what users actually need versus what they literally asked for.

Those two things are almost never the same.

What users ask for is usually a symptom. What they need is a diagnosis. Getting from one to the other requires a kind of judgment that does not live in any data set. It lives in the person who has processed enough feedback across enough contexts to pattern-match on the underlying problem.

At Anthropic, this has led to something unusual: the roles of engineers and product managers have started to merge. Because AI makes the mechanical execution of code cheap, the premium shifts to people who can observe behavior, form a thesis about what’s wrong, design an intervention, write the code to test it, and ship it inside a week. That is not a PM skill or an engineering skill. It is a taste skill that happens to require both.

Wu also talks about “model taste,” which is worth pausing on.

Different models fail in different ways. GPT and Claude and Gemini each have their own jagged edge. Knowing where a specific model’s logic breaks down, and designing around it rather than through it, is its own form of taste. You cannot learn it from the documentation. You learn it by watching the model fail on things you care about, and then asking it why.

The Danger Is Trusting AI Wrong.

There is a study that should make every engineer a bit uncomfortable.

1,372 people took cognitive reflection tests alongside an AI assistant. When the AI gave a wrong answer, participants accepted it 73% of the time. More disturbing: their subjective confidence in the answer actually went up, by about 12%, even when the answer was false.

They had borrowed the model’s confidence and mistaken it for their own.

This is what is called cognitive surrender. Not delegating a task. Surrendering the act of forming an opinion altogether. It is the difference between using a calculator for arithmetic and using a calculator to figure out what you are trying to compute.

In software development, cognitive surrender compounds. It shows up as comprehension debt. You paste a stack trace into a model, accept the patch because the tests pass, and deploy it without building a mental model of what actually went wrong. You do this fifty times across a codebase, and you end up with a system that runs but that nobody understands. The first novel edge case will be a fire.

Andrej Karpathy distinguishes between two postures toward AI-generated code. Vibe coding is accepting diffs because they feel plausible. Agentic engineering is retaining full accountability for the specification, the security, and the architectural judgment while using AI to move faster on the parts you already understand. The difference is not about tools. It is about whether you maintain an independent model of the problem or let the AI’s output become your mental model by default.

Vibe coding raises the floor. Agentic engineering raises the ceiling. You need taste to know which mode you are in.

Can You Teach Taste to a Machine?

This is one of the more interesting questions in AI research right now, and the answer is: partially, in narrow domains.

A research team built something called the TASTE dataset. Ten professional designers evaluated AI-generated graphic design outputs across nine distinct dimensions: color harmony, typographic craft, visual hierarchy, mood and tone, spatial accuracy. The goal was to see if off-the-shelf vision models could approximate the designers’ judgments.

None of the models exceeded a 0.55 macro agreement with the human majority. Scaling the model up did not help. The models just traded one kind of error for another.

Graphic design taste is multidimensional and contextual. A poster might have beautiful color harmony and a completely broken typographic hierarchy. A model that scores “overall preference” cannot see those dimensions independently. Human designers can. That gap is real and it is not closing fast.

The results are more interesting in scientific domains. A model called Scientific Judge, trained on 700,000 pairs of high- versus low-citation paper abstracts, achieved an 81.5% win rate at predicting which research ideas would have long-term impact. In structured text domains with a clear objective signal (citations are a proxy for community judgment), taste can be approximated mathematically. Even then, it is bounded by the domain and by the quality of the signal.

The short version: machines can approximate taste in narrow, structured, measurable domains. The rest of it is still yours.

The Misuse of Taste as Gatekeeping

One caveat worth naming: taste as a professional value can curdle into something toxic if it gets pointed at the wrong target.

There is a version of “I have taste and you don’t” that is just elitism with better vocabulary. Analyst Elliot Smith makes the point that the software market is full of products that critics love and users ignore, and products that look “clunky” by any design standard but solve genuine problems for millions of people. Notion beats Roam Research. Figma is beloved. But so is Excel, which has a UI that would fail most design school critiques.

True taste is not about satisfying critics. It is about caring enough about the actual human problem that you do not stop at “done.” You go a little further, toward the thing that actually delights someone. That orientation is toward the user, not toward the peer review.

The most valuable kind of taste in the AI era is practical, outward-facing, and relentlessly honest about the difference between “this looks good to me” and “this works for the person who needs it.”

How to Build Taste (Without Waiting for a Decade of Experience)

Taste is not innate. The research consensus on this is pretty clear. It develops faster than technical skill because you can consume work critically without being able to produce it. But it still requires deliberate practice.

A few things that actually accelerate it:

Critical deconstruction over passive consumption. Every time you encounter something that works, stop and articulate why. Not “I like this.” Why does the spacing feel generous? Why does the information hierarchy tell you where to look? Why does this code feel readable when structurally equivalent code does not? The articulation is the work. It forces your brain to form an explicit principle instead of a vague positive feeling.

Equally, articulate what you hate. Preferences without negatives are not taste. They are tolerance. The willingness to say “this is generic slop and here is exactly why” is the crucible where genuine judgment develops. AI outputs at the plateau of the Curation Zone tend to look fine. They are fine. You have to train yourself to see past “fine” to what “right” would look like.

Build small, high-signal evaluation sets. Cat Wu talks about this in the context of model evaluation: instead of a massive automated test suite that regresses to the mean, build ten exceptional, edge-case prompts that genuinely stress-test the quality you care about. The same principle applies to your own taste calibration. What are the ten outputs you would be proud of? What makes each of them work? That is your personal quality bar, made explicit.

Ship things and watch what happens. Taste is not a library you build in your head and then apply. It is calibrated by contact with reality. The feedback loop of building something, putting it in front of real people, and watching where it fails is irreplaceable. The gap between what you thought would work and what actually does is where taste lives.

The Real Competitive Moat

Every few years, the technology industry convinces itself that a new tool is the moat. SQL skills. Knowing JavaScript before everyone else. Machine learning. Kubernetes.

These things matter for a window. Then they commoditize. Then someone ships an abstraction layer that makes the underlying skill table stakes.

Taste does not commoditize. The more capable AI gets at execution, the more taste is the thing you cannot replicate by scaling compute. You cannot train a model on what matters to your specific users in your specific context with your specific understanding of the problem. You have to be the one who actually cares enough to figure that out.

The future will not belong to whoever generates the most output. It will belong to whoever knows, immediately and without ambiguity, which output to keep.

That is the skill. It gets harder to fake the better the tools get. Which means, if you have it, it only gets more valuable.

If you want to join the “Build Real AI Applications in 5 Weeks” cohort we this is the last week to get in at 20% off. If you have questions that I can answer you can reach to me and my team at contact@thecloudgirl.dev and we will get back to you as soon as possible.

Register Now

Most developers learn one RAG pattern and call it a day. They wire up a vector database, add some chunking logic, and ship. Few months later, they’re debugging why the system confidently hallucinates, ignores user history, or falls apart on multi-hop questions. The problem isn’t the retrieval step. The problem is they picked the wrong architecture for the job.

RAG is not one thing. There are at least ten meaningfully different patterns, each solving a different failure mode. This article walks through all of them, with an honest take on when each one earns its complexity.

Pattern 1: Simple RAG, Prototype Only

Think of Simple RAG like a library with a single intern. You walk in, hand them your question on a slip of paper, they run to the stacks, grab the most relevant pages they find, and read them back to you. Fast. Functional. But the intern has no memory of the last hundred people who asked similar questions, can’t judge whether the pages they grabbed are actually any good, and definitely can’t handle it when your question requires checking three sections of a book in the right order.

The pipeline is exactly what it sounds like: query comes in, embeddings are computed, nearest documents are retrieved, the LLM generates a response with those documents as context.

It works for demos. It breaks in production.

When it wins: Constrained question-answering on a clean, well-indexed corpus. An internal FAQ bot with <500 documents.

When it fails: Multi-turn conversations, noisy corpora, queries that require synthesizing multiple sources, anything where the user expects memory of prior turns.

What you give up: Almost everything context awareness, quality checks, routing logic.

Pattern 2: RAG with Memory

The intern from Pattern 1 now keeps a notebook. Every time you come back, they flip through it and bring context from your previous conversations into the retrieval process.

This sounds simple. The implementation is not. Memory introduces questions you don’t think about until they bite you: what counts as memory? How long does it persist? Does session memory contaminate results for other users sharing the same system? How do you handle contradictions between past context and retrieved documents?

RAG with Memory is the first pattern that requires you to think seriously about state. Most teams underestimate this.

When it wins: Customer support bots, personal assistants, any system where the user expects continuity across sessions.

When it fails: High-volume systems where memory accumulation becomes a retrieval problem in itself. When stored context is stale or wrong.

What you give up: Statelessness. Testing becomes harder. Memory bugs are subtle.

Pattern 3: Branched RAG

Imagine your question is actually three questions in a trench coat. “Summarize our Q3 sales, compare it to Q2, and flag any anomalies” isn’t one retrieval task. It’s three, with a synthesis step at the end.

Branched RAG decomposes a complex query into sub-queries, runs parallel retrieval pipelines for each one, then synthesizes the results. Like sending three different librarians to three different sections simultaneously, then having an editor stitch the findings together.

The gains are real. Complex queries get better answers because each sub-query targets the right corpus slice. The cost is latency and complexity: you’re now orchestrating multiple retrievals, handling partial failures, and doing synthesis that can introduce its own errors.

When it wins: Research assistants, competitive analysis tools, multi-faceted technical documentation queries.

When it fails: When query decomposition goes wrong and sub-queries don’t cover the original intent. When synthesis introduces hallucinations.

What you give up: Simplicity, latency. Debugging parallel retrieval is not fun.

Pattern 4: HyDE (Hypothetical Document Embeddings)

Here’s a counterintuitive idea: instead of embedding the user’s question and searching for similar documents, ask the LLM to generate a hypothetical answer first, then use that as your search query.

The intuition is that real document chunks are written like answers, not questions. The embedding space between “What causes transformer attention to scale poorly?” and a textbook paragraph explaining the quadratic complexity of self-attention is larger than you’d like. The embedding space between that paragraph and a fake-but-plausible explanation of the same thing is much smaller.

HyDE works because it aligns the query vector with the document vector space by making the query look like a document.

When it wins: Technical or domain-specific corpora where user queries are short and documents are long and dense. Academic search, internal knowledge bases with formal writing.

When it fails: Domains where the LLM’s priors don’t match the corpus. If the model generates a confident but wrong hypothetical, you retrieve the wrong documents and amplify the error.

What you give up: One extra LLM call on every query. And if the hypothetical is wrong, your retrieval is wrong.

Pattern 5: Adaptive RAG

Not every question needs the same retrieval strategy. “What’s our refund policy?” is a simple lookup. “Analyze our customer churn drivers across the last three quarters and suggest an intervention” is not.

Adaptive RAG puts a router in front of the retrieval pipeline. The router classifies the incoming query and sends it down one of several paths: no retrieval at all (for conversational turns or factual questions the model already knows), simple single-step retrieval, or multi-step retrieval for complex analytical tasks.

The analogy is a hospital triage system. You don’t give every patient an MRI. You assess severity and route them appropriately. A broken wrist doesn’t need the same resources as a chest X-ray.

The router itself is usually a smaller, cheaper model. Which means Adaptive RAG can actually lower your inference costs by avoiding expensive multi-step retrievals on questions that don’t need them.

When it wins: Mixed-use chatbots, enterprise assistants handling wildly varied query types, cost-sensitive deployments.

When it fails: When the router misclassifies. A complex query routed to simple retrieval gives a surface-level answer. Users lose trust fast.

What you give up: One more failure point. The router adds latency and its own error mode.

Pattern 6: Corrective RAG (CRAG)

Retrieval can return garbage. The nearest neighbor in embedding space is not always the most relevant document. CRAG adds a quality gate after retrieval that evaluates whether the retrieved documents actually support answering the question.

If the retrieved documents pass the quality check, they go straight to the LLM. If they fail, the system falls back to a web search (or another retrieval source) and reformulates the query before trying again.

Think of it as a fact-checker standing between the library and the writer. If the intern returns with pages that clearly aren’t relevant, the fact-checker sends them back rather than letting the writer work with bad material.

The implementation usually involves a lightweight evaluator model scoring retrieved chunks on relevance. Getting the scoring threshold right is fiddly and domain-specific. Too strict, and you’re doing web searches for everything. Too loose, and you’ve just built expensive Simple RAG.

When it wins: Any domain where retrieval quality is inconsistent. Medical, legal, or financial corpora where wrong context is worse than no context.

When it fails: When the fallback (web search) isn’t appropriate for the domain. When latency budget is tight.

What you give up: Latency and implementation complexity. You’re building an evaluator on top of a retriever on top of a generator.

Pattern 7: Self-RAG

Self-RAG is the most introspective of the patterns. The model doesn’t just retrieve and generate. It reflects. At each step, it asks itself a set of structured questions: Is retrieval even needed here? Are the retrieved documents relevant? Does my generated response actually follow from them?

These aren’t rhetorical questions. They’re special tokens or a structured prompting mechanism that forces the model to evaluate its own outputs at each stage of the pipeline.

The result is a generation loop that can catch its own errors before they reach the user.

Self-RAG is slow. It’s expensive. It’s also the right answer when you’re in a domain where confident errors are dangerous. Clinical decision support. Legal research. Financial compliance. Places where “I’m not sure” is a better output than a wrong answer delivered with high confidence.

When it wins: High-stakes use cases where hallucination has real-world consequences.

When it fails: High-traffic applications. The reflection loop isn’t cheap and doesn’t parallelize well.

What you give up: Throughput. Self-RAG trades speed for reliability.

Pattern 8: Agentic RAG

All the patterns above have one thing in common: they’re relatively static pipelines. Query in, answer out. Agentic RAG is different. The LLM isn’t at the end of the pipeline. It’s running the pipeline.

The model acts as an orchestrator. It decides what tools to call (search, code execution, API calls), evaluates the results, decides whether to keep going or it has enough information, and loops until it’s confident in its answer.

This is the “agent with memory and tools” pattern applied specifically to retrieval. Instead of a single retrieval call, you get a dynamic loop where the model can chain multiple search queries, execute code to verify facts, call APIs to get real-time data, and reason across all of it.

The gap between Agentic RAG and the other patterns is roughly the gap between a research assistant who asks follow-up questions and one who just answers whatever you said.

When it wins: Complex, open-ended research tasks. Anything requiring multi-source synthesis or tool use. When you genuinely don’t know what information you’ll need until you start looking.

When it fails: Unbounded loops are a real risk. Costs are unpredictable. Latency can spiral. Getting the agent to know when it’s done is genuinely hard.

What you give up: Predictability. Agentic systems are harder to debug, test, and cost-control.

Pattern 9: Multimodal RAG

Enterprise knowledge doesn’t live only in text. It lives in slide decks, charts, scanned PDFs, engineering diagrams, and images with no alt-text. Classic RAG ignores all of it.

Multimodal RAG routes all content types through a vision model that generates text descriptions, then indexes those descriptions into the vector database alongside traditional text chunks. The retrieval and generation pipeline stays mostly the same. The intake pipeline changes fundamentally.

The challenge is description quality. A vision model describing a bar chart has to capture the right numeric relationships and trends, not just “a chart with blue bars.” When the description is wrong, the retrieval is wrong.

When it wins: Organizations with large stores of non-text content. Slide decks, technical diagrams, scanned legacy documents.

When it fails: When vision model descriptions are too generic to differentiate documents. Dense technical charts are hard to describe well.

What you give up: Latency at indexing time and dependency on a capable vision model.

Pattern 10: Graph RAG

Here’s where it gets interesting. All the patterns above treat documents as independent chunks. Graph RAG treats them as nodes in a network of relationships.

The insight is that the most useful information is often not what a document says, but how it relates to other documents. In regulatory compliance, you need to know that Regulation A governs Vendor B who is a party to Contract C. A vector search for “Regulation A” gives you the text. A graph traversal gives you the network.

Graph RAG builds a knowledge graph from the corpus, then uses that graph structure during retrieval. Instead of nearest-neighbor search in embedding space, you get path traversal through a semantic graph. Entities, relationships, and their connections all become first-class retrieval targets.

The implementation complexity is non-trivial. Building and maintaining a knowledge graph over a dynamic corpus is a significant engineering investment.

When it wins: Knowledge domains with dense interconnections. Legal, compliance, medical, financial. Any domain where “what’s related to X” matters more than “what mentions X.”

When it fails: When the graph construction is noisy. Bad entity extraction poisons the graph. Also: flat corpora with no meaningful entity relationships.

What you give up: Implementation simplicity. Graph RAG is a significant infrastructure investment.

Which One Should You Actually Use?

Start with Simple RAG and layer on complexity only when you can name the specific failure mode you’re solving. Every pattern above adds latency, infrastructure, and debugging surface. None of them are free.

If your users have sessions and expect continuity, add Memory. If your corpus is inconsistent or untrustworthy, add a Corrective layer. If your queries vary wildly in complexity, add a Router. If you’re in a domain where wrong answers have legal or clinical consequences, Self-RAG is not optional. If your knowledge is fundamentally relational, Graph RAG is the only architecture that actually solves the problem.

The mistake most teams make is jumping to Agentic or Graph RAG because they sound sophisticated, before they’ve actually measured where their current system is failing. Measure first. The failure mode tells you the pattern.

One more thing: these patterns aren’t mutually exclusive. A production system might use Adaptive RAG for routing, CRAG for quality gating, and Memory for session continuity, all in the same pipeline. The ten patterns are building blocks, not competing choices.

Pick the ones that solve actual problems in your system. Not the ones that look impressive in architecture diagrams.

LAST Chance 30% Discount

You wake up, and before you’ve touched your phone, Hermes has already read the news, scanned your portfolio, checked what your competitors launched overnight, compiled it into a PDF, and dropped the whole thing into your Telegram. Zero prompts. Zero tabs. Zero minutes of your morning.

That’s not science fiction. That’s what we’re building here.

But to get there, we need to be clear about something: what you’re building with Hermes is not a chatbot. The distinction matters, and most tutorials skip right past it.

A Chatbot Responds. An AI Agent Acts.

Think of a chatbot like a brilliant librarian who only works on one request at a time. You walk up, ask a question, they answer, and the interaction is complete. They have no idea what you asked yesterday. They don’t remember your preferences. When you come back tomorrow, you’re a stranger.

An AI agent is more like a capable employee. You give them a task and they figure out the steps themselves. They pick the right tools, execute them in sequence, read the results, decide whether the task is done, and loop back if it isn’t. They remember what you worked on. They build skills. They get faster the more they do a particular job.

Hermes was built by Nous Research specifically for that second model. What makes it different from most agent frameworks isn’t the UI or the model support. It’s one architectural decision: every time Hermes finishes a task, it writes down what it learned.

It creates a skill file. A set of reusable instructions, specific to that exact type of work. The next time you ask for something similar, Hermes doesn’t figure out the process from scratch. It reads its own notes. And those notes rewrite themselves over time based on how you actually use the system.

That is the core architecture which we will unpack in the rest of this article.

Here is a video to check out as well.

Why Your Current AI Setup Has Amnesia

Here’s the problem with Claude, ChatGPT, and most AI tools: every new conversation starts from zero. You’ve explained your content style a hundred times. You’ve described your workflow, your preferences, your project names, the specific tone you like for email replies. Every single session, that context evaporates.

This isn’t a bug exactly. It’s a design decision. These systems are optimized for breadth across millions of users, not depth with any single one. They can’t afford to know you specifically.

Hermes is built around the opposite bet. It accumulates a profile of you across sessions: your preferences, your content style, your recurring workflows. You can even feed it your website and ask it to run a web search on you. Hermes builds a soul.md file that captures who you are and how you want things done. A week in, it knows you. A month in, it has context on your projects.

Three things make Hermes structurally different from what you’re probably already using:

Skill files are written after every task. The second time you run the same type of work, Hermes reads its own notes and skips the planning overhead. The tenth run is faster still.

Persistent memory across sessions means your preferences aren’t re-explained every time. Agent context isn’t cleared when you close the window.

Parallel execution means when you give it a multi-part task, Hermes splits it into sub-agents that run at the same time. Four competitors to monitor? Four sub-agents fire simultaneously, not sequentially.

Setting It Up Without Pain

The actual setup is less intimidating than most agent tutorials make it sound. You install from the GitHub repo with a single command. The installer scans your system, fills in what’s missing, and walks you through a configuration flow.

You pick your model provider (Anthropic, a local Ollama model, or whatever you pay for), select a messaging gateway (Telegram, Slack, or WhatsApp), and that’s the skeleton. For Telegram, you’ll need a bot token from BotFather. The process is: open Telegram, find BotFather, type /start, follow the prompts. Five minutes, and you have a token.

For web search, you can self-host Firecrawl or just point Hermes at DuckDuckGo. Both work. The paid options are faster but not required.

Two settings in the dashboard actually matter at the start:

Context compression defaults to 0.5. Leave it there. It keeps the agent’s working memory manageable without losing important context.

Session reset defaults to daily or after an inactivity window. This is fine for most workflows. You can tune everything else after you’ve used it for a week and have real opinions about what bothers you.

The Three Workflows That Changed My Morning

Once Hermes is running, the real test is whether it actually saves time on things you care about. Here are three that work well.

Morning briefing. Type the prompt once. Tell Hermes to pull headlines from specific sources, rank them, compile a PDF, and send it to you at 7am. From that point on, you don’t type it again. Hermes opens the sources simultaneously, reads, ranks, builds, and delivers. Every day, without being asked twice.

Stock watchdog. I follow a small list of positions, but I don’t want to refresh a dashboard every ten minutes. I also don’t want to miss a 5% move. So Hermes runs every weekday at 4:15pm, pulls closing prices, calculates percentage swings, and if something jumped more than 3%, it finds a headline that might explain it. Clean summary, straight to Telegram. No subscriptions needed. And because Hermes wrote a skill file after the first run, the second run was faster. The tenth was faster still.

Competitor intelligence. This is where parallel execution actually shows up. I give Hermes four sources to monitor. It doesn’t work through them one at a time. It spawns four sub-agents that run in parallel, each reading and summarizing independently. A few minutes later, a PDF lands in Telegram with one section per source: new launches, pricing changes, notable announcements. Companies pay market intelligence analysts $60,000 to $120,000 a year to do exactly this. Hermes does it for the cost of whatever hardware you’re already running.

Here’s what the parallel execution model looks like under the hood:

Pick the Right Model for the Job

This is the part every other tutorial buries or skips. Choosing a local model isn’t just about the file size that fits on your machine. It’s about matching the model to what you’re actually asking it to do.

Qwen 2.5 is the one for coding tasks. Gemma 4 and Qwen 38B are the solid choices for agentic tool calling, which is most of what Hermes does. Llama 3.3 handles fast summarization and general assistance well.

The practical setup: run Gemma 4 as your default agentic model, Qwen 2.5 Coder for anything involving code, and Llama for bulk summarization jobs. Keep a frontier model (Claude Sonnet or Opus, whatever you actually pay for) as a fallback for tasks that need deep reasoning or complex creativity. Swap models in Hermes config and you’re done.

The constraint is real: a 9GB local model isn’t going to match a frontier cloud model on complex multi-step reasoning chains. It’s slower and makes more mistakes on complicated tasks. That’s the trade-off. What you get back is zero cost per query, full data privacy, and a system that is entirely yours.

How It Fits with Paid AI Tools

Hermes and Claude or Copilot (or Claude Code, or Cowork) aren’t competing for the same jobs. Not every task needs Opus or Sonnet. Routing the right task to the right model is the actual skill.

News briefings, stock checks, competitor summaries, document reformatting, anything you’d run on a schedule? Local models handle that fine. Research tasks that need real synthesis across conflicting information, writing that needs actual judgment, complex code with edge cases that matter? Use a frontier model.

There’s also a real security difference worth naming. With Claude products, you’re trusting Anthropic’s infrastructure, which has serious security investment behind it. With a self-hosted Hermes, you’re managing the infrastructure yourself. That’s more control and more responsibility. If you’re new to self-hosting or don’t want to stay current on security patches, that’s worth factoring in.

Should You Build This?

Here’s a quick decision frame. Hermes is worth setting up if you run the same types of research or reporting tasks repeatedly, you want your agent to remember how you work across sessions, data privacy matters to you and you want work staying on your own hardware, or you’re comfortable managing a local server or your own machine.

You might be better off with a managed tool like Claude or Claude Code if you want zero infrastructure overhead, you need maximum reasoning quality on complex tasks without compromise, or you’re doing one-off tasks rather than recurring automated workflows.

The two aren’t mutually exclusive. The most efficient setup is exactly what the video shows: local agents handling scheduled, repetitive, structured work; paid frontier models handling the tasks that actually benefit from their capabilities.

Your morning can start with a full briefing already waiting. But that only happens if you set it up.