Many customers ask us about the difference between AI Cost Firewall and VCAL Server, and how to choose between them.

The two products are not the same. They differ in purpose, integration style, and the level of control they give to application teams. But they are built around the same core idea: helping AI applications avoid repeated LLM work, improve visibility into AI usage, and reduce unnecessary token spend.

This article is a simplified guide to the purpose of both products, the differences between them, and the situations where each product fits best.

Intro​

In many AI applications, users repeatedly ask questions that are not completely identical, but are close enough for the system to reuse a previous answer. A support chatbot may receive the same question in different wording. An internal assistant may repeatedly summarize similar documents. A RAG system may answer the same operational question many times for different users.

This is where caching and control layers become useful.

The VCAL project includes two related but different products for this problem: AI Cost Firewall and VCAL Server.

They both help reduce repeated LLM work and unnecessary token spend, but they are designed for different integration styles and different use cases.

The simple difference​

The simplest way to explain the difference is this:

AI Cost Firewall is a gateway.

AI Cost Firewall sits between an application and an LLM provider. It is designed for OpenAI-compatible APIs, so an existing application can often start using it with minimal changes.

VCAL Server is a semantic cache service.

VCAL Server is different. It does not try to behave like OpenAI, Anthropic, Gemini, or any other LLM provider. Instead, it exposes its own REST API for semantic cache operations. Applications integrate with it directly when teams want more control over how cache lookup, reuse, storage, and decision logic work.

Both approaches are useful, but they solve the problem at different levels.

What AI Cost Firewall does​

AI Cost Firewall is an OpenAI-compatible LLM gateway.

In a typical setup, the application sends requests to AI Cost Firewall instead of sending them directly to the LLM provider.

The flow looks like this:

Application
   ↓
AI Cost Firewall
   ↓
LLM provider

AI Cost Firewall can check whether a request is already known, whether an exact or semantically similar answer exists, and whether the upstream LLM call can be avoided.

For applications already using OpenAI-compatible APIs, the integration can be relatively simple. In many cases, the customer mainly changes the API base URL and API key configuration.

For example, instead of sending requests directly to an OpenAI-compatible provider, the application sends them to AI Cost Firewall first.

This makes AI Cost Firewall a good fit when the goal is to add a cost and cache control layer without deeply changing the application logic.

How VCAL Server is different​

VCAL Server is a standalone semantic cache server. It is provider-agnostic. It does not depend on a specific LLM provider, chat API, or model vendor. Instead, VCAL Server works with vectors, cache entries, answer payloads, similarity search, and cache-management logic.

A typical VCAL Server flow looks like this:

User question
   ↓
Application creates an embedding
   ↓
Application searches VCAL Server
   ↓
If a good cached answer exists, VCAL Server returns it
   ↓
If not, application calls the LLM provider
   ↓
VCAL Server stores the new question and answer

This means VCAL Server can be used with OpenAI, Anthropic, Gemini, Mistral, Ollama, Azure OpenAI, OpenRouter, local models, or custom internal AI systems.

The important requirement is that the application must generate or provide embeddings that match the vector dimension configured in VCAL Server.

For example:

Embedding model: 768 dimensions
VCAL Server configuration: 768 dimensions

VCAL Server does not care which LLM provider generates the final answer. It only needs the vector representation and the cache payload that the application wants to store.

Integration difference​

The biggest practical difference is how customers integrate the products.

AI Cost Firewall is designed for low-friction adoption in OpenAI-compatible applications. With AI Cost Firewall, the application can often continue using a familiar OpenAI-compatible request pattern. The firewall handles caching and cost-control logic in the gateway layer.

VCAL Server requires explicit application-level integration. With VCAL Server, the customer’s application needs to add cache lookup and storage logic. The application decides when to call VCAL Server, when to reuse a cached answer, when to call the LLM, and when to store a new answer.

Below is a simplified comparison:

This distinction is important.

AI Cost Firewall is easier when the customer wants to add caching and cost visibility quickly.

VCAL Server is better when the customer wants more direct control over semantic search, cache behavior, thresholds, storage, and integration logic.

Is VCAL Server provider-agnostic?​

Yes, VCAL Server is provider-agnostic because it works below the provider API level. It does not need to know whether the answer comes from OpenAI, Anthropic, Gemini, Mistral, Ollama, or a private model.

The customer's application is responsible for three things:

1. Creating embeddings.
2. Calling VCAL Server for cache lookup and storage.
3. Calling the selected LLM provider when there is no suitable cached answer.

This makes VCAL Server useful in environments where teams use different LLM providers or want to avoid being tied to one API style.

Where AI Cost Firewall can be deployed​

AI Cost Firewall is useful when the main goal is to reduce repeated LLM calls with minimal integration effort.

It is best suited for applications that already use OpenAI-compatible APIs and where teams want to add caching, cost visibility, and control without deeply changing the application code.

Common deployment areas include:

AI chatbots
Customer support bots, internal assistants, and FAQ systems often receive repeated or similar questions. AI Cost Firewall can reduce unnecessary upstream LLM calls and improve response speed.

OpenAI-compatible AI applications
Many applications already use OpenAI-compatible clients or gateways. AI Cost Firewall fits naturally into this pattern because it is designed around OpenAI-compatible API behavior.

Pilot AI deployments
Teams can evaluate LLM cost savings, caching behavior, and observability without rewriting the application. This makes AI Cost Firewall a practical starting point for controlled pilots.

In simple terms, AI Cost Firewall is best suited for situations where the customer wants a practical control layer in front of LLM traffic with minimal application changes.

Where VCAL Server can be deployed​

VCAL Server is useful when the customer wants a dedicated semantic cache service with direct control over integration logic.

It works especially well in systems where the application team wants to decide how cache lookup, similarity thresholds, answer reuse, metadata, and storage should work.

Common deployment areas include:

RAG systems
Retrieval-augmented generation systems often receive repeated questions over the same knowledge base. VCAL Server can help reuse previous answers or decisions when the semantic meaning is close enough.

Enterprise knowledge assistants
Internal assistants for documentation, policies, procedures, and technical support can benefit from semantic caching because many users ask similar questions in different ways.

Agent workflows
AI agents often repeat planning, classification, summarization, or decision steps. VCAL Server can cache these intermediate or final results.

Custom chatbot backends
Teams building their own chatbot logic can use VCAL Server when they need more control than a gateway-only approach. The application decides how to use similarity scores, thresholds, metadata, and stored answers.

On-prem and private infrastructure
VCAL Server can run inside the customer’s own infrastructure. This is useful when data locality, internal control, and private deployment are important.

Provider-independent AI systems
Because VCAL Server is provider-agnostic, it can support applications using different LLM providers, OpenAI-compatible APIs, Anthropic, Gemini, Ollama, local models, or custom internal systems.

In simple terms, VCAL Server is best suited for situations where the customer wants a flexible and powerful semantic cache that the application controls directly.

When to choose AI Cost Firewall (a simple guide)​

AI Cost Firewall is usually the better starting point when the customer wants a faster and simpler integration path.

It is especially useful when:

• the application already uses OpenAI-compatible APIs
• you want minimal code changes
• the main goal is to reduce repeated LLM calls
• the team wants cost visibility and cache metrics quickly
• the application architecture is suitable for a gateway layer

In simple terms, choose AI Cost Firewall when you want a practical gateway for reducing repeated LLM traffic with minimal or no application changes.

When you need VCAL Server​

VCAL Server is usually the better choice when the customer wants more control and is ready to integrate caching directly into the application workflow.

It is especially useful when:

• the application uses multiple LLM providers
• you need provider-agnostic caching
• the team wants direct control over similarity thresholds and cache behavior
• the system is a custom RAG, agent, or chatbot backend
• you want a standalone semantic cache inside your own infrastructure

In simple terms, choose VCAL Server when you want a flexible and powerful semantic cache service that your application controls directly.

Compliance-conscious deployment​

For many organizations, reducing LLM cost is only part of the problem. AI systems also need to fit internal security, privacy, governance, and regulatory requirements.

AI Cost Firewall and VCAL Server are designed to support compliance-conscious deployments.

They do not force teams to send their data through an uncontrolled external layer. Both products can be deployed inside the customer’s own infrastructure, where the organization can control network access, API keys, logs, metrics, retention policies, and operational monitoring.

This is especially important for teams working with customer support data, internal documentation, or sensitive operational information.

Can both products be used together?​

Yes, in some architectures they can complement each other.

AI Cost Firewall can provide a gateway layer for OpenAI-compatible traffic, while VCAL Server can provide a deeper semantic cache service for custom workflows, RAG systems, or application-controlled cache operations.

However, they do not have to be used together. Each product has its own purpose:

• AI Cost Firewall is focused on easy gateway-style adoption
• VCAL Server is focused on flexible, provider-agnostic semantic cache infrastructure

Final thoughts​

AI Cost Firewall and VCAL Server solve related problems, but they are not the same product.

AI Cost Firewall is an OpenAI-compatible gateway that helps applications reduce repeated LLM calls with minimal changes.

VCAL Server is a provider-agnostic semantic cache service that gives applications direct control over cache lookup, reuse, storage, and integration logic.

Both products are designed for real AI workloads where repeated or similar requests create unnecessary cost, latency, and infrastructure load. The right choice depends on the customer’s architecture.

If the goal is fast adoption with an OpenAI-compatible application, AI Cost Firewall is usually the better starting point.

If the goal is deeper control, provider independence, and custom integration, VCAL Server is the better fit.

I hope this guide helps you choose the right product for your AI application.

If you need more details, you can read the VCAL documentation or contact us directly.

When people talk about Edge AI, the conversation usually revolves around models. Larger context windows, smaller quantized variants, GPU acceleration, inference speed, and hardware optimization tend to dominate the discussion. But in practice, many real-world Edge AI deployments are constrained not by the model itself, but by the operational realities surrounding it.

Running AI at the edge means running software in environments that are fundamentally different from modern cloud infrastructure. These systems may operate with limited memory, modest CPUs, unreliable connectivity, restricted storage, or strict uptime requirements. They may be installed in factories, telecom cabinets, or remote locations where updates are difficult and maintenance windows are limited.

In these environments, the infrastructure surrounding the AI model becomes critically important.

Inference alone is rarely enough. Real systems require routing, telemetry, caching, authentication, observability, synchronization, and APIs to name a few. As Edge AI deployments mature, the supporting software stack increasingly determines whether the system remains practical to operate over time.

This is where small Rust binaries become unexpectedly valuable.

A simplified comparison illustrates how deployment characteristics differ between traditional runtime-heavy stacks and lightweight Rust-based infrastructure.

A great deal of modern AI infrastructure is assembled from heavyweight runtime environments. Python services, Node.js middleware, layered containers, package managers, dynamically linked dependencies, and multiple supporting sidecars work reasonably well in large cloud environments where storage, memory, and compute resources are abundant. Cloud-native ecosystems are optimized around this assumption.

Edge systems operate under very different constraints.

A deployment that feels trivial in Kubernetes can become fragile when moved into a constrained ARM device or an industrial Linux gateway. Every additional runtime layer increases image size, startup complexity, memory overhead, update time, and operational surface area. In disconnected or bandwidth-constrained environments, even transferring updates can become an engineering problem.

The difference between a small statically linked binary and a multi-hundred-megabyte container image is not merely aesthetic. It changes how systems behave operationally.

Rust fits this environment unusually well because it combines low-level efficiency with modern software engineering practices. Rust applications can achieve native performance while maintaining memory safety without relying on garbage collection. They can often be compiled into small standalone binaries with minimal runtime dependencies, especially when using musl-based static builds.

Operationally, this has several consequences.

Deployment becomes simpler because the application can often be distributed as a single executable without requiring external runtimes or complex dependency trees. Updating edge systems becomes faster because smaller artifacts transfer more efficiently across constrained networks. Startup times improve because the system avoids initializing large runtime environments. Memory consumption remains relatively predictable, which matters when multiple services must coexist on limited hardware.

Perhaps more importantly, the operational model itself becomes easier to reason about.

In edge environments, reliability frequently depends on reducing complexity rather than adding abstraction. A smaller deployment footprint generally means fewer moving parts, fewer compatibility problems, fewer patching requirements, and fewer failure scenarios. This becomes increasingly important in industrial or regulated environments where operational stability matters more than developer convenience.

Static linking is one of the more underrated aspects of this approach. A fully self-contained binary changes deployment from “install and configure an environment” into something much closer to “copy and run.” That may sound simplistic, but at scale, simplicity becomes an operational advantage. Especially in disconnected or semi-connected infrastructure, reducing environmental assumptions can significantly improve reliability.

Edge AI also introduces another practical challenge: repeated computation.

Many edge systems process highly repetitive workloads. Similar prompts, recurring semantic queries, repeated retrieval patterns, and predictable operational workflows appear constantly in production environments. Without an intermediate control layer, these systems repeatedly recompute results they have effectively already seen before.

This creates unnecessary pressure on hardware resources that are already constrained.

Semantic caching becomes particularly useful at the edge because compute resources are finite and expensive. Reducing repeated inference can improve responsiveness, lower latency, decrease hardware utilization, and reduce energy consumption. In disconnected environments, lightweight caching layers can also help systems remain responsive even when upstream connectivity is unstable or unavailable.

This is one reason why we believe lightweight infrastructure matters just as much as lightweight models.

At VCAL Labs, we have been building semantic infrastructure components in Rust not only because of performance characteristics, but because operational portability increasingly matters in AI systems. Small binaries, predictable resource usage, and minimal deployment friction align naturally with the realities of edge environments.

The future of Edge AI will likely depend on more than just advances in inference itself. It will also depend on whether the surrounding infrastructure can operate reliably in constrained, distributed, and operationally complex environments.

In many cases, the hardest problem is not running the model once.

The harder problem is continuously operating the systems around it.

Observability, routing, semantic caching, synchronization, APIs, telemetry, and traffic control all become part of the deployment surface. And as AI infrastructure moves closer to the edge, operational efficiency starts to matter just as much as raw model capability.

Small Rust binaries are not a universal solution to these problems. But they are remarkably well aligned with the operational realities of Edge AI infrastructure, and that alignment is becoming increasingly difficult to ignore.

Learn more about VCAL semantic infrastructure for AI systems: https://vcal-project.com/

AI Cost Firewall began as a lightweight OpenAI-compatible gateway designed to reduce LLM cost and latency through exact and semantic caching. Over time, the project evolved beyond simple request reuse and gradually became a broader operational layer for AI infrastructure.

Recent releases introduced semantic cache lifecycle management, provider flexibility, improved Prometheus and Grafana observability, configuration diagnostics, and detailed cost accounting. Those features improved the technical capabilities of the system significantly, but another important question remained:

How quickly can somebody actually deploy and evaluate the system in a real environment?

That question became the main focus of v0.1.9.

Unlike earlier releases that concentrated on internal infrastructure features, v0.1.9 focuses primarily on operational polish. The goal of this release is to reduce the friction between discovering the project and successfully running it with dashboards, cache reuse, and observable semantic behavior.

In practice, this means clearer deployment patterns, better onboarding, improved startup diagnostics, more actionable provider error messages, and significantly expanded operational documentation.

A Shift Toward Real Deployments​

One recurring observation during development was that many evaluation problems were not caused by semantic cache logic itself. Instead, the most common issues were operational:

• wrong provider base URLs
• Docker networking confusion
• embedding dimension mismatches
• empty dashboards
• TLS and certificate problems
• misunderstanding how semantic cache behaves

These are normal operational problems for infrastructure software, but they become barriers when somebody is evaluating a project for the first time.

v0.1.9 therefore introduces a more deployment-oriented structure for the repository and documentation. The project now includes runnable deployment examples under:

deploy/examples/

The new examples are designed to demonstrate practical deployment patterns instead of only isolated configuration snippets.

Included examples:

Each example includes a runnable Docker Compose stack, minimal configuration, example requests, expected behavior, and optional observability overlays where appropriate.

The intent is not only to make deployments easier, but also to make them more understandable.

A Practical Hybrid Deployment​

One particularly useful deployment pattern introduced in the examples is:

hybrid-openai-local-embeddings/

This deployment combines cloud chat inference with local embeddings.

In this setup:

• OpenAI handles chat completions
• Ollama generates embeddings locally
• Redis stores exact cache entries
• Qdrant stores semantic cache vectors

This pattern is interesting because it demonstrates a practical middle ground between fully local infrastructure and fully cloud-hosted inference.

Many organizations still want the quality and convenience of cloud-hosted chat models, but embedding overhead can become expensive once semantic cache traffic grows. Running embeddings locally can reduce or eliminate those costs while still preserving semantic reuse behavior.

The request flow becomes:

Application
    ↓
AI Cost Firewall
    ↓
Exact Cache (Redis)
    ↓
Semantic Cache (Qdrant + local embeddings)
    ↓
OpenAI upstream

This arrangement also keeps GPU requirements relatively modest compared to fully local chat inference stacks.

Faster Evaluation Flow​

The deployment examples intentionally avoid unnecessary complexity. The objective is to help operators get from zero to a working observable deployment as quickly as possible.

A typical startup sequence now looks like:

docker compose up -d
docker compose exec ollama ollama pull nomic-embed-text
docker compose restart ai-firewall

After startup, the deployment can immediately be validated using the health and readiness endpoints:

curl http://localhost:8080/healthz
curl http://localhost:8080/readyz

A simple request can then be sent through the firewall:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini-2024-07-18",
    "messages": [
      {"role": "user", "content": "Explain Redis briefly."}
    ]
  }'

Repeated requests should begin producing exact cache hits, while semantically similar prompts may eventually reuse semantic cache entries.

The important part here is not merely that caching exists, but that the behavior becomes visible and understandable during evaluation.

Observability as a First-Class Feature​

Semantic caching can easily become opaque if operators cannot see what the system is doing internally. One of the long-term goals of AI Cost Firewall has therefore been making semantic reuse behavior observable instead of hidden.

v0.1.8 introduced more advanced financial and cache metrics, while v0.1.9 improves the operational deployment and interpretation of those dashboards.

The project includes two Grafana dashboards with different purposes.

The Overview dashboard (see cover image) focuses on higher-level operational and business metrics such as:

• request traffic
• exact cache hits
• semantic cache hits
• gross savings
• embedding overhead
• net savings

Meanwhile, the Diagnostics dashboard focuses more heavily on semantic runtime behavior:

• semantic lookup latency
• threshold pass/fail behavior
• semantic candidate activity
• runtime cache diagnostics

This separation is intentional.

The Overview dashboard helps answer:

“Is this deployment actually reducing cost and upstream traffic?”

The Diagnostics dashboard helps answer:

“Why is semantic cache behaving the way it is?”

That distinction becomes increasingly important as semantic cache deployments grow larger and more complex.

Operational Problems Are Real Problems​

One of the themes of v0.1.9 is that operational clarity matters just as much as architecture.

Even a technically strong caching system becomes difficult to evaluate if deployment failures are confusing or poorly explained. For that reason, this release also improves startup diagnostics, runtime validation, and provider error handling.

Several deployment mistakes appeared repeatedly during testing and evaluation.

Wrong Base URLs​

A very common issue is configuring full endpoint paths instead of provider base URLs.

Incorrect:

https://api.openai.com/v1/chat/completions

Correct:

https://api.openai.com

AI Cost Firewall appends OpenAI-compatible routes internally. The same rule applies to embedding endpoints.

v0.1.9 improves diagnostics around this issue and makes related provider failures easier to interpret.

Qdrant Vector-Size Mismatches​

Another common operational problem involves embedding dimensions.

Different embedding models produce vectors of different sizes:

nomic-embed-text → 768
text-embedding-3-small → 1536

If the Qdrant collection vector size does not match the configured embedding model, semantic cache behavior will fail.

Earlier releases already validated vector sizes, but v0.1.9 improves the clarity of those startup diagnostics and explains the likely cause more explicitly.

Docker Networking Confusion​

Docker networking also causes frequent evaluation problems.

Inside containers:

localhost != host machine

This especially affects Ollama deployments.

Incorrect inside Compose networking:

http://localhost:11434

Correct:

http://ollama:11434

v0.1.9 expands troubleshooting documentation around these operational patterns.

Empty Dashboards​

Sometimes the infrastructure is healthy but Grafana dashboards remain empty.

This is usually caused by:

• no traffic being generated yet
• Prometheus scrape failures
• dashboard provisioning path problems
• observability overlays not running

Useful checks include:

curl http://localhost:8080/metrics

and:

http://localhost:9090/targets

The release documentation now explains these scenarios more clearly.

TLS and Self-Signed Certificates​

OpenAI-compatible providers are often deployed internally with self-signed certificates or non-public trust chains.

As a result, TLS problems became another recurring evaluation issue.

v0.1.9 improves diagnostics for:

• hostname mismatch
• SAN mismatch
• self-signed certificates
• TLS handshake failures
• provider connectivity failures

The objective is to make startup and provider failures more actionable for operators instead of surfacing only generic upstream errors.

Why This Release Matters

v0.1.9 is intentionally less focused on introducing major new algorithms or architectural subsystems. Instead, it focuses on operational maturity.

In practice, infrastructure software becomes useful only when people can deploy, observe, troubleshoot, and understand it quickly. That is especially true for semantic caching systems, where invisible behavior can otherwise become difficult to reason about.

This release is therefore an important transition point for AI Cost Firewall. The project is evolving from an experimental semantic cache layer into a more practical operational gateway for OpenAI-compatible AI infrastructure.

Resources

GitHub:

https://github.com/vcal-project/ai-firewall

Documentation:

https://ai-firewall.docs.vcal-project.com/

Originally published on Medium.com on April 22, 2026.
Read the Medium.com version

As LLM applications move from demos into production, many teams double down on one thing: prompt security. They refine system prompts, add guardrails, introduce moderation, and carefully control how users interact with the model. And yet, once real traffic arrives, something unexpected happens.

At first, everything works. The demo is smooth, responses are fast, costs are negligible.

But soon real usage begins. Costs spike, latency becomes inconsistent, errors become harder to understand, deployments start affecting live requests in subtle ways.

Nothing is obviously broken, but the system no longer feels predictable.

The Application Layer: Controlling Meaning and Behavior​

The application layer is where the logic of an AI product lives. It defines how prompts are constructed, how users interact with the system, and what the model is allowed to do.

This is where most teams focus first — and for good reason. Here, you are dealing with meaning, intent, and safety.

At this layer, the focus is on controlling what the model is allowed to do. In practice, that translates into questions like:

• Can a user manipulate the model through prompt injection?
• Can sensitive data leak through responses?
• Are outputs aligned with policy and expectations?

To solve this, teams build a combination of structural and defensive controls:

• Structured prompts and system messages
• Input validation and sanitization
• Output filtering and moderation
• Access control and business logic

These mechanisms are essential. Without them, the system is exposed at the semantic level.

In short, the application layer protects what the model means and does.

The Missing Layer: Controlling Execution and Behavior Under Load​

However, there is another category of problems that has nothing to do with meaning. They emerge only when the system is under real conditions.

Here are the most usual ones:

• Prompts gradually become larger and heavier
• Similar requests are repeated again and again
• Upstream providers introduce latency or intermittent failures
• Errors return in inconsistent formats
• Deployments interrupt in-flight requests

These are not prompt problems. They are system behavior problems. The application layer answers: “Is this prompt safe?” The next layer answers: “Should this request exist at all?”

This is where the AI Cost Firewall layer comes in.

Two distinct layers in LLM systems: the application layer controls meaning and safety, while the AI Cost Firewall controls execution, cost, and reliability.

Sitting between the application and the LLM provider, it acts as a control plane for LLM traffic. Its role is not to understand the prompt, but to ensure that every request is handled in a controlled, predictable, and observable way.

At this layer, the focus shifts to:

• How large is the request?
• Should this request even reach the provider?
• Is this a duplicate or semantically similar request?
• Did the upstream fail, timeout, or respond incorrectly?
• What happens if the system is shutting down?

To answer these, the AI Cost Firewall introduces the following guardrails:

• Prompt size supervision and request validation
• Error classification and normalized responses
• Timeout handling and upstream protection
• Exact and semantic caching
• Readiness checks and graceful shutdown behavior

This layer protects how the system executes and consumes resources.

Why These Layers Are Not Interchangeable​

It’s tempting to think that strong application-layer security is enough.

The difference becomes obvious when you look at what each layer is actually responsible for.

Two layers of LLM systems: controlling meaning vs controlling behavior

A perfectly secured prompt can still result in:

• A 2MB payload that strains your system
• Thousands of repeated requests driving unnecessary cost
• Silent upstream timeouts with no clear diagnostics

In other words, semantic safety does not guarantee operational stability.

A Familiar Pattern, Just Not Yet in AI​

If this separation feels unfamiliar, it’s only because LLM systems are still new.

In traditional web architecture, this distinction is well understood:

• the application handles authentication, authorization, and business logic
• the infrastructure layer (reverse proxy, API gateway) handles request validation, rate limiting, retries, and observability

In that world, tools like Nginx became essential — not because they understand your business logic, but because they control how requests flow through the system.

The same pattern is now emerging in AI systems.

AI Cost Firewall plays a role similar to Nginx but for LLM traffic.

It does not interpret prompts or enforce business rules. Instead, it ensures that every request is well-formed, controlled, observable, and efficient before it reaches the model.

And just like in web systems, skipping this layer might work in a demo, but it rarely holds up in production.

No one would deploy a production system that sends raw traffic directly to back-end services without passing through a control layer. And yet, this is exactly how many LLM applications operate today.

What Changes in Production​

In early prototypes, everything seems fine. Traffic is low, prompts are short, and errors are rare.

However, production changes the situation completely:

• Prompts accumulate context and silently grow
• Users repeat similar queries in slightly different forms
• Costs scale faster than usage
• Failures become harder to classify and debug

These issues don’t break the system immediately. They degrade it gradually, until behavior becomes unpredictable.

This is precisely the gap the AI Cost Firewall layer is designed to address.

The Key Insight​

LLM security is not just about safe prompts. It’s about safe system behavior under real conditions.

The application layer ensures the model behaves correctly. The AI Cost Firewall ensures the system behaves reliably.

Both are required to move from a working demo to a production-grade system.

Final Thought​

Most teams start by asking: “How do we control what the model says?”

But in production, a more important question emerges: “How do we control what the system does under real conditions?”

That’s where the second layer becomes essential.

Originally published on Dev.to on April 13, 2026.
Read the Dev.to version

A month ago, I wrote about reducing LLM costs using caching.

The idea is simple: don’t send the same or similar request to the model twice.

It works well in demos. It even works well in early testing.

And then production starts.

Production Reality: Where LLM Systems Start Breaking​

At first, everything looks under control. Requests are small, traffic is predictable, and caching delivers immediate savings. You see fewer calls to the model and faster responses. It feels like the problem is solved.

But real systems don’t stay simple for long.

Prompts begin to grow. What used to be a short question turns into a long conversation with accumulated context, system instructions, and sometimes entire documents pasted by users. Requests become heavier, slower, and more expensive in ways that caching alone cannot fix.

At the same time, failures start to blur together. A timeout, a malformed request, and an upstream provider error all look the same from the outside. Without clear separation, debugging becomes guesswork, and cost anomalies become difficult to explain.

Then there’s latency. A request times out — but what actually happened? Was the provider slow? Did the request even reach it? Should you retry it or not? Without visibility into upstream behavior, you’re operating blind.

Even semantic caching, which looks almost magical at first, becomes a tuning problem. Similarity thresholds that worked in testing suddenly feel off. Some responses are reused too aggressively, others not at all. Without insight into what the system is actually doing, you’re left adjusting numbers and hoping for the best. This is all similar to how prompts are tuned — but here, the feedback loop is missing.

Finally, the moment that exposes everything: deployment.

You restart the service during traffic, and suddenly there are dropped requests, inconsistent responses, and unpredictable behavior. What worked perfectly in isolation now reveals gaps in lifecycle handling.

The Missing Layer: LLM Systems Have No Traffic Control​

What all of this points to is a deeper issue.

LLM applications don’t just need optimization. They need a control layer.

In traditional systems, we never send traffic directly to application logic. There is always a layer in front — something that validates, routes, filters, and observes. Tools like Nginx became essential not because they were convenient, but because they made systems predictable.

LLM systems are now facing the same reality.

From Calling Models to Controlling Requests​

When you introduce a control layer in front of LLMs, the perspective changes.

The question is no longer just “how do I call the model?” but “should this request reach the model at all?”

Is it valid? Has it already been answered? What happens if it fails?

Cost optimization becomes a side effect of something bigger: managing traffic properly.

From Caching to Control: What Changed in Real Deployments​

This is where the AI Cost Firewall evolved.

It started as a caching layer — combining exact matches in Redis with semantic search in Qdrant. That alone reduced a significant portion of redundant requests.

But real deployments made it clear that caching is only the beginning. The system needed to behave predictably under load, during failures, and across deployments. So the focus shifted.

Readiness and liveness became explicit, separating a healthy process from one that is actually ready to handle traffic. Shutdown behavior was redesigned to drain in-flight requests instead of dropping them. Restarts became controlled events rather than risky moments.

Errors were no longer just errors. They were classified: validation issues, upstream timeouts, provider failures, internal faults — each telling a different story about what went wrong.

Upstream behavior stopped being a black box. Latency became measurable, timeouts became visible, and slow responses could finally be distinguished from real failures.

Semantic caching also became observable. Instead of guessing whether it works, you can now see how many candidates are evaluated, how often thresholds pass or fail, and how long lookups take. What used to feel like a heuristic now becomes something you can tune with confidence.

And perhaps most importantly, the system itself became visible while it runs. You can tell whether it is ready, whether it is shutting down, and how it behaves under real traffic — not just in theory.

At this point, semantic caching stops being a black box.

This is what diagnostics visibility looks like in practice:

Instead of guessing thresholds, you now have feedback. Instead of assumptions, you have data.

Why Predictability Matters More Than Features​

None of these changes are flashy.

They don’t improve model quality or add new capabilities.

But they solve something more fundamental: they make the system predictable.

And without predictability, cost optimization doesn’t hold.

Caching Starts It, Control Makes It Work​

Reducing LLM costs is easy when everything is controlled and small.

It becomes difficult when requests grow, failures mix together, and systems need to operate continuously under real conditions.

At that point, the problem is no longer about saving tokens. It’s about understanding and controlling the flow of requests before they ever reach the model.

Caching is where it starts. Control is what makes it work in production.

If you want to experiment with the tool, the AI Cost Firewall project is open-source and designed to run as a drop-in OpenAI-compatible gateway in front of existing applications:

https://github.com/vcal-project/ai-firewall

Built and maintained by the VCAL Project team — feedback and real-world use cases are very welcome.

Originally published on Medium.com on April 5, 2026.
Read the Medium.com version

After publishing the first release of AI Cost Firewall, I thought the hard part was done.

The idea was simple and it worked immediately: avoid sending duplicate or semantically similar requests to the LLM, and you reduce cost.

I described that initial approach in more detail here: How to Reduce OpenAI API Costs with Semantic Caching

And it did work.

But once I started pushing it further — adding more metrics, handling edge cases, running real traffic through it — it became clear that the initial idea was only a small part of the problem.

Reducing LLM cost is not just about caching. It’s about understanding where the cost actually comes from, what “savings” really mean, and what begins to break when a system moves from a controlled demo into something closer to production.

The First Insight Still Holds​

The original observation hasn’t changed, and neither has the core architecture. The system still solves the same underlying problem.

• Users repeat themselves.
• Applications repeat themselves.
• Agents repeat themselves.

Often the wording changes slightly, but the intent remains the same. From the model’s perspective, however, every variation is a brand new request. And every request has a cost.

So yes — caching works. It reduces cost immediately, often without any changes to the application itself.

But that’s only the surface. The deeper questions only appear once you try to rely on it.

The First Misconception: “Caching Is Free”​

In the beginning, the results looked almost too good.

In a demo environment, exact cache hits dominated. When a request hit the cache, it meant no API call, no tokens, and almost zero latency. It felt like pure gain, as if cost reduction came with no trade-offs.

That illusion disappears the moment you introduce semantic caching properly. Because semantic caching requires embeddings.

To determine whether two requests are similar, you first need to convert them into vectors. That means calling an embedding model, storing the result, and comparing it against existing data. Only then can you decide whether to reuse a response or forward the request to the LLM.

And embeddings are not free.

At that point, the equation changes:

Net savings = avoided LLM cost − embedding cost

This is where things become more delicate.

If your similarity threshold is too low, you generate embeddings too often. If your traffic is highly unique, most of those embeddings never lead to a cache hit. If your embedding model is expensive, the optimization starts working against you.

What initially looked like a simple cost reduction mechanism becomes something that requires careful balance.

That was the moment when the project stopped being just a clever shortcut and started behaving like a system that needs tuning.

Where the Savings Actually Come From​

Looking at real metrics changed another assumption.

Intuitively, semantic caching feels like the main feature. It’s the “intelligent” part of the system. But in practice, most of the savings come from something much simpler — exact matches.

A surprisingly large portion of traffic is not just similar — it is identical. The same prompt appears again and again, sometimes minutes apart, sometimes hours later. Once you see it in real data, it’s hard to ignore.

Semantic caching still matters, but its role is different. It extends the coverage rather than forming the base.

Without exact caching, the system loses most of its immediate impact. Without semantic caching, you miss additional opportunities. But they are not equal contributors, and treating them as such leads to wrong expectations.

What Breaks in Real Usage​

As soon as real traffic enters the system, subtle issues begin to surface.

One of the first is payload size.

LLM requests tend to grow over time. Prompts accumulate context, system messages expand, conversation history becomes longer. In some cases, payloads become unexpectedly large — either naturally or intentionally.

Without limits, a single request can consume disproportionate resources. What seemed like a minor edge case quickly turns into something that needs explicit control.

Another issue is validation.

If you accept any model name, any payload structure, and any input format, your system remains flexible but your metrics lose meaning. Cost calculations become inconsistent, comparisons stop being reliable, and “savings” become difficult to interpret.

Adding strict validation changes that. It makes the system more predictable, but also more opinionated.

At that point, it is no longer just a transparent proxy. It becomes a controlled gateway. And that shift is intentional.

It’s worth noting that this layer is not a security firewall in the traditional sense. It does not attempt to detect malicious prompts, prevent prompt injection, or enforce content policies. Those concerns belong to the application layer, where context, user intent, and business logic are better understood. The goal here is different: to control cost, reduce unnecessary requests, and make LLM usage more predictable.

Architecture Matters More Than It Seems​

From the outside, the architecture still looks simple. One layer in front of the LLM, minimal components, no intrusive changes to the application.

But over time, the reasoning behind each choice becomes more important.

• Redis handles exact matches because it is fast and predictable.
• Qdrant supports semantic search efficiently without adding unnecessary complexity.
• Rust ensures that this layer can sit in the request path without introducing latency or instability.

Individually, these are straightforward decisions.

Together, they define whether the system can operate reliably under load. Because once this layer becomes part of the critical path, it is no longer optional. If it slows down, everything slows down. If it fails, everything fails.

At that point, optimization is no longer the only goal. Stability becomes equally important.

The Difference Between Demo Traffic and Reality​

It is easy to produce impressive results in a controlled environment.

• high cache hit rates
• clear cost savings
• clean, predictable behavior

But those results are shaped by the input.

Real systems behave differently. There are always new queries, unexpected variations, and edge cases that were not part of the initial design. You never reach 100% cache hits — and that’s not a failure.

A healthy system still generates misses. It still calls the model. It still adapts to new inputs.

The goal is not to eliminate LLM usage entirely. The goal is to eliminate unnecessary usage. That distinction becomes much more important once you move beyond a demo.

Where This Is Going​

What started as a simple way to avoid duplicate requests is gradually evolving into something broader.

It is no longer just about caching. It becomes a control layer for how LLMs are used:

• cost visibility
• request validation
• provider abstraction
• traffic control

Caching is still at the center, but it is no longer the whole story. It is the entry point into a larger set of concerns that appear once LLM usage grows.

Final Thoughts​

The original idea was simple: Don’t pay twice for the same answer.

That still holds. But applying that idea in a real system reveals a different challenge.

The difficult part is not avoiding the call. It is understanding when avoiding it actually makes sense. Because cost optimization, in practice, is not just about reducing usage. It is about understanding your system well enough to reduce costs in the right way.

You can explore the project here:

https://github.com/vcal-project/ai-firewall

Originally published on Medium.com on March 21, 2026.
Read the Medium.com version

A simple OpenAI-compatible gateway that eliminates duplicate requests and cuts token usage​

While working on LLM-powered tools for my customer, I kept seeing something that didn’t feel right.

Users were asking the same or similar questions again and again. Support queries repeated. Internal assistants received nearly identical prompts. Even AI agents were looping through similar requests.

At first, it didn’t look like a problem. That’s just how users behave.

But then I looked at the cost.

Every repeated question meant another API call. Another batch of tokens. Another charge. Over time, it added up more than was expected.

I realized something simple:

We are paying multiple times for the same answer.

Why Existing Solutions Didn’t Quite Work​

Initially I looked at the available tools.

Redis helped with exact caching, but only when the prompt was identical. The moment a user rephrased the question slightly, the cache missed. “How do I get access to Jira?” and “Cannot get access to Jira” were treated as completely different requests.

I also explored RedisVL, which brings vector search capabilities into Redis. It moves in the right direction by combining caching and similarity in one place. But in practice, it still requires setting up embedding flows, defining schemas, tuning similarity thresholds, and integrating it manually into the LLM request pipeline.

Vector databases like Milvus, Weaviate, or Qdrant seemed promising as well. They can detect semantic similarity effectively, but integrating them into the request flow means building additional pipelines, managing embeddings, and writing glue code.

All of these tools are powerful, but they aren’t simple.

More importantly, none of them are designed as a drop-in layer in front of an LLM API. There was no unified solution that combined caching, semantic matching, and cost awareness in one place.

What I Built Instead​

At some point, I decided to take a step back and ask a simple question:

What if we just put one smart layer in front of the LLM?

That’s how the AI Cost Firewall started.

Instead of modifying applications or adding complex pipelines, AI Firewall intercepts requests before they reach the model. If it’s already seen a similar request, it returns the cached response. If not, it forwards it, stores the result, and moves on.

From the application’s perspective, nothing changes. It still talks to an OpenAI-compatible API.

But behind the scenes, unnecessary calls disappear.

How It Works (Without the Complexity)​

I intentionally kept the architecture minimal.

At the core, there’s a Rust-based API gateway that speaks the same language as the OpenAI API. For caching, I use Redis for exact matches and Qdrant for semantic similarity. Prometheus and Grafana provide visibility into what’s happening.

A request comes in, we check the cache, and only if needed do we call the LLM.

That’s it.

No SDK rewrites. No major architectural changes. Just one additional layer.

Why I Chose Rust​

Since this component sits directly in the request path, performance matters.

I chose Rust because it provides low latency and predictable performance without garbage collection pauses. It handles concurrency well and keeps the memory footprint small, which makes it ideal for containerized deployments.

Most importantly, we can trust it not to become the bottleneck.

Why I Open-Sourced It​

This layer sits between the application and the AI provider. That’s a sensitive place.

I felt it had to be transparent and auditable. Open source makes it easier to trust, easier to adopt, and easier to extend.

It also keeps the core idea simple: reducing costs shouldn’t introduce new risks or lock you into a vendor.

Getting Started in Minutes​

I wanted the setup to be as simple as possible.

Clone the repository, start Docker, and point your application to a new endpoint.

git clone https://github.com/vcal-project/ai-firewall
cd ai-firewall
cp configs/ai-firewall.conf.example configs/ai-firewall.conf
nano configs/ai-firewall.conf # Replace the placeholders with your API keys
docker compose up -d

After that, you just replace your API base URL with:

http://localhost:8080/v1/chat/completions

That’s the entire integration.

What Changed for Me​

Once I started using this approach, two things became obvious.

First, a surprisingly large portion of requests was served directly from cache. The reason? All of them were already answered before.

Second, response times improved whenever the cache was hit.

I didn’t need to optimize prompts or switch models to see an effect. Just avoiding redundant calls made a noticeable difference.

To make this visible, I add a simple Grafana dashboard.

It shows how many requests are served from cache vs forwarded to the LLM, along with the estimated cost savings in real time.

The key metrics are:

• cache hit ratio (how many requests never reach the LLM)
• total tokens saved
• estimated cost savings

What surprised me most was how quickly the savings accumulated even with relatively small traffic.

What Comes Next​

I see this as a starting point rather than a finished product.

Next, I’m focusing on adding support for other LLM providers beyond OpenAI. Expanding analytics is another priority, along with exploring multi-model setups and smarter routing.

There’s still a lot to build — and that’s exactly the point.

Final Thoughts​

AI costs don’t spike all at once. They grow quietly, request by request.

And in many cases, a large part of that cost is unnecessary.

We didn’t need a more complex system to reduce it. We just needed to stop sending the same request twice.

Sometimes the most effective optimization is the simplest one:

Not calling the model at all.

If you’re running LLM-powered tools and want to reduce costs without changing your application architecture, you can try it here:

https://github.com/vcal-project/ai-firewall

Originally published on Dev.to on March 16, 2026.
Read the Dev.to version

Exact + semantic caching for AI applications​

In today’s era of AI adoption, there is a distinct shift from integrating AI solutions into business processes to controlling the costs, be it the costs of a cloud solution, a local LLM deployment, or the cost of tokens spent in chatbots. If your solution includes repeated questions and uses an OpenAI-compatible model, and if you are looking for a simple, free and effective way to immediately cut your company’s daily token costs, there is one infrastructural solution that does it right out of the box.

AI Cost Firewall is a free open-source API gateway that decides which requests actually need to reach the LLM and which can be answered from previous results without additional token costs.

The gateway consists of a Rust-based firewall “decider”, a Redis database, a Qdrant vector store, Prometheus for metrics scraping, and Grafana for monitoring. All the tools are deployed with a single docker compose command and are available for use in less than a minute.

Once deployed, AI Cost Firewall sits transparently between your application and the LLM provider. Your chatbot, AI assistant, or internal automation continues to send requests exactly the same way as before with the only difference that the API endpoint now points to the firewall instead of directly to the model provider. The firewall then performs an instant check before deciding whether the request should actually reach the LLM and raise your monthly bill.

How the firewall reduces token costs​

AI Cost Firewall eliminates unnecessary token spends using two layers of caching.

Exact match cache (Redis / Valkey)​

The first step is an extremely fast exact request match check. Each incoming request is normalized and hashed. If an identical request was previously processed, the firewall immediately returns the stored response from Redis. This lookup takes microseconds and costs zero tokens. For workloads with frequent identical prompts such as customer support or internal documentation assistants this alone can already reduce a significant portion of LLM traffic.

Semantic cache (Qdrant)​

The second layer addresses the case of semantic similarity: questions are similar but not identical.

For example:

User A: Provide a one-sentence explanation of what Kubernetes is.
User B: What is Kubernetes? Give me a one-sentence explanation.

Even though the wording differs, the semantic value and thus meaning of these questions is essentially the same (if you are interested in what semantics in AI is, have a look at my article From words to vectors: how semantics traveled from linguistics to Large Language Models).

To detect these situations, AI Cost Firewall uses a semantic vector search. Each request is embedded using a lightweight embedding model, and the resulting vector is compared against previously stored queries using Qdrant, a high-performance vector database designed specifically for this. If the similarity score exceeds a certain threshold, the firewall returns the previously generated answer instead of sending the request to the LLM again. In this way, a single LLM response can be reused dozens or even hundreds of times without extra tokens expense.

Forwarding only when necessary​

If neither the exact cache nor the semantic cache contains a suitable answer, the firewall forwards the request to your upstream model provider. Besides being provided to the user, the returned response is then stored in both Redis and Qdrant for future reuse. The workflow therefore becomes (simplified):

Client → AI Cost Firewall
           ↓
      Redis check
           ↓
     Qdrant semantic check
           ↓
   (only if needed)
           ↓
        LLM API

The LLM is only called when a genuinely new question appears.

With this approach, the AI Cost Firewall does not only save the costs but also rockets the response time improving the users’ satisfaction (Customer Satisfaction Score, CSAT).

OpenAI compatible by design​

One of the most practical aspects of AI Cost Firewall is that you do not have to touch your application to integrate it. What you do is you simply switch the base URL to the firewall’s endpoint:

client = OpenAI(
    base_url="http://localhost:8080/v1"
)

From the application’s perspective, nothing changes. The same requests and responses flow through the system. However, now the firewall intelligently “decides” whether the model actually needs to be called and the money has to be spent.

This tool is compatible with:

• OpenAI models
• Azure OpenAI
• local OpenAI-compatible servers
• many hosted inference platforms

In other words, any system that already works with the OpenAI API can immediately benefit from cost reduction. And more than that, other models are going to be added soon by the project developers.

Observability built in​

One of the integrated features of the AI Cost Firewall is its built-in monitoring. It consists of Prometheus for scraping the metrics and integrated Grafana Dashboard. Both services are launched automatically by docker compose using preconfigured Prometheus YAML and a prebuilt Grafana dashboard JSON, so the monitoring stack is ready immediately without any manual configuration.

Prometheus metrics allow you to track:

• number of cache hits
• semantic matches
• forwarded requests
• estimated cost savings
• active requests

You can immediately visualize these metrics with the Grafana dashboard to see exactly how much the firewall is saving in real time (with a 5-second delay to be honest).

Why it works well​

AI Cost Firewall works because it targets a structural characteristic present in almost every LLM application:

• repeated user questions
• overlapping knowledge queries
• duplicated agent prompts

By caching responses and using semantic similarity search, the system converts repeated LLM calls into near-zero-cost lookups.

Why near-zero and not fully zero? Because semantic matching still requires generating embeddings for incoming queries. However, embedding costs are typically orders of magnitude lower than generating full LLM responses.

Another advantage of the firewall is its intentionally minimal architecture:

• Rust firewall gateway
• Redis for exact caching
• Qdrant for semantic caching
• Prometheus + Grafana for monitoring

This simplicity makes it easy to deploy, maintain, and scale.

When AI Cost Firewall is most effective​

The biggest savings with AI Firewall occur in systems where similar questions appear frequently. You will immediately benefit from the AI Cost Firewall integration if your system includes any or several of the following components:

• customer support chatbots
• internal company knowledge assistants
• documentation Q&A systems
• developer copilots
• AI help desks
• AI Agents performing any of the above tasks

In these environments, the same core questions appear repeatedly across many users. Even when questions are phrased differently, the semantic cache can reuse the same answer multiple times.

Advanced users may also appreciate the integrated TTL (Time-to-Live) feature which allows you to set up the duration of the response kept in Redis’s memory before replaced with a newly generated one. The same feature for Qdrant is currently under development and will be introduced soon.

Try it in 60 seconds​

If you want to see how AI Cost Firewall works, you can deploy the whole stack locally or on a small server in less than a minute.

Example:

git clone https://github.com/vcal-project/ai-firewall
cd ai-firewall
cp configs/ai-firewall.conf.example configs/ai-firewall.conf
nano configs/ai-firewall.conf # Replace the placeholders with your API keys
docker compose up -d

This launches:

• AI Cost Firewall
• Redis (exact cache)
• Qdrant (semantic cache)
• Prometheus (metrics scraping)
• Grafana (monitoring dashboard)

Once the containers start, simply point your OpenAI client to:

http://localhost:8080/v1

Within seconds, the gateway is ready to accept OpenAI-compatible requests. Similar to Nginx and other infrastructure gateways, you only need to add your API keys to the configuration file. From that point on, every request automatically passes through the cost-saving pipeline while previous responses are stored in Redis and Qdrant for future reuse.

Because the gateway itself is stateless, multiple firewall instances can be deployed behind a load balancer, allowing the system to scale horizontally with growing traffic.

                         ┌───────────────────────┐
                         │        Clients        │
                         │  Chatbots / Agents /  │
                         │  Internal AI Apps     │
                         └───────────┬───────────┘
                                     │
                                     ▼
                         ┌───────────────────────┐
                         │     Load Balancer     │
                         │   Nginx / HAProxy     │
                         └───────────┬───────────┘
                                     │
                ┌────────────────────┼────────────────────┐
                │                    │                    │
                ▼                    ▼                    ▼
     ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
     │ AI Cost Firewall │  │ AI Cost Firewall │  │ AI Cost Firewall │
     │    instance 1    │  │    instance 2    │  │    instance 3    │
     └─────────┬────────┘  └─────────┬────────┘  └─────────┬────────┘
               │                     │                     │
               └─────────────┬───────┴───────────┬─────────┘
                             │                   │
                             ▼                   ▼
                 ┌──────────────────┐   ┌──────────────────┐
                 │   Redis / Valkey │   │      Qdrant      │
                 │  Exact cache     │   │  Semantic cache  │
                 └─────────┬────────┘   └─────────┬────────┘
                           │                      │
                           └──────────┬───────────┘
                                      │
                                      ▼
                         ┌───────────────────────┐
                         │  Upstream LLM API     │
                         │ OpenAI / Azure / vLLM │
                         └───────────────────────┘


     ┌─────────────────────── Observability Stack ─────────────────┐
     │                                                             │
     │  AI Cost Firewall metrics ─────► Prometheus ─────► Grafana  │
     │                                                             │
     └─────────────────────────────────────────────────────────────┘

Architecture of a horizontally scalable AI Cost Firewall deployment.

Conclusion​

With the growing adoption of AI, it is sometimes painful to watch the steady increase of company expenses related to LLM tokens. Large Language Models are extremely powerful, but they can also become quite expensive when used at scale. It feels even more unfair when you realize that a significant portion of LLM traffic consists of repeated or semantically similar questions.

AI Cost Firewall addresses this inefficiency with a simple idea: do not send the same or similar question to the model again and again. Instead, reuse answers that were already generated for identical or semantically similar queries.

By combining exact caching with semantic similarity search, the firewall allows previously generated answers to be reused safely and efficiently. The result is lower token consumption, faster responses, and reduced infrastructure costs.

Because the gateway is OpenAI-compatible, integration requires only a small configuration change. No application refactoring is needed. If your system includes chatbots, knowledge assistants, developer copilots, or AI agents that answer recurring questions, AI Cost Firewall can reduce token usage by 30–75% immediately after deployment.

And since the entire stack runs with a single docker compose command, you can try it in minutes.

Sometimes the most effective optimization is not a new model, a larger GPU cluster, or a complex architecture.

Sometimes it is simply not paying twice for the same answer.

If you find this open-source project useful, a GitHub star will help the project grow.

⭐ https://github.com/vcal-project/ai-firewall

Originally published on Dev.to on January 17, 2026.
Read the Dev.to version

Why meaning moved from definitions to structure — and what that changed for modern AI​

When engineers talk about semantic search, embeddings, or LLMs that "understand" language, it often sounds like something fundamentally new. Yet the problems modern AI systems face — meaning, reference, ambiguity, and context — were already central questions in linguistics and philosophy more than a century ago.

This article traces how the concept of semantics evolved across disciplines: from linguistics and philosophy, through symbolic AI and statistical NLP, and finally into the neural architectures that power modern large language models, and why this history matters for how we design retrieval, memory, and language systems today. The journey reveals that today's AI systems are not a break from the past, but the convergence of long-standing ideas finally made computationally feasible.

Linguistic Origins: Meaning as a System, Not a Label​

Modern semantics begins not with computers, but with language itself. In the late 19th and early 20th centuries, linguists began to reject the naive idea that words simply "point" to things in the world. One of the most influential figures in this shift was Ferdinand de Saussure, who argued that language is a structured system of signs rather than a naming scheme.

Saussure proposed that each linguistic sign consists of two inseparable parts: the signifier (the sound or written form) and the signified (the concept evoked). Crucially, the relationship between the two is arbitrary. There is nothing inherently "dog-like" about the word dog. Its meaning arises because it occupies a position within a broader system of contrasts: dog is meaningful because it is not cat, not wolf, not table.

This was a radical idea at the time. Meaning, Saussure claimed, is relational. Words derive significance from how they differ from other words, not from direct correspondence with reality. This insight quietly laid the conceptual groundwork for everything from structural linguistics to modern vector-based representations.

Philosophy of Language: Meaning, Logic, and Composition​

While linguists focused on structure, philosophers sought precision. In particular, Gottlob Frege transformed semantics by embedding it into formal logic. Frege introduced a critical distinction between sense — the mode of presentation of an idea, and reference — the actual object being referred to.

This distinction explained how two expressions could refer to the same thing while conveying different information. "The morning star" and "the evening star" both refer to Venus, yet they are not interchangeable in all contexts. Meaning, therefore, could not be reduced to reference alone.

More importantly, Frege formalized the idea of compositionality: the meaning of a sentence is determined by the meanings of its parts and the rules used to combine them. This principle became foundational not only in philosophy, but later in programming languages, logic systems, and early AI models.

In retrospect, compositionality is what allowed meaning to be treated as something computable, at least in theory.

Early Artificial Intelligence: When Meaning Was Symbolic​

When I studied linguistics at university many years ago, everything up to this point was already part of the curriculum. Structural linguistics, philosophy of language, and formal semantics provided a solid theoretical foundation. What none of us could have anticipated at the time was how directly these ideas would later intersect with computer science in what would come to be called artificial intelligence.

When AI emerged as a field in the mid-20th century, it inherited philosophy's confidence in symbols and logic. Early systems assumed that meaning could be explicitly represented through formal structures: symbols, predicates, rules, and ontologies. To "understand" language was to transform symbols according to carefully designed rules.

For a while, this worked. Expert systems, knowledge graphs, and first-order logic engines achieved impressive results in narrowly defined domains such as medical diagnosis, chemical analysis, and configuration problems. Within carefully bounded worlds, symbolic semantics appeared tractable.

Natural language, however, quickly exposed the limits of this approach. Human language is ambiguous, context-dependent, and constantly evolving. Encoding all possible meanings and interpretations proved not merely difficult, but fundamentally unscalable. Symbolic systems were brittle: they failed not gradually, but catastrophically, when faced with inputs that deviated even slightly from their assumptions.

Semantics, it turned out, was far messier than logic had allowed, and far more resistant to being fully written down.

The Statistical Shift: Meaning Emerges from Usage​

A quiet revolution began when linguists and computer scientists started to look not at rules, but at usage patterns. The idea that meaning could be inferred from how words are used rather than how they are defined gained traction in the mid-20th century.

The core insight was simple but profound: words that appear in similar contexts tend to have similar meanings. Instead of encoding semantics explicitly, one could measure it statistically by analyzing large corpora of text.

This approach, known as distributional semantics, reframed meaning as something empirical rather than prescriptive. Words became vectors of co-occurrence statistics. Similarity was no longer binary or rule-based, but graded and approximate.

This was a decisive break from symbolic AI and a return, in spirit, to Saussure's relational view of meaning.

Word Embeddings: Geometry Becomes Semantics​

Distributional ideas matured dramatically with the introduction of neural word embeddings, particularly models like Word2Vec. Instead of relying on sparse frequency counts, these models learned dense, low-dimensional vector representations optimized to predict linguistic context.

What emerged surprised even their creators. Semantic relationships appeared as geometric regularities in vector space. Differences between vectors encoded analogies, hierarchies, and semantic proximity. Meaning became something you could measure with cosine similarity.

This was not symbolic understanding, but it was not random either. It was structure: learned rather than designed.

For the first time, machines exhibited behavior that looked like semantic intuition, despite having no explicit definitions or rules.

Contextual Semantics: Meaning Is Not Fixed​

Static embeddings had a fundamental limitation: each word had exactly one vector, regardless of context. But human language does not work that way. The meaning of a word shifts depending on surrounding words, speaker intent, situation, and even emotion.

Transformer-based models, particularly BERT, addressed this by making representations contextual. Instead of asking "What does this word mean?", the model learned to ask "What does this word mean here?"

Through attention mechanisms, transformers model relationships between tokens dynamically. Meaning is no longer stored in a single vector per word, but distributed across layers and activations that respond to context.

This marked a crucial step toward pragmatic semantics: language as it is actually used, not as it is abstractly defined.

Large Language Models: Semantics as Emergent Structure​

Large language models such as GPT do not contain explicit semantic representations in the traditional sense. They are trained to predict the next token in a sequence. And yet, at scale, they display behaviors that look strikingly semantic: summarization, reasoning, translation, abstraction.

The key idea is emergence. As models compress vast amounts of linguistic data, they internalize regularities about the world, language, and human communication. Semantics arises not as a module, but as a side effect of learning efficient representations.

These models do not "know" meaning in a philosophical sense. But they operate in a space where syntax, semantics, and pragmatics are inseparable, and where relational structure dominates.

When Meaning Becomes Operational​

For practitioners building semantic search systems, RAG pipelines, or LLM-adjacent infrastructure, this history is not academic background — it is an explanation of why certain designs consistently work while others fail. Exact matching breaks down because natural language rarely repeats itself verbatim. Embeddings succeed not because they are clever, but because they mirror how meaning behaves in practice: approximately, relationally, and with tolerance for variation.

Once this is understood, several architectural consequences follow naturally. Retrieval quality depends less on perfect recall and more on selecting representations that preserve semantic neighborhoods. Caching strategies become viable only when equivalence is defined by similarity rather than identity. Evaluation metrics must account for graded relevance instead of binary correctness. Even system boundaries shift: components no longer exchange "facts", but approximations of meaning that remain useful within context.

Semantic systems are effective precisely because they do not attempt to eliminate ambiguity. They absorb it. Whether you are designing a vector store, placing a semantic cache in front of an LLM, or building a long-term memory layer for conversational systems, you are implicitly making choices about how much approximation your system tolerates and where that tolerance is enforced.

Closing Thought: Semantics as Shared Infrastructure​

What began as a linguistic insight, that words gain meaning through their relations to other words, has quietly become an organizing principle for entire computational systems. Meaning no longer lives in dictionaries, rules, or symbols, but in patterns: in how expressions cluster, diverge, and reappear across vast landscapes of language. Semantics is no longer something a system contains; it is something a system moves through.

This shift took more than a century to unfold. It required philosophers to separate sense from reference, linguists to abandon naming theories, and engineers to accept approximation over certainty. Only when data became abundant and computation relatively cheap did this long trajectory converge into something operational. Semantics, once debated in lecture halls and footnotes, has become infrastructure — implicit, distributed, and shared.

That idea, radical when first proposed, has been waiting over a hundred years for enough data and compute to become practical.

And now, finally, it has.

Originally published on Medium.com on November 27, 2025.
Read the Medium.com version

Today edge computing is reshaping the way AI systems are deployed. Instead of sending every request to centralized cloud infrastructure, more computation is happening on devices closer to end-users. These “edge environments” include IoT gateways, on-premise servers, mobile devices, micro-VMs, serverless functions, and browser-based applications. The appeal is clear: moving computation closer to where data is generated reduces latency, minimizes bandwidth requirements and allows organizations to satisfy strict data-privacy rules.

At the same time, WebAssembly (WASM) has emerged as a portable, sandboxed runtime for executing code in highly constrained or security-sensitive environments. Originally designed for browsers, WASM now runs in cloud edge workers, serverless platforms, and isolated environments where traditional binaries cannot be executed. These runtimes often restrict access to system calls such as networking, threading, or the local filesystem. They operate under strict memory limits, sometimes as low as tens of megabytes, and they prioritize deterministic, predictable execution.

Altogether, while offering obvious advantages, running AI components at the edge introduces its own challenges, especially when applications rely on semantic search, embeddings, or large language models (LLM).

A major issue arises when AI applications repeatedly generate similar responses to similar prompts. In a cloud setting this inefficiency is tolerable, but at the edge it becomes costly. Edge nodes often have hard limits on CPU time and memory allocation, meaning that even small local language models may struggle to meet real-time latency budgets. A semantic cache — a system that stores answers together with an embedding vector and returns a cached answer when the incoming request is semantically similar — is a natural solution. However, building such a system for constrained environments is significantly more difficult than building one for the cloud.

The first challenge is memory. Classical vector databases and similarity search engines rely on complex indexing structures such as HNSW graphs, which are fast but memory-intensive. Standard configurations easily grow to hundreds of megabytes and often assume the availability of multi-threading, background maintenance processes, and dynamic memory growth. Edge workers and WASM isolates cannot accommodate this. In many cases, the runtime enforces strict caps on linear memory and disallows growing beyond a fixed boundary. This immediately rules out most existing semantic search libraries, even before considering cold-start overhead or storage.

The second constraint is the execution environment itself. WASM runtimes typically do not expose POSIX-like APIs (Portable Operating System Interface, a family of standards developed by IEEE that define consistent application programming interfaces). Features such as mmap, file descriptors, or native sockets are unavailable unless the host explicitly provides them through WASI (WebAssembly System Interface), and even then, support varies. This makes it almost impossible to run vector databases “as-is,” because they depend heavily on operating system functionality and persistent background services. In edge environments developers have only a few milliseconds to initialize modules, produce a response, and return control to the runtime. A semantic cache that takes hundreds of milliseconds to load an index simply cannot be deployed in these contexts.

Cold-start behavior is another architectural concern. Unlike long-running cloud servers, edge workers may be rapidly created and destroyed. A new isolate might handle only one or two requests before being recycled. For AI applications, this means that any semantic cache must load extremely quickly — ideally in a few milliseconds — and must not rely on heavy initialization or dynamic graph reconstruction. Snapshotting becomes essential: developers need the ability to store the cache state in a compact format that loads deterministically and quickly into memory.

Then, there is also the question of energy and cost efficiency. Edge nodes operate on limited power budgets, especially in IoT scenarios. Recomputing the same embedding or calling an external LLM repeatedly wastes both energy and bandwidth. Reducing redundant inference calls requires a semantic memory layer that can match incoming queries to existing knowledge without exceeding stringent resource constraints.

Privacy regulations add an additional layer of complexity. One of the motivations for moving AI workloads to the edge is to keep sensitive data local. But to do that effectively, the system must avoid unnecessarily sending repeated questions or logs to a central model. A semantic cache therefore becomes not just a performance optimization but a privacy mechanism: if the system can answer from its local memory, no data transmission to external LLMs is required. Unfortunately, building such a cache in environments with restricted storage, no access to background processes, and strict runtime quotas is a non-trivial task.

These are the conditions under which traditional semantic search infrastructure begins to struggle. Large vector databases simply assume too much: too much RAM, too much access to the operating system, too much startup time, and too much persistence. Even lightweight semantic caches designed for server applications often rely on threading, shared memory, file-based checkpointing, or dynamically growing allocations. Most embedding-based caches were never designed with WASM runtimes, edge workers, or IoT gateways in mind.

This is precisely the gap that newer designs aim to address. Solutions like VCAL approach semantic caching not as a distributed system or standalone service but as a small in-process library that can run with minimal memory and without heavy OS dependencies. Instead of behaving like a database, it behaves more like a CPU-level cache for AI reasoning, storing question-answer pairs and their embeddings in an optimized structure that can fit within the constraints of edge and WASM environments. By avoiding reliance on network calls, background threads, or large indices, such systems become suitable for serverless workers, browser WASM modules, or embedded devices with limited RAM.

In this sense, the semantic cache becomes a missing piece of infrastructure for edge AI. As more organizations push inference closer to the user, the need for a lightweight, deterministic, low-memory semantic lookup system grows. The limitations of edge platforms — from strict memory caps to rapid cold starts — make this a difficult problem, and the lack of suitable solutions has slowed the adoption of AI features outside centralized cloud environments. As WASM matures and edge utilities evolve, semantic caching may become a standard part of the AI pipeline, enabling faster, cheaper, and more privacy-preserving deployments across a wide range of devices.

Originally published on Medium.com on November 6, 2025.
Read the Medium.com version

When “intelligence” wastes cycles​

Most teams building LLM-powered products eventually realize that a large portion of their API costs come not from new insights, but from repeated questions.

A support bot, an internal assistant, or an analytics copilot, all encounter thousands of near-identical queries:

“How do I pass the API key to the local model gateway?”
“Why is the dev database connection timing out?”
“How can I refresh the cache without restarting the service?”

Each of those prompts gets re-tokenized, re-embedded, and re-sent to an LLM even when the model has already answered an equivalent question a minute earlier.

What do we have as a result? Burned tokens, wasted latency, and duplicated reasoning.

Vector databases solved storage, not reuse​

The industry's first instinct was to throw vector databases at the problem. They excel at persistent embeddings and semantic retrieval, but they were never built for reuse. What they lack are TTL policies, eviction strategies, and atomic snapshotting of in-flight state. In other words, they store knowledge, not memory.

Traditional vector databases follow a key:value paradigm: they persist embeddings indefinitely so they can be queried later, much like records in a datastore. A semantic cache, by contrast, treats embeddings as dynamic memory — governed by similarity, expiration, and adaptive retention. Its goal is not to archive information, but to avoid redundant reasoning across millions of semantically similar requests.

With a semantic cache such as VCAL, cached answers can stay valid for days or weeks, depending on data volatility and TTL settings. This moves caching from short-term repetition avoidance to long-horizon semantic reuse where reasoning itself becomes a reusable resource rather than a recurring cost.

In essence, VCAL bridges the gap between data retrieval and cognitive efficiency, turning past computation into future acceleration.

From data stores to memory layers​

In my previous Dev.to article, I explained how we built VCAL, a Rust-based semantic cache that sits between your app and the LLM. Instead of persisting every vector, it memorizes embeddings for a short time, indexed by semantic similarity and metadata.

When a new query arrives, VCAL compares it to cached vectors. If it is close enough — a cache hit — the LLM call is skipped, and the stored answer is returned in milliseconds. Otherwise, the request proceeds normally, and the response is stored for future matches.

The design combines concepts from vector search and traditional caching systems, enhanced with features for resilience and monitoring:

• HNSW index for ultra-fast approximate similarity search.
• TTL and LRU eviction for automatic cache turnover.
• Snapshotting for persistence between restarts.
• Prometheus metrics for observability.

All of it runs on-prem, next to your model or gateway with no remote dependencies.

Why local caching changes the economics​

Unlike vector databases, a local semantic cache has one simple purpose: avoid redundant reasoning. Each avoided LLM call translates directly into saved tokens, lower API bills, and shorter response times.

In real deployments we’ve seen:

• 30–60 % reduction in LLM calls
• Millisecond-level latency on repeated queries enabling near-real-time responsiveness
• Predictable resource usage: no external round-trips, no cloud egress costs, and no multi-tenant contention

At scale, the more your users interact, the greater the savings become. Instead of paying per token for every repetition, you amortize prior reasoning across sessions and teams.

And because VCAL runs inside your private environment, all caching and embeddings stay under your control ensuring data privacy, compliance, and deterministic performance even in regulated industries.

A new layer in the AI stack​

If you visualize the modern LLM stack, the simplified design looks like this:

User → Application → LLM Gateway → Model

or, if a RAG (Retrieval-Augmented Generation) framework is involved:

User → Application → Retriever → Vector DB → (context) → LLM Gateway → Model

Adding a semantic cache such as VCAL introduces this new dimension:

User → Application → Retriever → Vector DB → (context) → Semantic Cache → LLM Gateway → Model

Here, the cache checks whether a semantically equivalent query was already answered. If found, the response is returned instantly — skipping tokenization, embedding, and inference altogether. If not, the request continues as usual, and the new answer is stored for future reuse.

Vector databases still matter but they belong to the knowledge layer, not the inference path. What has been missing so far is a memory layer that prevents repeated reasoning altogether. Semantic caching fills the missing “memory” slot in between. It is not a replacement for RAG, it is a complement. While RAG injects context, caching avoids duplication.

Engineering for low latency​

Achieving millisecond response times in semantic caching requires more than just a fast similarity search algorithm. It’s the result of careful coordination between data structures, memory layout, and concurrency control.

The cache can be implemented efficiently in systems programming languages such as Rust, using an HNSW-based index for approximate nearest-neighbor search. HNSW provides logarithmic-scale query complexity while maintaining accuracy for large collections of embeddings, making it suitable for workloads that reach millions of cached entries.

Low latency also depends on predictable memory management and lock-free or fine-grained synchronization between threads. Instead of allocating and freeing vectors dynamically, embeddings are often stored in preallocated arenas or memory-mapped regions to minimize fragmentation and system calls. Parallel workers can update the index or evaluate similarity thresholds concurrently, so that retrieval scales with the number of available cores.

In practice, a semantic cache can be deployed as a lightweight service beside an inference gateway, communicating over local HTTP or gRPC. It can also be embedded directly into an application process when minimal overhead is required, for example, within an agent runtime or API handler.

The bigger picture​

Caching has always been an invisible driver of performance — from CPU registers that reuse instructions, to CDNs that reuse content, to databases that reuse queries.

Each generation of systems extends the notion of what can be reused. As language models enter production, we are witnessing a shift toward semantic reuse: reusing meaning rather than data. This enables systems to recall previous reasoning instead of repeating it — a step toward more efficient and sustainable AI infrastructure.

In this new layer of the AI stack, semantic caching becomes a form of reasoning memory: it stores the results of understanding, not just storage operations. Instead of recomputing the same insight across thousands of near-identical prompts, we can recall it instantly — with full control over latency, privacy, and cost.

Further reading​

For readers interested in implementation details and open-source examples:

• How I Created a Semantic Cache Library for AI (Dev.to)
• vcal-project.com for general information, documentation, and architectural notes

Thank you for reading! Semantic caching is still an emerging concept — every real-world use case helps shape how we think about efficient reasoning. Share yours in the comments if you’d like to join the conversation.

Originally published on Dev.to on October 27, 2025.
Read the Dev.to version

Have you ever wondered why LLM apps get slower and more expensive as they scale, even though 80% of user questions sound pretty similar? That’s exactly what got me thinking recently: why are we constantly asking the model the same thing?

That question led me down the rabbit hole of semantic caching, and eventually to building VCAL (Vector Cache-as-a-Library), an open-source project that helps AI apps remember what they’ve already answered.

The “Eureka!” Moment​

It started while optimizing an internal support chatbot that ran on top of a local LLM. Logs showed hundreds of near-identical queries:

“How do I request access to the analytics dashboard?”
“Who approves dashboard access for my team?”
“My access to analytics was revoked — how do I get it back?”

Each one triggered a full LLM inference: embedding the query, generating a new answer, and consuming hundreds of tokens even though all three questions meant the same thing.

So I decided to create a simple library that would embed each question, compare it to what was submitted earlier, and if it’s similar enough, return the stored answer instead of generating an LLM response, all this before asking the model.

I wrote a prototype in Rust — for performance and reliability — and designed it as a small vcal-core open-source library that any app could embed.

The first version of VCAL could:

• Store and search vector embeddings in RAM using HNSW graph indexing
• Handle TTL and LRU evictions automatically
• Save snapshots to disk so it could restart fast

Later came VCAL Server, a drop-in HTTP API version for teams that wanted to cache answers across multiple services while deploying it on-prem or in a cloud.

What It Feels Like to Use​

Unlike a full vector database, VCAL isn’t designed for long-term storage or analytics. I didn’t want to build another vector database.
VCAL is intentionally lightweight. It is a fast, in-memory semantic cache optimized for repeated LLM queries.

Integrating VCAL takes minutes.
Instead of calling the model directly, you send your query to VCAL first.
If a similar question has been asked before — and the similarity threshold can be tuned — VCAL returns the answer from its cache in milliseconds. If it’s a new question, VCAL asks the LLM, stores the result, and returns it.
Next time, if a semantically similar question comes in, VCAL answers instantly.

It’s like adding a memory layer between your app and the model — lightweight, explainable, and under your full control.

Lessons Learned​

• LLMs love redundancy. Once you start caching semantically, you realize how often people repeat the same question with different words.
• Caching semantics ≠ caching text. Cosine similarity and vector distances matter more than exact matches.
• Performance scales beautifully. A well-tuned cache can handle thousands of lookups per second, even on modest hardware.
• It scales big. A single VCAL Server instance can comfortably store and serve up to 10 million cached answers in memory, depending on embedding dimensions and hardware.

What’s Next​

We’re now working on a licensing server, enterprise snapshot formats, and RAG-style extensions, so teams can use VCAL not just for Q&A caching, but as the foundation for private semantic memory.

If you’re building AI agents, support desks, or knowledge assistants, you’ll likely benefit from giving your system a brain that remembers.

You can explore more at vcal-project.com - try the free 30-day Trial Evaluation of VCAL Server or jump into the open-source vcal-core version on GitHub.

Thanks for reading!
If this resonates with you, please drop a comment. I’d love to hear how you’re approaching caching and optimization for AI apps.