AI Agents Aren’t Production Ready - and Access Control Might Be the Reason

-Daniel Bass

Enhancing application performance through the use of AI agents is becoming a very common practice - whether due to industry-wide hype or actual helpful use cases. Regardless, it helps automate workflows and enhance user experiences across various industries.

These AI agents, however, often require access to highly sensitive data, introducing significant security risks. Without standardized access control solutions, development teams often build security measures from scratch—leading to cases where AI assistants expose sensitive patient information and several AI agents being compromised through sophisticated prompt injection attacks.

Protecting AI agents in this context is becoming just as important as the capabilities of AI agents themselves - if not more so.

To address this problem, we at Permit.io have been working for a while on a structured security approach I wish to introduce to you today:

The Four-Perimeter Framework.

This method provides a comprehensive strategy for protecting AI agents by implementing fine-grained access control, ensuring compliance, and mitigating potential attack vectors. Before I go into the solution, let’s talk more about the problem space.

The Problem: AI Security Risks in Sensitive Domains

Traditional application security measures and access control mechanisms fall short when it comes to handling AI agents. While conventional access control systems like role-based access control (RBAC) excel at managing static permissions, they struggle with the dynamic nature of AI responses. Unlike traditional applications where inputs map to predictable outputs and access patterns are well-defined, AI generates unique responses for each query. This means the access control and input validation techniques we’re used to can’t anticipate all possible outputs.

The way AI systems access data also doesn’t fit security standards. To provide accurate and helpful responses, AI agents need broad access to various data sources and knowledge bases. Classic RBAC proves too rigid for this need - you can’t simply assign static permissions when the data requirements change based on context. This calls for a more sophisticated, attribute-based approach that can make dynamic access decisions.

Another concern is the new attack vectors that AI systems introduce. Conventional security measures weren’t designed to handle prompt injection attacks, model extraction attempts, or data poisoning attacks. These target vulnerabilities unique to AI systems, requiring entirely new protection methods.

The solution comes in the form of a comprehensive security framework built around four essential perimeters -

The Solution: The Four-Perimeter Framework

As AI applications differ from standard software in how they process inputs, retrieve data, execute actions, and generate outputs, they also introduce unique security risks. To address these challenges, we need a structured security approach that applies access control at every stage of AI interaction. And that’s exactly what I want to showcase here:

The Four-Perimeter Framework is designed to enforce identity-aware, fine-grained authorization throughout the AI lifecycle. It introduces security boundaries that govern what data AI models can access, what operations they can perform, and how their responses are validated.

The framework consists of four parts:

1. Prompt Filtering – Ensuring only validated, safe inputs reach AI models.
2. RAG Data Protection – Controlling AI access to external knowledge sources.
3. Secure External Access – Defining AI agent permissions when interacting with external tools.
4. Response Enforcement – Applying compliance checks and filtering AI-generated outputs.

By applying fine-grained access control (FGA) principles across these perimeters, AI agents remain secure, auditable, and compliant—without sacrificing their flexibility and functionality.

Where Does FGA Come In?

As mentioned earlier, traditional access control methods like RBAC fail to handle AI’s dynamic, context-dependent behavior. This is where fine-grained authorization (FGA) comes into play, specifically through attribute-based access control (ABAC) and relationship-based access control (ReBAC):

• ABAC for Prompt Filtering AI models process unstructured inputs, making it difficult to apply traditional security rules. ABAC solves this by extracting structured attributes from AI prompts and using them in policy-based access decisions.

• ReBAC for RAG Data Protection Retrieval-augmented generation (RAG) allows AI models to pull information from vector databases. ReBAC provides a natural way to enforce security in this setup by defining relationships between users, data sources, and AI agents. Instead of assigning static permissions, ReBAC dynamically grants or denies access based on how data objects relate to one another—enabling context-aware retrieval permissions.

By combining ABAC and ReBAC, AI applications gain flexible, policy-driven access control mechanisms that adapt to different AI workflows without requiring manual intervention.

What Does a Step-by-Step Implementation Look Like?

Let's walk through a practical implementation to see the Four-Perimeter Framework in action. This example secures an AI agent by validating prompts, enforcing data protection, restricting external access, and moderating responses—leveraging Permit.io AI Access Control integrations at each step.

1. Implement Prompt Filtering

The first security perimeter focuses on validating and sanitizing AI inputs before they reach the model. We can enforce structured input handling and enforce access policies on prompts before they reach the AI model:

• Authorization checks are embedded into the prompt validation pipeline, preventing unauthorized data from influencing AI outputs.
• Role, attribute, and relationship-based access control are enforced, ensuring only approved parameters pass to the AI.
• Any policy violations are logged and blocked in real-time, preserving an auditable trail of access decisions.

Here are the three methods to filter prompts - from easiest to most advanced:

1. Token validation - An efficient method for simple checks like prompt length or the general allowance of running inputs in the system. Permission check functions in this method examine the structured attributes of textual prompts.
2. Pattern Matching - in this method, the permission check examines patterns in the prompt text. Checking, for example, if the pattern matches an SQL query or code example.
3. AI Classification - the most advanced method in this list, AI classification uses AI to analyze the prompt using a dedicated system prompt, classifying it into a structured input that can be precisely analyzed in the permissions check.

Prompt Filtering with PydanticAI

To best demonstrate the implementation of efficient prompt filtering, here’s an example of using PydanticAI tools to filter user inputs to AI agents. PydanticAI is an agent framework that takes advantage of Python’s famous Pydantic library and its static typing capabilities and turns it into a structured AI framework that handles unstructured data.

The following GitHub repository contains a full implementation of the framework for PydanticAI: github.com/permitio/permit-pydanticai

@financial_agent.tool
async def validate_financial_query(
    ctx: RunContext[PermitDeps],
    query: FinancialQuery,
) -> bool:
    """Key checks:
    - User has explicitly opted in to AI financial advice
    - Consent is properly recorded and verified
    - Classifies if the prompt is requesting advice

    Args:
        ctx: Context containing Permit client and user ID
        query: The financial query to validate

    Returns:
        bool: True if user has consented to AI advice, False otherwise
    """
    try:
        # Classify if the prompt is requesting advice
        is_seeking_advice = classify_prompt_for_advice(query.question)

        permitted = await ctx.deps.permit.check(
            # The user object with their attributes
            {
                "key": ctx.deps.user_id,
            },
            # The action being performed
            "receive",
            # The resource being accessed
            {
                "type": "financial_advice",
                "attributes": {"is_ai_generated": is_seeking_advice},
            },
        )

        if not permitted:
            if is_seeking_advice:
                return "User has not opted in to receive AI-generated financial advice"
            else:
                return "User does not have permission to access this information"

        return True

    except PermitApiError as e:
        raise SecurityError(f"Permission check failed: {str(e)}")

2. Enforce Data Protection –

Next, we secure retrieval-augmented generation (RAG) queries by ensuring AI models can only access authorized knowledge sources. This can be achieved through fine-grained filtering of the AI agent data on behalf of the user within the AI workflow:

• The fine-grained policy uses advanced data filtering models, such as relationship-based access control, to allow advanced query filtering to the policy engine.
• The vector and graph sources connected to the AI agents have metadata that can help the engine manipulate the unstructured query, filtering it to include only allowed data.
• Every time the agent fetches data from the RAG knowledge base, it filters the results per the user's permission.

When the agent application performs RAG filtering, it uses one of the following methods:

• FilterObjects—In this method, the graph application fetches all the relevant data from the RAG and then filters it per-user permissions. This function's advantage is that it keeps the policy engine stateless and unaware of RAG data.
• GetUserPermissions - In this method, the agent provides the RAG with an unstructured query. The RAG then calls the GetUserPermissions function, appending a filter query to the unstructured RAG query. This allows the query to be filtered to only the resources the user can access.

RAG Data Protection with Langchain

Langchain, the famous AI application framework, is known (also) for its retriever components that provide great support in custom retrievers that can be assembled with any kind of data source, making the implementation of secure RAG easy. For example, with the SelfQueryRetriever, you can wrap the usual RAG with a GetUserPermissions function that will append the filtered data to the query. Here’s an example of the PermitSelfQueryRetriever that gets the user, action, and the type of the RAG resources to filter the results for the LLM:

# 1. Build a small in-memory vector store
  embeddings = OpenAIEmbeddings()
  vectorstore = FAISS.from_documents(docs, embedding=embeddings)

# 2. Initialize the PermitSelfQueryRetriever
retriever = PermitSelfQueryRetriever(
    api_key=os.getenv("PERMIT_API_KEY", ""),
    pdp_url=os.getenv("PERMIT_PDP_URL"),
    user=USER,
    resource_type=RESOURCE_TYPE,
    action=ACTION,
    llm=embeddings,
    vectorstore=vectorstore,
    enable_limit=False,
)

The following repository contains the full implementation of the framework for Langchain: https://github.com/permitio/langchain-permit

3. Secure External Access –

AI agents often interact with external APIs, services, and automation tools. Without proper access control, they risk executing unauthorized actions. Thus, we must ensure AI-driven operations are restricted based on machine identity policies. Here’s how this can be achieved:

• Using models like MCP, which defines a server-client model where AI agents interact with external services, is a great step in enabling security for AI external access (e.g., databases, APIs, payment systems).
• The AI action infrastructure can use a service like Permit.io to perform authorization checks at the action level, checking who’s making a request and what action they want to perform.
• Developers are able to assign machine identities to AI agents, limiting their capabilities to pre-approved functions only.

While GPTs and natural language interfaces aren’t new, letting them perform actions on behalf of users is a fairly new challenge many engineers struggle with. We can notice three distinct levels of access control important to ensure here:

1. Direct access security - the simplest level. A user asks an AI agent to perform an operation, such as an HTTP call. The AI agent knows to check if the operation is allowed, considering the current context and user.
2. Agent-to-agent communication - in this next level, we require a cascading flow of identities and permissions that allow agents to perform actions themselves based on the principle of least privilege. At this level, we would want to use the relationship between identities - human and non-human - to cascade different levels of permissions.
3. Access request flows - the most exciting level of external access is when the AI agent understands the need to request access directly from a human user when it wants to perform an operation. The agent knows which user to ask for permission, and the operation will only be performed once this permission is granted.

MCP, Human-in-the-Loop, and External Access

Model Context Protocol is a new protocol introduced by Anthropic, which solves the problem of letting AI agents perform proactive actions and serves as the greatest enabler for securing external access.

In the following example, we show an MCP server that knows how to manage access request flows by bringing humans into the loop and uses Permit.io’s APIs to ask for permissions.

You can see the full code example for the MCP access request server here: https://github.com/permitio/permit-mcp

@mcp.tool()
async def request_access(username: str, resource: str, resource_name: str) -> dict:
  """
  Initiate access request call to a specific resource in the system

  Args:
    username: The username of the person requesting access
    resource: The resource type the user is request access for
    resource_name: The name of the restaurant to request access for

  """
  
  login = await permit.elements.login_as({ "userId": slugify(username), "tenant": "default"})
  print(login)
  
  url = f"https://api.permit.io/v2/facts/{PROJECT_ID}/{ENV_ID}/access_requests/{ELEMENTS_CONFIG_ID}/user/{slugify(username)}/tenant/default"
  payload = {
      "access_request_details": {
          "tenant": "default",
          "resource": resource,
          "resource_instance": resource_name['id'],
          "role": "viewer",
      },
      "reason": f"User {username} requests role 'viewer' for {resource_name}"
  }
  headers = {
      "authorization": "Bearer YOUR_API_SECRET_KEY",
      "Content-Type": "application/json",
  }
  async with httpx.AsyncClient() as client:
      await client.post(url, json=payload, headers=headers)
      return "Your request has been sent. Please check back later."

4. Validate AI Responses –

The final perimeter enforces content moderation and compliance on AI-generated responses. This can be done by creating a workflow that applies policy-based filtering before delivering AI outputs:

• A permission check can be added to the agent application, enabling policy enforcement at each step of the chain before the user actually gets a response.
• Combined data validation and authorization workflows ensure that only validated and permitted data can move forward.
• The response that returns to the users can be edited based on predefined limitations, thus allowing for data masking or notifying the user of use limitations.

Response Filtering with Langflow

In the following example, we use Langflow, a visual no-code AI applications editor, to create a permissions check component that’s positioned before any chat response to the user. Using LangFlow's friendly flow modeling capabilities, you can easily append another component that masks unwanted details in returned responses. This repository contains all the access control components required for secured Langflow applications.

Conclusion

AI systems are rapidly integrating into applications across industries, but their security frameworks remain underdeveloped compared to traditional software. Without proper identity-aware access control, AI agents risk exposing sensitive data, executing unauthorized operations, and generating responses that fall outside compliance guidelines.

The Four-Perimeter Framework offers a structured way to secure AI workflows at every stage—from validating prompts and protecting retrieval-augmented generation (RAG) data to controlling AI-driven external actions and enforcing response compliance. By leveraging fine-grained authorization (FGA) with ABAC and ReBAC, developers can ensure AI agents remain both functional and secure, adapting dynamically to real-world security scenarios.

Instead of building access control from scratch, Permit.io’s AI Access Control integrations with PydanticAI, LangChain, MCP, and LangFlow allow teams to embed security directly into their AI applications—without disrupting development workflows. Try it yourself here.

That said, this field is still relatively new, and best practices for securing AI agents are still evolving. The Four-Perimeter Framework is an experimental approach, and while it provides a strong foundation, I recognize that real-world deployments always require further refinement and adaptation.

I’d love to hear your feedback, thoughts, and ideas on how to improve this framework to make it even better for production-ready AI systems - comment here, or hit me up in our Slack community.