Introducing Azure AI Proxy

For some reason, Azure diagnostic settings currently don’t provide any insight into prompts and their responses. Microsoft suggests setting up APIM as a proxy, but that leads to vendor lock-in and requires configuring Event Hubs if you want to connect everything to a monitoring solution like ElasticSearch. To avoid all that complexity and stay cloud-agnostic, We’ve written a simple transparent proxy that captures these details. You can run it as a sidecar next any data shipper or as a standalone container. This solution is completely open-source, and welcomes any contributions.

Quick Start

This software is designed to run as a container, needing just a handful of environment variables for setup. It’s simple to deploy alongside any data shipper, or run it as a standalone service

Configuration

Azure AI Proxy can be configured using environment variables:

Environment Variable	Description	Default Value
AZURE_OPENAI_ENDPOINT	URL of the Azure OpenAI service endpoint	your-deployment.openai.azure.com/
LISTEN_ADDR	Address and port for the proxy to listen on	localhost:8080
LOG_FILE_PATH	File path for request/response logs	openai_proxy.json
PROXY_API_KEY	API key for proxy authentication (optional)	(none)

Extra authentication

When PROXY_API_KEY is set for extra hardening, the proxy requires clients to include the key in the X-API-Key header, effectively requiring 2 API keys.

Run examples

run the binary directly, it can be found on GitHub Releases. A minimal viable configuration is to set the AZURE_OPENAI_ENDPOINT and PROXY_API_KEY environment variables. The proxy will listen on localhost:8080 by default, but you can change this with the LISTEN_ADDR variable.

bash

export AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
export PROXY_API_KEY=your-secret-key

./azure-ai-proxy

powershell

set AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
set PROXY_API_KEY=your-secret-key

azure-ai-proxy.exe

Deployment examples

Below are some examples of how to deploy the Azure AI Proxy in different environments. You can choose the one that best fits your infrastructure.

kubernetes

# This configuration includes a Filebeat sidecar for log shipping, which is optional and can be replaced with your preferred logging solution.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: filebeat-azure-ai-proxy
spec:
  template:
    spec:
      containers:

      - name:  azure-ai-proxy
        image: edgeforge/azure-ai-proxy
        env:
        - name: AZURE_OPENAI_ENDPOINT
          value: "https://openai-prd.openai.azure.com/"
        - name: LOG_FILE_PATH
          value: "/var/log/azure-ai-proxy/log.json"
        volumeMounts:
          - name: log-volume
            mountPath: /var/log/azure-ai-proxy

      - name: filebeat
        image: elastic/filebeat
        args: [
          "-c", "/usr/share/filebeat/filebeat.yml",
          "-e",
        ]
        volumeMounts:
          - name: log-volume
            mountPath: /var/log/azure-ai-proxy
          - name: config
            mountPath: /usr/share/filebeat/filebeat.yml
            readOnly: true
            subPath: filebeat.yml
            
      volumes:
        - name: log-volume
          emptyDir: {}
        - name: config
          configMap:
            defaultMode: 0644
            name: filebeat-config
---
# You'll need to define a `filebeat-config` ConfigMap for this setup. Below is a minimal example, you'll need to add your own authentication and other required settings.
filebeat.inputs:
- type: filestream
  id: azure-ai-proxy
  enabled: true
  paths:
    - /var/log/azure-ai-proxy/log.json
  parsers:
    - ndjson:
        keys_under_root: true 
        overwrite_keys: true
        add_error_key: true
        timestamp_field: Timestamp
        timestamp_format: RFC3339Nano
  scan_frequency: 5s

docker-compose

services:
  azure-ai-proxy:
    image: edgeforge/azure-ai-proxy:latest
    container_name: azure-ai-proxy
    ports:
      - "8080:8080"
    volumes:
      - ai_proxy_data:/openai_proxy.json
    restart: unless-stopped

volumes:
  ai_proxy_data:

Usage Examples

Once the proxy is running, you can test it with a simple API call:

bash

export PROXY_API_KEY="yourapikey"
export AZUREAI_API_KEY="yourapikey"
curl -X POST \
  "http://localhost:8080/openai/deployments/gpt-4o/chat/completions?api-version=2025-01-01-preview" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $PROXY_API_KEY" \
  -H "api-key: $AZUREAI_API_KEY" \
  -d '{"messages":[{"role":"user","content":"Say hello"}],"max_tokens":1000}'

powerShell

$env:PROXY_API_KEY="yourapikey"
$env:AZUREAI_API_KEY="yourapikey"
curl -X POST `
  "http://localhost:8080/openai/deployments/gpt-4o/chat/completions?api-version=2025-01-01-preview" `
  -H "Content-Type: application/json" `
  -H "X-API-Key: $($env:PROXY_API_KEY)" `
  -H "api-key: $($env:AZUREAI_API_KEY)" `
  -d '{"messages":[{"role":"user","content":"Say hello"}],"max_tokens":1000}'

Architecture

A detailed architecture overview of the Azure AI Proxy is available here. It explains how the proxy works, its components, and how it integrates with Azure OpenAI services.

Improvements?

If you are interested in this solution but need additional features, feel free to contact us directly or open an issue on GitHub.