Integrating LLMs with Sensitive Data in Production: OpenAI API vs Azure OpenAI?

API
1

Hi everyone,

I’m currently working on a web application hosted on Azure that handles sensitive user data. Users connect through this platform, and we’ve been asked to integrate an LLM to allow them to query certain information in a concise way.

I have a couple of concerns regarding architecture and data security:

  • Would it be appropriate to integrate directly with the OpenAI API using an API key from my OpenAI account? I understand that data is transmitted over TLS, but I’m unsure whether this is sufficient for handling sensitive data in a production environment.

  • Or would it be better to deploy a model through Azure (e.g., Azure OpenAI or potentially Anthropic via Azure), keeping all data processing within our cloud environment?

I’m trying to understand the trade-offs between these two approaches, especially in terms of security, compliance, and best practices for handling sensitive information.

What would you recommend in this scenario?

Thanks in advance!

3

Hey Felix,

I don’t know Nonym myself, but when you say this “ensures” that no sensitive PII or secrets leave your network, I wonder how it achieves that in practice.

It seems likely to me that it cannot be 100% determinate — just as the LLM itself isn’t.

If it’s rule-based, it will miss edge cases.
If it’s model-based, it’s inherently probabilistic.

So it feels more like risk reduction than a guarantee.

I’d be interested to understand where that line is drawn in reality.

This may also be of interest in the broader scope :confused:

https://arxiv.org/pdf/2602.16800

Zero Data Retention request path is too unclear and frustrating for small EU companies
6

Thank sharing https://arxiv.org/pdf/2602.16800 I’ll go through it. Looks very interesting