AI-driven pentest: what we learned

Retyc is built to exchange sensitive files, and our promise fits in one sentence: your files are encrypted on your device, and nobody but your recipients can read them, not even us. A promise like that has to be tested regularly. Until now we did it with code reviews, automated tests and AI audits, but always limited to reading code. Anthropic's Cyber Verification Portal let us go further: a real pentest of Retyc, run by Claude. Here is how it went.

Anthropic's Cyber Verification Portal

Asking an AI to look for vulnerabilities raises an obvious question: what stops someone from turning it against other people's systems? Anthropic governs this kind of use with the Cyber Verification Portal. You have to be admitted before you can use Claude for offensive security work on your own systems.

We were granted access, and we set ourselves a strict framework. The pentest covered our development environment only, * never production, never our customers' data*. It relied on dedicated test accounts spread across several organizations and several plans, to check the separation between customers. And every finding had to come with a written report: how to reproduce it, what the real impact was, what the fix should be.

The development environment carries none of the protections that surround production: no CrowdSec, none of our hardened infrastructure configuration. That is deliberate. We wanted to see the flaws of the application itself, without a perimeter defence hiding them. The findings below are therefore harsher than what an attacker would meet facing our production.

A bumpy start

Not everything worked on the first try. Anthropic's documentation contained translation errors. Our first tests with Opus 5.5 actually ran on Opus 4.8. We only understood this from the warnings the AI itself raised.

The fix was to move to Opus 5 (5, not 5.5, mind the difference!), which raised no warning. Everything that follows comes from the audit run with that model.

As of writing, the Cyber Verification Portal does not yet allow the Fable or Opus 5.5 models.

How the pentest went

The pentest ran in several passes, including a complete restart that reused no conclusion from the previous one, and a verification round after each wave of fixes. Claude had access to the source code, to get a complete view of the application.

One rule above all: Claude was not allowed to simply read the code and infer vulnerabilities from it.

On the backend, Claude listed every endpoint exposed by our API and read the whole codebase (several tens of thousands of lines). It then checked its hypotheses under real conditions, with 4 accounts spread across 3 organizations, anonymous calls and API keys. Every finding it kept was reproduced: none was merely deduced from reading the code.

On the web application, Claude drove a headless Chrome. It logged in, unlocked the encryption key and actually sent a file containing a known marker. It captured every request leaving for the server, to see what really leaves the browser.

Among other things:

• fuzzing
• HTTP header injection and malformed requests
• hunting for XSS and SQL injection
• hunting for concurrency issues and race conditions
• checking the separation between organizations and between roles
• checking the security of the integration API (key scopes, IP address restriction)
• attempting privilege escalation inside an organization

What held

The most important part first: end-to-end encryption held. During the upload, no request contained the file content, its name or a private key. File names and types are always encrypted. Unlocking your key triggers a single request, and your passphrase never leaves your browser.

The audit also confirmed that the separation between organizations holds on the data: a user can neither read nor modify the transfers, datarooms or settings of another organization. Inside an organization, a member cannot grant themselves more rights than they have. For the integration API, key permissions and IP address restriction are properly enforced. Finally, sign-in follows the good practices of the OpenID Connect protocol.

No XSS vulnerability was found, and no SQL injection was possible.

What was found

Most findings fall under what is called hardening: extra protections that fix no exploitable vulnerability, but that limit the damage of a future problem or a configuration mistake.

Among the things we fixed or improved:

• the browser security policy (CSP), already very strict on scripts: only one exception, needed by the component that displays images, was broader than necessary. It now allows only the precise code that component needs
• a server that refuses to start when a configuration secret has been forgotten, instead of running with a default value
• cleaner error messages when a sign-in token is incomplete
• explicit locking of a file after upload in a dataroom

Two findings stand out, though, and we would rather talk about them openly.

In a dataroom, a user with the "contributor" role is not allowed to delete files, and the API did refuse. They could still reach the same result through a detour: move a folder into a dataroom they own themselves, then delete it there. The deletion took with it the files other members had put in that folder, including those of the owner of the original dataroom, and permanently. A check on the ownership of the destination folder was missing.

On email sign-up, used in particular to send files through a deposit box, the verification code could be guessed by trying a large number of combinations: the code was too short and nothing capped the number of attempts. The proof of work required on each attempt was not enough, because the same token could be replayed. Our reverse proxy rate limit narrows the window considerably in production, but we do not want that control to rest on it: the number of attempts is now capped, and a proof-of-work token is good for one use only.

None of these points allowed anyone to read a file: the content stayed encrypted end to end in every case. We fixed them first.

The false positives

Default values

Claude also reported points that were not real security problems. It flagged, for instance, that some configuration variables of our API (salts, secrets...) had a default value. Those values only served in development: in production they are always overridden.

We took the opportunity to set rules in our configuration anyway: a secret value never has a default, and the server refuses to start when one is missing, even in development or in CI. It must also meet a minimum length.

These rules have applied in production since the Retyc beta. Writing them explicitly into our development configuration lets us catch an omission before it reaches production, or the on-premises instances of our customers.

Debug mode

Claude also noted that some API routes, reserved for our developers, were reachable in debug mode. That mode cannot be enabled in production.

The Keycloak configuration in development

Locally, our development infrastructure runs on Docker Compose. At initialization, the Keycloak realms are imported from one JSON file per realm. Claude noted that this configuration was not optimal: brute-force detection was not enabled, and password complexity requirements were weak.

In production these points are configured properly. The whole production Keycloak configuration is versioned, reproducible (with OpenTofu) and auditable.

What we did with the findings

We treated every finding as a bug in its own right. One fix per problem, so each could be reviewed and verified separately. A regression test for each fix, which we checked failed before the fix and passed after. And one firm constraint: no fix was allowed to create a breaking change.

We fixed every problem and every hardening recommendation. Heavier improvements are planned for the coming weeks.

We did have to make one exception. An improvement to the dataroom required changing the behaviour of an API route. We decided to apply it immediately. As a result, every version of the Retyc CLI before 1.3.0 can no longer add files to a dataroom.

What we take away

An AI does not replace a human audit. But reading thousands of lines of code, listing dozens of endpoints and replaying every hypothesis with several accounts is days of work for a team.

Every finding came with its reproduction steps, an honest assessment of the impact, including when it was low, and a proposed fix. Claude also listed what it had checked and found sound. That is the part we were not expecting, and it is the one that served us most.

Human review remains indispensable. Some proposed fixes would have broken existing uses, and we had to discuss and adapt them before applying them.

Another lesson, about data validation: we were expecting too much from Pydantic. EmailStr accepts uppercase, and it is right to, since the local part of an address is case-sensitive per the standard. A string containing a null byte is equally valid in Python, even though PostgreSQL refuses it. Normalizing input stays our job, at the API boundary. We made it explicit where it was missing.

We are also going to work on dedicated skills, to improve and frame our pentests.

We intend to repeat the exercise regularly, on top of our code reviews and our tests.

In short

This audit confirmed the essential: your files stay encrypted end to end, unreadable by our team as by anyone else, even if a flaw were hiding in our code.

On almost every file transfer and storage service, it is the service that holds the keys. Support has access, administrators too, sometimes third parties.

At Retyc the server never holds the keys: even the most serious findings gave access to no file at all.

Do you have questions about Retyc security, or would you like to know more about our approach? Write to us.