[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"BLOG_POST_en_ai-pentest":3,"BLOG_SURROUND_en_ai-pentest":347},{"id":4,"title":5,"authors":6,"body":9,"category":329,"date":330,"description":331,"extension":332,"image":333,"meta":334,"navigation":335,"path":336,"seo":337,"sitemap":340,"stem":341,"tags":342,"__hash__":346},"blog\u002Fblog\u002Fen\u002Fai-pentest.md","AI-driven pentest: what we learned",[7],{"name":8},"Emilien Mantel",{"type":10,"value":11,"toc":311},"minimark",[12,16,21,24,32,41,45,57,72,81,85,88,91,94,97,100,125,129,136,139,142,146,149,152,166,169,172,181,187,191,196,199,206,211,215,218,222,225,235,239,242,245,259,263,266,269,272,284,287,290,294,297,300,303],[13,14,15],"p",{},"Retyc is built to exchange sensitive files, and our promise fits in one sentence: your files are encrypted on your\ndevice, and nobody but your recipients can read them, not even us. A promise like that has to be tested regularly. Until\nnow we did it with code reviews, automated tests and AI audits, but always limited to reading code. Anthropic's Cyber\nVerification Portal let us go further: a real pentest of Retyc, run by Claude. Here is how it went.",[17,18,20],"h2",{"id":19},"anthropics-cyber-verification-portal","Anthropic's Cyber Verification Portal",[13,22,23],{},"Asking an AI to look for vulnerabilities raises an obvious question: what stops someone from turning it against other\npeople's systems? Anthropic governs this kind of use with the Cyber Verification Portal. You have to be admitted before\nyou can use Claude for offensive security work on your own systems.",[13,25,26,27,31],{},"We were granted access, and we set ourselves a strict framework. The pentest covered our development environment only, *\n",[28,29,30],"em",{},"never production, never our customers' data","*. It relied on dedicated test accounts spread across several organizations\nand several plans, to check the separation between customers. And every finding had to come with a written report: how\nto reproduce it, what the real impact was, what the fix should be.",[13,33,34,35,40],{},"The development environment carries none of the protections that surround production: no\n",[36,37,39],"external-link",{"url":38},"https:\u002F\u002Fcrowdsec.net\u002F","CrowdSec",", none of our hardened infrastructure configuration.\nThat is deliberate. We wanted to see the flaws of the application itself, without a perimeter defence hiding them.\nThe findings below are therefore harsher than what an attacker would meet facing our production.",[17,42,44],{"id":43},"a-bumpy-start","A bumpy start",[13,46,47,48,52,53,56],{},"Not everything worked on the first try. Anthropic's documentation contained translation errors. Our first tests with\nOpus ",[49,50,51],"code",{},"5.5"," actually ran on Opus ",[49,54,55],{},"4.8",". We only understood this from the warnings the AI itself raised.",[13,58,59,60,64,65,68,69,71],{},"The fix was to move to ",[61,62,63],"strong",{},"Opus 5"," (",[49,66,67],{},"5",", not ",[49,70,51],{},", mind the difference!), which raised no warning. Everything that\nfollows comes from the audit run with that model.",[73,74,75],"blockquote",{},[13,76,77,78,80],{},"As of writing, the Cyber Verification Portal does not yet allow the Fable or Opus ",[49,79,51],{}," models.",[17,82,84],{"id":83},"how-the-pentest-went","How the pentest went",[13,86,87],{},"The pentest ran in several passes, including a complete restart that reused no conclusion from the previous one, and a\nverification round after each wave of fixes. Claude had access to the source code, to get a complete view of the\napplication.",[13,89,90],{},"One rule above all: Claude was not allowed to simply read the code and infer vulnerabilities from it.",[13,92,93],{},"On the backend, Claude listed every endpoint exposed by our API and read the whole codebase (several tens of thousands\nof lines). It then checked its hypotheses under real conditions, with 4 accounts spread across 3 organizations,\nanonymous calls and API keys. Every finding it kept was reproduced: none was merely deduced from reading the code.",[13,95,96],{},"On the web application, Claude drove a headless Chrome. It logged in, unlocked the encryption key and actually sent a\nfile containing a known marker. It captured every request leaving for the server, to see what really leaves the browser.",[13,98,99],{},"Among other things:",[101,102,103,107,110,113,116,119,122],"ul",{},[104,105,106],"li",{},"fuzzing",[104,108,109],{},"HTTP header injection and malformed requests",[104,111,112],{},"hunting for XSS and SQL injection",[104,114,115],{},"hunting for concurrency issues and race conditions",[104,117,118],{},"checking the separation between organizations and between roles",[104,120,121],{},"checking the security of the integration API (key scopes, IP address restriction)",[104,123,124],{},"attempting privilege escalation inside an organization",[17,126,128],{"id":127},"what-held","What held",[13,130,131,132,135],{},"The most important part first: ",[61,133,134],{},"end-to-end encryption held",". During the upload, no request contained the file content,\nits name or a private key. File names and types are always encrypted. Unlocking your key triggers a single request, and\nyour passphrase never leaves your browser.",[13,137,138],{},"The audit also confirmed that the separation between organizations holds on the data: a user can neither read nor modify\nthe transfers, datarooms or settings of another organization. Inside an organization, a member cannot grant themselves\nmore rights than they have. For the integration API, key permissions and IP address restriction are properly enforced.\nFinally, sign-in follows the good practices of the OpenID Connect protocol.",[13,140,141],{},"No XSS vulnerability was found, and no SQL injection was possible.",[17,143,145],{"id":144},"what-was-found","What was found",[13,147,148],{},"Most findings fall under what is called hardening: extra protections that fix no exploitable vulnerability, but that\nlimit the damage of a future problem or a configuration mistake.",[13,150,151],{},"Among the things we fixed or improved:",[101,153,154,157,160,163],{},[104,155,156],{},"the browser security policy (CSP), already very strict on scripts: only one exception, needed by the component that\ndisplays images, was broader than necessary. It now allows only the precise code that component needs",[104,158,159],{},"a server that refuses to start when a configuration secret has been forgotten, instead of running with a default value",[104,161,162],{},"cleaner error messages when a sign-in token is incomplete",[104,164,165],{},"explicit locking of a file after upload in a dataroom",[13,167,168],{},"Two findings stand out, though, and we would rather talk about them openly.",[13,170,171],{},"In a dataroom, a user with the \"contributor\" role is not allowed to delete files, and the API did refuse. They could\nstill reach the same result through a detour: move a folder into a dataroom they own themselves, then delete it there.\nThe deletion took with it the files other members had put in that folder, including those of the owner of the original\ndataroom, and permanently. A check on the ownership of the destination folder was missing.",[13,173,174,175,180],{},"On email sign-up, used in particular to send files through a ",[176,177,179],"a",{"href":178},"\u002Fproducts\u002Fdeposit-box","deposit box",", the verification code\ncould be guessed by trying a large number of combinations: the code was too short and nothing capped the number of\nattempts. The proof of work required on each attempt was not enough, because the same token could be replayed. Our\nreverse proxy rate limit narrows the window considerably in production, but we do not want that control to rest on it:\nthe number of attempts is now capped, and a proof-of-work token is good for one use only.",[13,182,183,186],{},[61,184,185],{},"None of these points allowed anyone to read a file",": the content stayed encrypted end to end in every case. We fixed\nthem first.",[17,188,190],{"id":189},"the-false-positives","The false positives",[192,193,195],"h3",{"id":194},"default-values","Default values",[13,197,198],{},"Claude also reported points that were not real security problems. It flagged, for instance, that some configuration\nvariables of our API (salts, secrets...) had a default value. Those values only served in development: in production\nthey are always overridden.",[13,200,201,202,205],{},"We took the opportunity to set rules in our configuration anyway: ",[61,203,204],{},"a secret value never has a default",", and the server\nrefuses to start when one is missing, even in development or in CI. It must also meet a minimum length.",[73,207,208],{},[13,209,210],{},"These rules have applied in production since the Retyc beta. Writing them explicitly into our development\nconfiguration lets us catch an omission before it reaches production, or the on-premises instances of our customers.",[192,212,214],{"id":213},"debug-mode","Debug mode",[13,216,217],{},"Claude also noted that some API routes, reserved for our developers, were reachable in debug mode. That mode cannot be\nenabled in production.",[192,219,221],{"id":220},"the-keycloak-configuration-in-development","The Keycloak configuration in development",[13,223,224],{},"Locally, our development infrastructure runs on Docker Compose. At initialization, the Keycloak realms are imported from\none JSON file per realm. Claude noted that this configuration was not optimal: brute-force detection was not enabled,\nand password complexity requirements were weak.",[73,226,227],{},[13,228,229,230,234],{},"In production these points are configured properly. The whole production Keycloak configuration is versioned,\nreproducible (with ",[36,231,233],{"url":232},"https:\u002F\u002Fopentofu.org\u002F","OpenTofu",") and auditable.",[17,236,238],{"id":237},"what-we-did-with-the-findings","What we did with the findings",[13,240,241],{},"We treated every finding as a bug in its own right. One fix per problem, so each could be reviewed and verified\nseparately. A regression test for each fix, which we checked failed before the fix and passed after. And one firm\nconstraint: no fix was allowed to create a breaking change.",[13,243,244],{},"We fixed every problem and every hardening recommendation. Heavier improvements are planned for the coming weeks.",[13,246,247,248,254,255,258],{},"We did have to make one exception. An improvement to the dataroom required changing the behaviour of an API route. We\ndecided to apply it immediately. As a result, every version of the ",[176,249,253],{"href":250,"rel":251},"https:\u002F\u002Fgithub.com\u002Fretyc\u002Fretyc-cli",[252],"nofollow","Retyc CLI","\nbefore ",[49,256,257],{},"1.3.0"," can no longer add files to a dataroom.",[17,260,262],{"id":261},"what-we-take-away","What we take away",[13,264,265],{},"An AI does not replace a human audit. But reading thousands of lines of code, listing dozens of endpoints and replaying\nevery hypothesis with several accounts is days of work for a team.",[13,267,268],{},"Every finding came with its reproduction steps, an honest assessment of the impact, including when it was low, and a\nproposed fix. Claude also listed what it had checked and found sound. That is the part we were not expecting, and it is\nthe one that served us most.",[13,270,271],{},"Human review remains indispensable. Some proposed fixes would have broken existing uses, and we had to discuss and adapt\nthem before applying them.",[13,273,274,275,279,280,283],{},"Another lesson, about data validation: we were expecting too much from\n",[36,276,278],{"url":277},"https:\u002F\u002Fpydantic.dev\u002Fdocs\u002Fvalidation\u002Flatest\u002Fget-started\u002F","Pydantic",".\n",[49,281,282],{},"EmailStr"," accepts uppercase, and it is right to, since the local part of an address is case-sensitive per the standard.\nA string containing a null byte is equally valid in Python, even though PostgreSQL refuses it.\nNormalizing input stays our job, at the API boundary. We made it explicit where it was missing.",[13,285,286],{},"We are also going to work on dedicated skills, to improve and frame our pentests.",[13,288,289],{},"We intend to repeat the exercise regularly, on top of our code reviews and our tests.",[17,291,293],{"id":292},"in-short","In short",[13,295,296],{},"This audit confirmed the essential: your files stay encrypted end to end, unreadable by our team as by anyone else, even\nif a flaw were hiding in our code.",[13,298,299],{},"On almost every file transfer and storage service, it is the service that holds the keys. Support has access,\nadministrators too, sometimes third parties.",[13,301,302],{},"At Retyc the server never holds the keys: even the most serious findings gave access to no file at all.",[13,304,305,306,310],{},"Do you have questions about Retyc security, or would you like to know more about our\napproach? ",[176,307,309],{"href":308},"\u002Fabout\u002Fcontact-us","Write to us",".",{"title":312,"searchDepth":313,"depth":313,"links":314},"",2,[315,316,317,318,319,320,326,327,328],{"id":19,"depth":313,"text":20},{"id":43,"depth":313,"text":44},{"id":83,"depth":313,"text":84},{"id":127,"depth":313,"text":128},{"id":144,"depth":313,"text":145},{"id":189,"depth":313,"text":190,"children":321},[322,324,325],{"id":194,"depth":323,"text":195},3,{"id":213,"depth":323,"text":214},{"id":220,"depth":323,"text":221},{"id":237,"depth":313,"text":238},{"id":261,"depth":313,"text":262},{"id":292,"depth":313,"text":293},"Security","2026-09-27","Through Anthropic's Cyber Verification Portal, we had Retyc pentested by Claude, on our development environment.","md",null,{},true,"\u002Fblog\u002Fen\u002Fai-pentest",{"title":5,"description":338,"ogTitle":5,"ogDescription":339},"Retyc was pentested by Claude through Anthropic's Cyber Verification Portal. End-to-end encryption held, and most of what came back was hardening. Method, results and lessons.","Claude pentested Retyc. End-to-end encryption held. What we learned.",{"loc":336},"blog\u002Fen\u002Fai-pentest",[343,344,345],"security audit","pentest","artificial intelligence","itcQ4sK6ozkY-gmX4t_6MwfrM8DsjFUbMNOIERKUDlg",[333,348],{"title":349,"path":350,"stem":351,"description":352,"children":-1},"Anatomy of an encrypted upload in the browser","\u002Fblog\u002Fen\u002Fanatomy-of-an-encrypted-upload","blog\u002Fen\u002Fanatomy-of-an-encrypted-upload","A post-quantum key pair per transfer, 8 MB chunks encrypted one by one in a Web Worker, an API that refuses classic keys. What the browser does between the moment you drop a file and the moment the last byte leaves, with the code to show for it."]