Abdolmadjid Masoomi

What I Look For When I Audit a Service

A short list, in the order I check it, and why the first item is almost always the last word

Published
2026-09-12
Length
4 min read · 714 words
Status
first person account

A practical procedure for assessing whether a service can do what it says about your data, written as an order of operations rather than a checklist. Most assessments are settled by the first two questions; the rest establish how much of the answer was deliberate.

The order matters more than the list

Most assessment checklists are unordered, so attention gets spread evenly across items that are not evenly important. An hour goes on cipher choices while the question of who can decrypt goes unasked.

I work in sequence instead. Two questions decide most outcomes. Everything after them is establishing whether those two answers were designed or arrived at by accident — which matters, because an accidental right answer does not survive the next refactor.

One: who holds the key

Always first.

If the provider holds it, the architecture has already answered the question. No control added later outweighs that fact, and the remainder of the assessment shifts from can they read this to are they competent and disciplined enough not to. That is a real question with real answers, and it is a different question.

If the user holds it, the risks move rather than disappearing. Recovery becomes the exposure, and the integrity of the client — the code doing the encrypting — becomes the thing worth examining, because that code arrives from the party you are protecting yourself from.

Neither answer is disqualifying. But everything downstream means something different depending on which one it is.

Two: what happens on password reset

The fastest architectural probe available, answerable from a help page in about a minute, and it cannot be finessed.

If a service restores your data after a complete password reset, it holds a route to your key. Not as a policy. As an arithmetic consequence.

If it tells you plainly that a lost passphrase means lost data, that is a costly sentence to publish — support will be handling those conversations forever — and people do not publish costly sentences they were not forced into by their own design.

So: one page, one minute, and the key-custody question is usually settled without opening any technical documentation at all.

Three: what is collected that the feature does not need

Surplus is the clearest signal of intent, because nothing collects extra data by accident for long.

Permissions beyond function. Fields marked required that plainly are not. Telemetry on by default. Identifiers that persist across sessions, and across logged-out sessions in particular — which exist for one purpose and it is not diagnostics.

None of these individually proves anything. Together they describe what the service is for, as distinct from what it does.

Four: what the service says it cannot do

I look for stated limits, and their absence is more informative than most stated features.

A published limitation is expensive. It loses sales, it constrains the roadmap, and somebody in marketing argued against it. A document containing no limitations is not describing a system — nothing real has no edges. It is describing an intention to sell.

"We cannot recover your files if you lose your passphrase" tells me more than three paragraphs about commitment to security.

Five: how they behave when something is wrong

A named security contact. A disclosure policy that says what happens after a report. Past issues described in public rather than fixed silently. A changelog that admits a fix was a security fix.

This predicts future behaviour better than any control currently in place, because controls change and disposition does not. A team that describes its mistakes has decided something about itself that will still be true after the people change.

What I do not bother with

Certification badges without a scope, since the scope is the whole content of the claim and its absence is deliberate.

Long policy documents, which describe what a company is permitted to do rather than what it intends to, and are written to survive challenge rather than to inform.

Feature-count comparisons, which measure surface area.

And any claim of strength phrased as an adjective.

The order again

Key custody. Reset behaviour. Surplus collection. Stated limits. Conduct when wrong.

A service failing the first question can still be the right choice. Plenty of work is not sensitive, and provider-held keys buy search, previews, sharing and recovery that people genuinely want. The point of the sequence is not purity. It is that you should know which arrangement you chose, on the day you chose it, rather than on the day it matters.