Capture the Flag (CTF) exercises are one of the ways to learn digital forensics because they make you do the work. You have to find the artifact, understand what it means, sometimes validate it manually, and usually figure out at least one thing you did not know before you started. A good CTF can force you into unfamiliar file systems, mobile applications, logs, databases, memory, network data, or tools you would not otherwise touch. That is useful training. It builds technical skill, persistence, and confidence.

The other benefit and perhaps the most important is self-induced stress. The pressure of not knowing if you can solve it, or trying to beat the clock against competitors, creates a noticeable feeling of pressure but without the stakes of a real case.

But every CTF contains an assumption that does not exist in a real investigation. In a CTF, there is an expected answer. You know that before you start. A CTF is gradable against a key.

Someone designed the challenge. Someone designed the question and established the expected answer before you ever saw the evidence. The artifacts were created, selected, or preserved in a way that makes the problem solvable. You might not know where the answer is, and you might spend hours looking for it, but you know it exists.

That changes the way you think about the evidence.

If the challenge asks which USB device was connected, you assume there is evidence of a USB device. If it asks what file was exfiltrated, you assume a file was exfiltrated. If it asks which IP address the attacker used, you assume the IP address exists somewhere in the evidence. If it asks who committed the activity, you assume the evidence can lead you to a person.

So you keep digging until you find it. That is exactly what you should do in a CTF. This is not a flaw of a CTF; it is the definition of it. A challenge with no recoverable answer is a broken exercise or a research problem.

Real investigation: You have no guarantee the available evidence can answer the question.

The problem begins when you bring that same expectation into a real investigation.

Real evidence makes no promise that an answer exists. Sometimes the data you need was never created or was overwritten. Sometimes the wrong device was collected. Sometimes a cloud service kept only a few days of records. Sometimes three people knew the password and two people used the same computer.

And sometimes, after doing everything correctly, the answer is still: I don't know.

In most CTFs, “I don't know” means you haven't finished. In a real investigation, “I don't know” may be the only defensible finish.

This is important because the technical mindset required to find a known answer and artifacts is not exactly the same as the investigative mindset required to determine whether an answer can be found and supported at all.

Suppose an employee is suspected of stealing confidential files before leaving a company. You receive the employee's laptop and begin examining it. You find that a USB storage device was connected the night before the employee resigned. You find that several sensitive files were opened around the same time. You recover LNK files and Jump Lists associated with some of those documents.

The story begins writing itself:  Employee. USB drive. Sensitive files. Resignation.

It makes sense, and that is where an investigator should fight against complacency. A plausible story is not a proven story. An investigation will potentially need defensibility to a hostile audience while a CTF is checked against an answer key.

Was the employee actually using the computer at that time? Did anyone else have access to the account? Was the USB device theirs? Do the artifacts show that the sensitive files were copied to the device, or only that they were opened while the device happened to be connected? Was remote access possible? Was there synchronization software running? Are the timestamps being interpreted correctly? What additional artifacts should exist if the files were actually copied? Are those artifacts present? What would you expect to find if the theft theory were wrong?

A CTF rarely forces you to challenge the premise of the challenge itself. The scenario exists because someone created it. There is a hidden story, and your job is to reconstruct it.

A real investigation does not give you that luxury of knowing a story can be reconstructed. You first have to determine whether the story you are building is actually supported by the evidence.

That is why I think there is a meaningful difference between technical analysis exercises and investigative exercises. They overlap and can support each other. But they are not the same thing.

A technical challenge often works like this: find the artifact, interpret the artifact, answer the question. Go to the next question.

An investigation is complex. You don't know how many questions you will have to ask to find the answer, or whether the answers even exist. Worse yet, you may never know if you were right because there is no answer key.

In those situations, the problem is no longer finding the right artifact. Finding an artifact may answer a technical question. Deciding what it proves in the case answers an investigative one. That is a different skill altogether.

CTFs also give us something else that real investigations often do not: they tell us what question matters.

• “What program established persistence?”
• “What was the filename?”
•  “What IP address was used?”
• “What USB serial number was connected?”

The question itself narrows the universe of data. You know what kind of evidence matters before you start looking. That gives you a head start on tools you will choose and a competitive path forward.

Real cases get dropped on your desk with nothing that simple.

You may receive several terabytes of data and a sentence that says, “We think this employee stole confidential information sometime during the last six months.”

Now the first problem is not finding evidence. It is deciding what you are actually trying to establish.

What would theft look like in this environment? USB transfer? Personal email? Cloud storage? Printing? Screenshots? Remote access? A personal device? A file-sharing service? Which systems would record those actions? What logs should exist? What time period matters? Which facts are already known? Which facts are merely allegations? What evidence would support the allegation? What evidence would weaken it?

Those questions have to come before the artifact hunt. Without them, everything becomes interesting. Interesting fills your storage devices. Relevant moves the investigation.

A technically impressive examination can still be an investigative failure because you can be technically correct about every artifact you found and still be wrong about what happened.

A CTF constrains the relevance problem because someone else has already selected the evidence universe and decided which questions you are trying to answer. You may still have to separate signal from noise, but you did not have to decide what the investigation was about.

And that means sometimes ignoring technically interesting things. It also means resisting the urge to force every artifact into a story, or dig so deep into every artifact that you miss the relevant ones.

Everyone wants answers. If someone spends money on a forensic examination, they expect the examiner to tell them what happened. If charges are pending, everyone is demanding to know the story of the data.

But the purpose of an investigation is not to manufacture an answer. It is to determine what the evidence can support. Sometimes the evidence supports a strong conclusion and other times not.

• The USB device was connected, but I cannot determine what was copied to it.
• The activity is associated with the account, but I cannot identify who controlled that account at the time.
• The file was accessed, but I cannot determine whether it was exfiltrated.
• The evidence supports one explanation more strongly than another, but important uncertainty remains.

Those are not weak conclusions if they accurately reflect the evidence. Pretending to know more than the evidence supports is not an investigation or analysis.  

This is also where persistence can become a problem. Persistence is one of the things CTFs teach extremely well. You do not have the flag yet, so you keep going. Search another artifact. Try another parser. Review another log. Write another script. Look again.

In a challenge, that behavior is rewarded because you know that there is something to find.  If you know an answer exists, persistence is a virtue. CTFs can cause confidence in the wrong things. Being good at finding artifacts in a CTF subtly becomes “I am good at investigations.”

If you merely believe an answer must exist, persistence can become confirmation bias. At that point you are no longer asking what the evidence says. You are looking for evidence to agree with you. There is an important difference between refusing to give up too early and refusing to accept that the evidence may not support your preferred answer. 

The point is not that CTFs are bad. The point is that they are very good at teaching certain things, and we should be clear about what those things are. We should be clear in telling people what the goals and objectives of CTFs are.

They teach technical analysis well. They teach how to search, parse, correlate, validate, and persist. Competitive CTFs reward speed. The mistake isn't training with CTFs. The mistake is confusing success in a controlled problem with competence in an uncontrolled investigation that adds judgment, uncertainty, relevance, competing explanations, and decisions that may never produce an answer.

A CTF ends when you find the answer or the clock runs out. A real investigation should end when you can defend your conclusion.

Just remember the one thing every CTF gives you that a real investigation does not: Someone already knows the expected answer. In a real investigation, nobody may know the answer, or those who do know may lie to you, give you false leads, or retract their story after bias led you down a rabbit hole.

DFIR is not a technical sport. The artifacts may be technical. The conclusions are investigative. And when the question is who did what, the investigation eventually leaves the machine and reaches the human being behind it.