Building Your First Dataset, and the Responsibility That Comes With It

Datasets are the building blocks for solving a problem and achieving a purpose. A model's performance depends greatly on the data it is being trained on. That's why decisions about what to build a dataset for, where to source it from, and how to organize it matter as much as the modeling itself. Building a dataset is a challenge in itself, and building one for the first time can feel chaotic.

The myth of "just data"

I expected my first challenge in building a dataset would be technical, such as writing a scraper, cleaning messy text, or figuring out train/test splits. The hardest part turned out to be a question I hadn't prepared for: who decides what counts? Every added row was a small verdict about what was valid, what counted as noise, and what information was important enough to add. It was on me to decide what becomes part of the dataset.

Collecting data to build a dataset can seem tedious, whereas training the model, tuning it, and watching it improve feels like the real work. But most of what a model later gets right or wrong was already decided here, in this early, unglamorous stage. The model only learns what the dataset has. If the dataset is missing certain examples, fails to correctly label any data, or has wrong information, then the model cannot invent a way around it. The dataset is not the framework for building; it is the building.

Defining the task before you collect anything

Before collecting a single example, the question arises: what is the purpose of this task? The answer shapes the whole process before it even begins.

For example, when building a dataset for classifying email as "spam" or "not spam", the task appears simple. It seems like a clean, factual label: either it's spam or it isn't. But how do we decide that? What makes an email spam? Does all promotional email get labeled as spam? Do you label an email in a different language as spam by default? Answer these differently, and you get a different dataset. Based on the answers, the model will learn what counts as spam, and that will shape its performance.

I assumed the hard part was cleaning and modeling, but surprisingly, this is the hardest and most important work. The decisions that led to the collection of the data determine its value, and correctly documenting them determines its usability. That gap, between how confidently you use a word like "spam" and how hard it is to define, is where the dataset actually starts taking shape.

Collection: where representation gets decided

Once you know what you're trying to capture, the next decision is where you go looking for it. This matters because different collection methods don't just gather different amounts of data; they gather different kinds of data.

Let's assume you're building a small dataset of customer feedback to test a sentiment classifier. One option is to scrape public reviews from a site. It's fast, and you'll get a lot of examples quickly. But you'll also mostly get people who were motivated enough, either with anger or joy, to write a public review, and you miss most of the real customers, the quietly satisfied ones. Another option is to ask a smaller, familiar group of people to write feedback directly to you. You'll get more balanced and thoughtful responses, but now your dataset reflects the opinions and writing style of a narrow circle, not the public at large.

Neither method is wrong, but each one determines "who counts as a customer" before you've written a single line of modeling code. Ignoring it only makes the model's performance worse. It resurfaces later as the model that performs strangely well on angry or happy reviews and strangely badly on mild, everyday feedback. Spending hours debugging the model won't fix it, because the real issue was decided weeks earlier, at collection.

Labeling and annotation: whose judgment counts as truth

After collecting all the examples, the next step is to label them. Someone has to decide what each example means, and the model will treat it as the "ground truth".

For instance, how would you label this comment, "Not bad, I guess", positive or negative? You set the instruction that any comment that does not explicitly have any negative words in it is considered positive. So, this comment gets labeled as positive. In another case, the instructions say a comment expressing hesitation gets labeled as neutral. So, the same comment gets labeled as neutral. Neither instruction is wrong. However, for the model, the only truth is the instruction you write down.

Although "ground truth" sounds like a rule written in stone, it is actually a rule someone makes under pressure and without having the idea fully thought through. That doesn't make labeling arbitrary. It makes it a decision worth making carefully, and worth writing down honestly, rather than treating as a detail to rush past.

Documentation as an ethical practice, not paperwork

By the time you have collected and labeled the dataset, you have gained knowledge of its structure and purpose. This is where the documentation is required, instead of waiting for the "real work" to be done. In this documentation, you explain who this dataset is for and who it might not serve well.

Documentation of a dataset is like an instruction manual. It describes what the dataset is, how it was made, and what it's meant for. Applied to a first dataset, this doesn't need to be formal. Only a few questions need to be answered: Who collected this, and why? What's likely missing or overrepresented? What should this dataset not be used for? This practice can help you figure out a problem before it becomes a debugging session. Skipping this step means the gap stays invisible until you or someone else runs into it without any warning.

Consequences and reuse: the dataset outlives you

There's a point where the dataset stops being yours. After you publish it or share it with someone, the data is out of your hands. It gets used for purposes you didn't intend and cannot fully know. Someone fine-tunes a different model on it or uses it on a different task that greatly differs from your plan. The decisions made earlier about what to include, how to label, what counted as normal quietly travel with it.

It doesn't matter which field you are working in; the data is treated the same. The information collected for one purpose gets used at a different time, in a different scenario, for a different purpose. Once data exists, it has a life beyond the reason you collected it. But the responsibility for this data doesn't disappear; it just becomes harder to see.

Before you collect anything: a few questions worth sitting with

There is no specific checklist that promises to resolve the uncertainty or make the process any easier. But a few questions are worth considering before collecting anything to avoid making any decisions by accident:

• What am I actually trying to capture, and have I written that definition down?
• What is overrepresented in how I'm collecting this, and what is missing?
• If someone else labeled this instead of me, would they land on the same answer?
• Who might use this dataset in a way I haven't thought of?
• If I had to explain this dataset's limitations out loud, what would I be embarrassed to admit?

The last question is the hardest to answer and the most vital one. Every dataset has its limits, and it is important to highlight them before passing it to someone else. The work isn't making that answer disappear; it's being willing to write it down.