When Impact Measurement Misleads: How Charities Can Cut Through the Noise

A practitioner’s guide to moving from "vanity metrics" to data that actually supports your mission.

We’ve all been there. You’re staring at a spreadsheet full of green KPIs and "successful" outcomes, yet on the ground, your team is exhausted, and your beneficiaries are still struggling. There is a quiet, invisible gap between what the Board sees and how the work actually feels.

I’ve seen this from both sides. I spent years as a Chief Scientific Officer designing these frameworks, and then moved into operational leadership where I had to actually run services using them. That shift changed everything for me. It taught me that if a metric doesn’t help a practitioner make a better decision or improve someone's experience, it’s just expensive noise.

With AI making it easier than ever to collect and generate data, there’s a real risk of things getting even more cluttered. Right now the sector doesn't need more data but we do need the right data. I’ve put together a five-step plan to help you move away from "vanity metrics" and toward a system that actually supports your mission. This is for you if:

• You’re collecting plenty of data but aren't sure it’s actually making your service better.

• You’re a leader trying to give funders what they need without burning out your staff.

• You want your measurement to be about people, not just numbers.

For a staff-friendly tool stack and a one-page cheat sheet, see: The 2025 Charity Impact Measurement Tool Cheat Sheet

Why impact measurement usually goes wrong (and the traps to watch for)

Before we look at how to fix things, we need to be honest about why so many systems break down. In my experience, both as the person designing these frameworks and the person trying to use them to run a service, there are five common patterns that turn good intentions into "expensive noise."

1. When data collection feels like "homework" for your people

If your staff and the people you support don’t see the point of the data, they’ll treat it as a box-ticking exercise. When data collection feels like a chore forced on them by the board or a funder, the quality drops and the resentment grows.

What to do instead: Work with your team to decide what actually matters. People are much more likely to collect good data if they know it helps them do their job better.

2. Focusing on statistics but losing the human story

It’s easy to get obsessed with spreadsheets, but stats alone rarely tell the whole story. You might have great numbers, but if you aren't capturing the lived experience of the people you’re helping, you’re missing the point of why you’re there.

What to do instead: Don’t let numbers stand alone. Use personal stories to explain the "why" behind the "what."

3. Using measurement tools that don't fit your daily operations

I’ve seen many charities try to use "off-the-shelf" metrics that just don’t fit their culture or their daily workflow. If a reporting tool is clunky or uses language that your team wouldn't use in real life, it’s going to fail.

What to do instead: Keep it simple and keep it relevant. Choose measures that feel like a natural part of the conversation, not an interruption to it.

4. Ignoring the external context behind your results

Charities don't operate in a vacuum. Sometimes your outcomes might look "worse" on paper, but when you look at external factors. Think, like a cost-of-living crisis or a change in local policy. You’ve actually done an incredible job just to hold steady.

What to do instead: Track the context. If you know what’s happening in the wider world, you can explain your results with more honesty and confidence.

5. Tracking metrics that no longer align with your mission

Too often, charities keep measuring things they stopped doing years ago, simply because "that’s how we’ve always done it." If your goals have changed but your metrics haven't, you're wasting time on data that no longer has a home.

What to do instead: Review your metrics regularly. If a piece of data isn't helping you make a decision or prove an impact, stop collecting it.

Five steps to transform your impact measurement (without an overhaul)

You don’t need a massive budget or a total system redesign to start making your data more meaningful. Here is a simple, five-step plan to help you move toward a system that actually supports your mission.

1. Start by clarifying the ‘Why’ behind your mission

Before you look at spreadsheets, get your leadership and frontline staff in a room. You need to agree on what you’re actually trying to achieve. If the goal isn't clear, the data will always feel like a burden.

• The conversation to have: Ask your team: "What is the one outcome we exist to achieve?" and "What would failure look like for the people we support?"

• Try this: Find the one goal your team would rally around, even if it feels ambitious. That’s your starting point.

2. Define the ‘How’ through shared experience

Metrics shouldn’t be handed down from the board; they should be grounded in the reality of the work. The best way to do this is to talk to the people who are actually involved.

• The conversation to have: Host a small workshop with a mix of staff and beneficiaries. Ask them: "What changes would show us that we’re actually succeeding?"

• Try this: When was the last time you asked the people you support what success looks like to them? Their answers might surprise you and lead to much better metrics.

3. Choose metrics that matter (not just ones that flatter)

It’s tempting to pick the numbers that always look good, but "vanity metrics" don't help you improve. You need metrics that tell you the truth, even when it’s uncomfortable.

• The conversation to have: For every metric you track, ask: "Is this tied to our mission?" and "Can we actually take action if this number changes?"

• Try this: Identify the one metric that would keep you up at night if the numbers started to drop. That’s a metric that matters.

What to avoid (Vanity Metric)	Why it’s misleading	What to track instead (Meaningful)
Total number of interventions	Shows how busy you are, not the result.	Beneficiary quality of life or progress score.
Event attendance numbers	Counts bodies in the room, not impact.	How skills were used after the event.
Number of website hits	Shows reach, but not engagement.	The number of people taking a specific action.

4. Change how you talk about data with your team

If data training is just a dull slide deck, people will switch off. To get buy-in, you need to make it relevant to their daily work and show them how it makes their lives easier.

• The conversation to have: Ditch the formal training for hands-on sessions. Sit down together and look at real data. Ask: "What is this telling us about our service today?"

• Try this: Find the biggest skeptics on your team and involve them in designing the training. If you can convince them, the rest of the team will follow.

5. Be brave enough to stop measuring what isn't working

One of the most important parts of "tidying the system" is knowing what to throw away. If a metric doesn’t inform a decision or inspire your team, it’s just clutter.

• The conversation to have: Every six months, look at your reporting and ask: "Has this sparked any action lately?" and "Would we actually miss this if it was gone?"

• Try this: Ask your team: "Which metric would you drop tomorrow if you had the choice?" If they all point to the same one, it’s time to let it go.

Ready to move forward?

Impact measurement isn’t just another task on your to-do list. It’s how you protect what actually matters for the people you support and stop wasting energy on the things that don't. By following these five steps and only introducing new tools when your system is ready for them you can turn your data from a heavy administrative burden into a tool for making better decisions.

• A quick reflection: Which metric on your report feels like "expensive noise" today? What would happen if you just stopped tracking it?

• Next step: If you want to see where your gaps are, you can take my two-minute impact self-assessment here.

FAQ for Charity Teams Measuring Impact

1. How do we measure impact without overwhelming our staff?

Start small. You only need two or three metrics that actually guide your decisions, plus one human story to provide context. The best way to reduce overwhelm is to give your team "permission" to stop collecting data that isn't being used. When people see that the data they do collect leads to real change, the perceived burden starts to feel like a benefit.

2. Which metrics should we track first?

Pick the ones that link directly back to your "why" ie. the reason your charity exists in the first place. Don't choose the metrics that make you look good; choose the ones you would actually miss if they disappeared tomorrow. If you aren't sure, ask your frontline team what success looked like in their most recent shift.

3. How do we actually stop collecting data that no longer serves us?

You have to be intentional about it. Every six months, look at your reporting and ask: "Has this piece of data sparked a single action or decision in the last half-year?" If the answer is no, drop it. Getting rid of unhelpful noise is the fastest way to tidy your system.

4. When should we start using AI for impact measurement?

Only after your core process is clear and trusted. If your current metrics are messy or your team doesn't believe in them, AI will just help you make a mess faster. Once you’ve "tidied the house," simple AI tools are great for things like summarising feedback or spotting patterns in qualitative data. I’ve written more about safe AI implementation in charities here.

5. How do we report to boards and funders without it becoming a massive job?

Keep it focused. Give them the headline (your "why"), the two or three metrics that prove progress, and one story that shows the human side of the work. As these measures were co-created with your team, they are much easier to defend and much quicker to write up than a fifty-page report full of vanity stats.

Change doesn’t start with a workshop; it starts with one honest conversation that builds trust and momentum.

Let's Chat

Note: Examples are for illustrative purposes only; no official affiliation with the organisations or tools mentioned is claimed. AI systems can be unpredictable, so always keep personal or sensitive data out of third-party tools and ensure your implementation follows your own organisation’s data protection policies.