Questions to ask MIS, and what a good answer looks like

You do not need to understand the ILR to hold it to account. You need to know which questions expose a problem.

The best questions are about what is known to be wrong, not about whether everything is right. Nobody can answer the second one honestly.

A question like 'is our data accurate?' invites reassurance. A question like 'how many learners are recorded as in learning past their planned end date, and what is the largest reason?' invites a number, and the number is either small and explained or it is a problem.

The pattern to look for in an answer is specificity. A good answer names a figure, a cause and an owner. A weak answer describes a process.

The questions below map onto measurable positions in the return, so each can be answered from a report rather than from opinion. Where a college cannot produce the number, that is itself the finding.

Four questions cover most of what can go wrong. Ask the same ones every month so the trend is visible rather than the snapshot.

• How many learners are recorded as in learning who have not attended recently?
• How many aims have ended with no outcome recorded?
• What moved in earnings since last month, and why?
• What is outstanding on the data quality and monitoring reports?

Each of these has an observable answer that comes from a report rather than from an opinion, which is what makes them useful. A college that cannot produce the number has told you something important without meaning to.

Asking the same four every month turns them into a trend. A count that rises month on month is a control failure, and the trend makes that visible in a way a single figure never does.

The fourth question is the one that most often uncovers something. Reports that arrive and are not worked accumulate quietly, and they test exactly what an audit will test.

The first two map to standing reports over the current return: learning deliveries with no actual end date whose planned end date has passed, and learning deliveries with an actual end date and no outcome recorded.

The third is the period-on-period comparison of the funding reports, worked at total level then by funding model then by curriculum area.

The fourth is the exception log: every flagged record with its outcome recorded as corrected, evidenced, or awaiting evidence.

The funding monitoring reports are lists DfE sends of records it wants a provider to look at. Appearing on one does not mean a record is wrong. Many of the records on them turn out to be entirely correct and simply need their reason recorded.

So the question is not how many errors are on the reports. It is what has been done about each record, and how many are still sitting there.

• Which funding monitoring reports have we been sent, and how many records are on each?
• How many of those records have we worked through to a conclusion, and how many are still open?
• Of the ones we concluded, how many were correct and how many were corrected?
• For the correct ones, where is the evidence that makes them explainable next year?

Asking how many errors there are invites a defensive answer, because the honest first answer is that nobody knows yet. Asking how many records are still open invites a number, and that number is the one that matters: a record nobody has worked is the same risk whether it turns out to be right or wrong.

The third question is the one that tells you about the college rather than about the data. A set of reports where everything turned out to be correct has usually not been investigated; a set where everything was an error usually means somebody corrected records to make a report go away.

The fourth question is the one an audit asks. A record that was correct and has nothing recorded against it is indistinguishable, a year later, from one nobody looked at.

Each report has a defined population, so the count is reproducible rather than a matter of opinion, and the same question can be asked every time the reports arrive.

The outcome to record against each record is one of three: corrected, evidenced as correct, or awaiting evidence. Anything that is not in one of those three states is still open, whatever anybody believes about it.

Where a correction was needed, the useful follow-up is how many other records share its cause. A report listing eleven learners is more often one process than eleven mistakes.

Weak answers describe a process. Good answers give a number and name a person.

None of this means anybody is being evasive. Most weak answers come from people who have never been asked for the number before.

"We run a report on that" tells you a report exists, not what it says. "It is under control" is a judgement, not a measure. "The auditors did not raise it" describes a sample at a point in time.

A good answer has three parts: the number, the reason, and who owns it. If any part is missing, the follow-up question writes itself.

Where the answer genuinely is not known, that is useful information too. It usually means nobody has been asked to produce it, which is a fixable problem rather than a concealed one.

Build the four measures into a standing one-page report with the count, the trend and the named owner for each. That converts the monthly conversation from an investigation into a review.

Report by curriculum area rather than in total. A college-level figure hides the one area where the process has broken, which is exactly the thing worth finding.