Why Your Learning Data Is Not Ready for AI (and how Learning Engineering can solve this)
On your mark. Get set. Actually, hold on a second.
Ever get the sense that you just aren’t ready? Like you’ve got a good idea about what’s going on and you know what the objective is, but somehow everything just hasn’t come together. And you don’t know why. That’s the way it is with artificial intelligence for most learning and training organizations. But it doesn’t have to be that way.
After all, you wouldn’t take part in a track meet if you hadn’t prepared yourself for the race. You wouldn’t just show up and wing it. You wouldn’t choose to run the 5k just because you read on LinkedIn that it was the most popular race.
So why would you engage with AI before you’ve actually done the work required to be successful?
Every organization is talking about AI.
Some are piloting tutors. Some are experimenting with generative feedback. Some are imagining adaptive learning pathways, automated coaching, predictive analytics, and personalized workforce development at scale. But the uncomfortable truth is that most learning data is not ready for AI.
That does not mean organizations lack data. In fact, many have too much of it. They have LMS completion records, assessment scores, survey responses, course metadata, content usage reports, platform logs, HR records, credential data, simulation outputs, and dashboard exports. The problem is not the absence of data. The problem is that most of this data was never designed to work together, and certainly was not meant to be consumed by AI.
AI Needs More Than Content
A lot of AI-for-learning conversations start with content. How can we generate it faster? How can we summarize it? How can we create practice questions, lesson plans, study guides, and coaching prompts? Those might be useful applications. But they are not the real transformation. The real transformation happens when learning systems become observable, measurable, adaptive, and connected to performance. That requires more than content generation. It requires data architecture.
An AI system can only reason effectively about learning if it has access to meaningful evidence. That evidence has to describe what people did, under what conditions, with what supports, toward what goals, and with what outcomes. That is a very different data problem from “did the learner complete the course?”
Completion Is Not Competence
Many organizations still rely on training records that say a person completed something. But completion is a weak signal. It does not tell us whether the learner can perform under pressure. It does not tell us whether they retained what they learned, let alone why. It does not tell us whether they can transfer knowledge into a new context. It does not tell us whether the learning experience changed future decision-making.
For AI-enabled learning and training systems, this matters enormously. If the data going into the system is shallow, the recommendations coming out will be shallow. If the data is fragmented, the AI will inherit that fragmentation. If the learning model is unclear, the AI will optimize for whatever is easiest to measure rather than what actually matters.
This is how organizations end up with very sophisticated tools pointed at very weak evidence. It is also why learning organizations often come away feeling defeated after undertaking an AI project. Because it is not about the implementation of the technology, it’s about whether that implementation was made cognizant of the needs, and the limitations, of the organization itself.
Learning Data Needs Meaning
Modern learning ecosystems need more than raw events. They need data that carries meaning. That means being able to describe learning activity in a consistent way across systems. It means aligning learning experiences with competencies, performance expectations, metadata, and records that can move across platforms. It means designing learning data so that humans and machines can interpret it reliably.
This is where standards supporting xAPI, xAPI Profiles, Learning Metadata, Shareable Competency Definitions, and Enterprise Learner Records become important. Context forces us to move from a mindset of “we have implemented AI” to a mindset of “our solutions are actually built to solve the problems we have” and all that brings with it.
And data standards built for context help organizations move from isolated records to interoperable learning ecosystems. They make it possible to connect learning activity with performance evidence. And they help create the conditions under which AI can support learning in a way that is measurable, explainable, and useful.
Dashboards Are Not Enough
Many organizations think they have solved the learning data problem because they have dashboards. Dashboards are helpful (sometimes). Dashboards can help you see your data (sometimes). But dashboards are not a solution, they are a feature. Because dashboards are not architecture.
A dashboard shows what data is available. It does not necessarily make that data meaningful, interoperable, trusted, or actionable. A dashboard can tell you what happened in one system. It may not help you understand how activity across many systems contributes to readiness, capability, competence, or performance. We’ve seen beautiful dashboards filled with meaningless junk data.
Because “dashboards” do not equal “analytics”. Depending on the way they’ve been set up, they may not even equal “reporting”. So, if you are making acquisition decisions relevant to your learning ecosystem and your priority is “dashboards”, you are doing it wrong.
The next generation of learning analytics will not simply report on learning. It will help control, adapt, and improve learning systems. When you hear about advanced analytics, that doesn’t mean prettier dashboards. That means a system built on data. And the ability to leverage that data to perform any number of business functions. This requires data instrumentation close to the learning event. It requires semantic consistency. It requires technical infrastructure. It requires learning science. It requires governance. In other words, it requires Learning Engineering.
Learning Engineering Is the Bridge
Learning Engineering brings together learning science, human-centered design, data architecture, software engineering, and iterative improvement. It is the discipline organizations need when learning must be measured across complex environments. It is especially important when those environments include simulations, AI-enabled systems, workforce platforms, credentialing systems, and operational performance data.
A Learning Engineering approach asks different questions about what evidence would show that learning occurred and what activity should be captured. Learning Engineers examine how systems exchange data and recommend what standards should govern that data. They explore what competencies or performance outcomes an organization is trying to support and help business stakeholder understand the decisions that could be improved should this data be made accessible. These are the things that have to be considered before AI can deliver on its promise.
Before You Add AI, Assess Your Learning Data Readiness
If your organization is serious about AI for learning, start with the foundation. Ask yourself if you have a learning data strategy and whether your systems are interoperable. Get your ISDs and software engineers in a room together and ask them if your learning experiences are instrumented, if your competencies are machine-readable, if your records auditable and meaningful.
Bring in a data scientist and a team of business users and examine whether your analytics tied to real decisions. Bring in leadership and have a real talk about whether you’ve developed an ethical framework for the adaptive or AI-enabled learning systems that you are developing and procuring.
If the answer to any of these questions is unclear, the next step is not another AI pilot. The next step is advisory with a partner who has been down this road many times before you.
Yet Advisory
Yet Advisory helps organizations design, evaluate, and improve modern learning ecosystems. Our expertise covers Learning Engineering, AI Ethics for Learning, xAPI and xAPI Profiles, Total Learning Architecture, and Digital Learning Ecosystems. Whether you need a strategic advisory engagement, a workshop series, or a keynote event, Yet can help your organization move from fragmented learning data to measurable, interoperable, AI-ready learning infrastructure.
Claim 20% off your first advisory engagement today.