Testing LMSs
Now that you’ve narrowed down your list to the 5 or 6 LMSs that came out best in the spreadsheet, let’s talk next steps. By this point, I’ll admit, I usually have a favorite or two. There’s usually at least one system where the UI looks better than the others, or a system that has a cool feature I’m dying to try or something else that makes one system stand above the others.
I’ve done testing a couple of different ways, and I’ve not found the “perfect” way yet. As with life, this, too, is a work in progress. To test, we have to have access to the system. For most companies, that means going through some sales calls first. You’re going to need to contact the company and talk with a sales rep. You’ll have a conversation or two around your needs and wants – they want to see if you’re a good fit for them, too. There’s usually a demo of the system. The demo should reinforce what you’ve already noted in your sheet – your requirements are met by this LMS. It’s also your time to ask questions – make sure those needs are going to be met. Have something that’s high on the “nice to have” list? Ask about it. Talk about your use cases. This LMS may have a way to do things that you haven’t considered.
The demo can sometimes be a good time to get a sneak peek at integrations. Some companies have instances of the LMS set up with most major integrations so they can be demoed. Sometimes if you want to see the integrations demoed you’ll need to schedule separate calls.
Honestly, I don’t mind the calls. This is a company that you’re signing on with for years and you want to get a good feel for them and how they operate. Speaking with multiple people, especially from support and other areas you’re likely to deal with often, is never a bad thing.
There’s also the matter of timing. If other people utilize the LMS, they are likely going to want to test it as well. I’m kind of split on the timing. Typically I’ve gone ahead and done testing on my own at this point and I’ve usually been able to eliminate another candidate or two. In my head, that’s better for everyone else who’s going to be testing; They’re familiar with what they need, but not the needs of the entire LMS and often they’re not officially tasked with testing. You have their needs on the requirements list. In testing solo now, it’s likely less testing that the larger group will have to do in the future.
As the counterpoint, it’s good to get as much input from stakeholders as possible, usually as early as possible. Your instructional designers or sales trainers might see a gap that you would miss. Also, they will get more time with the system this way and they are ultimately the people who will be putting content into the system. A lot of this decision is going to depend on where you work, how your teams are set up, and your relationship with the stakeholders.
Assuming you are currently using an LMS, make a list of the things you do consistently and what the pain points are. If this is your first LMS, make the list assuming what your most important and most frequent tasks are going to be.
Regardless of who is involved in the testing, the point is going to be the same: run through the tools of the LMS as you would in regular usage. Integrations aren’t usually going to be set up and occasionally there may be some features you can’t access, but do your best to run through things as you would when performing regular tasks.
How you get the actual trial and how long you have access to it will vary from company to company. There are some companies out there that let you create free sandboxes on your own. Most companies will give you a time-limited trial that has access to all of the features of the product. The exception I usually see here is if there are Premium features that come at an additional price; Sometimes those are included in the trial those, sometimes they aren’t – it depends on the company and whom you’re talking to. I’ve had trials last from one week to several months – again, it depends on the company.
Then there are the more restrictive companies. They either don’t do trials at all or will only let you have access to the product after you’ve had training. This is a red flag for me. If a company doesn’t let you trial the product, I’m out. An LMS is not something you buy without testing it. Period.
The companies that don’t give access without training are becoming more frequent. After all, they want you to have a great experience with their product and, regardless of what you say, they don’t really know what your skill level is. Sometimes it seems like it might be annoying, but it’s usually another person at the company to talk to, another resource for when questions come up, and someone who knows the product to walk you through it.
I’ll admit I’m a fan of getting into a product and learning by “playing around.” But if someone wants to walk me through first and show me how to create courses and enroll users why say no? It’s usually a sixty- to ninety-minute call where someone will walk you through the steps of doing major tasks in the LMS. It can be a good time for what-ifs. As they’re walking through a task, ask them what-if X happens. Keep it realistic, of course, you’re not trying to waste anyone’s time. But if you’ve used an LMS that had trouble with special characters in names, ask if this LMS does and ask them to show you.
Sometimes the training can be extreme. I recall one company that wanted me to go through several hours (I think it was four) of training before they’d give me a trial. If your product is so complicated that it needs hours to show someone how to use it at a basic level, there are problems.
Now that you know who is testing, you have access, and you have your list of things that need to be tested. Where do you start?
Coming from Instructional Design and Graphic Design, I usually start with tasks in those areas. I just find them fun. I’ll typically customize the look and feel of the system first. Can you change the colors? Upload a logo? Can you change the favicon? Don’t only look at how easy it is to do those tasks, but how easy was it to find where they were located and how well were they documented? Was there a nice little i to hover over and get additional informational text? Or descriptions right on the page? Was there no additional information on the page at all?
How about content? Can you upload SCORM? (A must for any LMS, IMO) Can you create content? How easy is it? Once you have content, put it into a course. How easy is it to add that content to the course? To organize the course? Can you rename the things you need to?
How will you get users into the system? Even if you don’t think you’ll have to do it manually, I’d still look at how you manually create users and how you enroll them into courses. You never know when things will fail and you’re going to have to do it the old fashioned way.
If you’re going to utilize quizzing or testing, take a good look at what the LMS offers compared to what you want. This is especially important if you currently do certifications or are thinking about them. Can you set a time limit? Number of attempts? How about question pools? How easy is it to add a question? Some things you might not think about – how is reporting impacted if you change a question? What about if you delete it entirely?
Does the LMS alert users to certain events (enrollment, pass/fail of an exam, pass/fail of a course, achievement of a certification, etc)? If so, what are those events? Can you add more or are they set by the system? How do the alerts go out? Are they customizable? If so, how much? Can you customize the entire thing, just a few portions, or something in between?
How are users getting into these courses? If it’s educational, you’re probably looking at an integration with your SIS. If you’re not, can users enroll in courses for free? Do they have to pay for every course? Test self-registration and eCommerce, respectively.
Run through any special features the LMS has. Are they things you’ll actually use? Will they make your life easier? Or are they just bells and/or whistles?
When your trial is complete, you’ll usually have a follow-up call with the salesperson. Honesty really is the best policy here. Tell them how the trial really went. Will a feature absolutely not work for you? Let them know. If you see some minor annoyances, but nothing major, let them know that, too. There may be a way to change that behavior or may a fix for that is in an upcoming patch. Communication is key here.
Once you’ve gone through and tested multiple systems, you’ll usually find that at least two or three of them will meet your needs. How do you determine which LMS is right for you? We’ll talk about that next time.
This was originally published on LinkedIn on January 21, 2020.
