Learning Nurse Newsletter - January 2018
Welcome to the January 2018 edition of the Learning Nurse Newsletter. As we enter our 10th year of operation, we thought it would be appropriate to update you about is what happening at LearningNurse.org. (More timely and frequent updates are available on Twitter.)
Developing Effective Online Quizzes: Lessons learned
With the Learning Nurse website entering its tenth year of operation, we thought it would be appropriate to share our experiences about its most popular component – the nursing education quizzes. Some statistics on the utilization of our nursing quizzes are provided later in this newsletter.
Applications of tests and quizzes
Education and training tests and quizzes are very useful and have numerous applications.
- Self-assessment of knowledge. Research shows that most professionals have an exaggerated perception of their level of knowledge. Quizzes provide a great reality check.
- Diagnostic tools. We have used tests to develop self-scoring personality diagnostic tests on the topics of conflict resolution, empathy, listening, learning styles, time management and work stress.
- Discovery learning. Quizzes that provide immediate feedback are more effective for learning and long-term retention than reading and memorization.
- Challenge exams. It is inefficient to have learners spend time on material that they already know. Tests can be used to identify those areas where individuals need to learn new knowledge and skills.
- Screening. Employers often test perspective candidates to ensure that nurses have the basic and core knowledge essential to provide effective and safe nursing care. For example, one area of concern is the ability to accurately calculate medication dosages.
- Knowledge checks. We routinely use quizzes to check knowledge comprehension and retention at the end of our e-Learning modules.
- Final exams. For most of the online courses we develop for ourselves and clients, we now include a final exam. The final exam results (usually with a Certificate) are used by nurses to meet requirements for their professional continuing education / competence programs.
- Compliance. Many organizations require compliance testing, monitoring and reporting. These can include such health related skills as CPR and First Aid to more general topics such as sexual harassment and emergency preparedness.
Components of an effective testing system
An effective testing development and management software system should have the following features and capabilities.
- Easy to use by subject matter experts (SMS) with little or no technical programming skills.
- Have a full range and variety of questions types, e.g., multiple choice / response, true-false, matching, fill-in-the-blank, hot spots, etc.
- Provide flexible scoring and grading options.
- Should be able to incorporate multimedia such as narration, images and videos.
- Must have answer feedback options such as no feedback, feedback by question, or feedback by response option.
- Have test items banks with options for item selection and random presentation of questions.
- Have flexible delivery options such as website, learning management system, CD-ROM and print.
- Be available on different delivery platforms such as desktop, laptop and mobile devices (Apple, Android and Windows).
- Should be compatible with evolving browsers, e.g., HTML5 is now preferred as Adobe Flash is being phased out.
- Tracking and reporting of performance and results including sending the results to a LMS or to a database and/or e-mailing results to instructors and learners.
- For formal education use, learner registration, management and administration features are a must.
Benefits of online quizzes
For Learning Nurse, our online quizzes offer us many benefits.
- Self-operating. Once our quizzes were set up, they run themselves.
- Availability. The quizzes are available (and utilized) 24/7 by users around the globe.
- Scalability. There are few limits as to the number of users that we can accommodate.
- Updateability. It is relatively easy to update and maintain our quizzes.
- Affordability. Since we create our own quizzes, cost is not an issue.
- Cost-effectiveness. For other organizations, development and production costs can be spread over thousands of users.
- Individualization. Nurses can select which topics to learn at a time and place convenient to them.
- Pacing. Learners can progress at a rate that best suits their style and circumstances.
- Feedback. We provide instant feedback as this has been shown to be most effective for long-term memory retention.
- Engagement. Interactive quizzes require active learning which is interesting and engaging.
- Safety and comfort. Our learners control who sees their results thus ensuring their confidentiality and privacy.
- Randomization. Questions and responses are randomly selected and presented from a larger pool of questions.
- Results data. Our quiz results are tracked and periodically analyzed to provide insights into learning needs, knowledge levels and quiz effectiveness (see our quiz data analysis below).
Challenges with online quizzes
However, we discovered that online quizzes have challenges as well.
- Best for knowledge. Quizzes are most effective in assessing and teaching knowledge; they are less effective for the psychomotor and affective domains.
- No instructor. Instructors / teachers play important roles in motivating, keeping learners on track, and providing assistance and support.
- Less group interaction. Online quizzes are not great at encouraging learning from others.
- Requires discipline. Self-directed, online learning requires greater individual motivation and discipline.
- Software updates. Every time there is a minor or major quiz software release, the quizzes must be updated and republished.
- Evolving technologies. A perfect example was our need to totally change our Adobe Flash-based quizzes to HTML5 as Flash is being deprecated.
- Diversity. Different browsers, operating systems, devices and screen sizes are a constant challenge in getting our quizzes to work across diverse platforms and applications.
- Mobile devices. These present special problems with compatibility, iOS / Android updates, printing certificates and sending results to the tracking database table.
- Client support. It is a challenge to provide support for a large and diverse clientele.
Quiz development and management
Here is how we develop and manage our Learning Nurse quizzes.
- All of our quizzes are created and maintained in Microsoft Word. We discovered that it is NOT a good idea to create and enter the questions directly into the quiz program. With the quiz in Word, it is easier to switch to new quiz software if necessary (e.g., Moodle).
- The quiz questions, answer options and feedback are copied and pasted from Word into the quiz software program. (We currently use iSpring’s Quiz Maker software).
- For our quizzes, we typically use multiple choice/ response, true/false and matching type of questions.
- For topics such as anatomy, we often add the appropriate images that we purchase from Bigstockphoto.com.
- Our typical quiz is 25 questions long; the questions are randomly selected from pools of up to 100 items.
- Quiz questions are submitted one at a time with immediate feedback (often detailed).
- We discovered that our quizzes were being used for course assignments, CE credits and screening tests. This required the option to enter the learner’s name if they wanted it on the quiz results, or the certificate. Otherwise, learners can enter “Guest” or any other name.
- We have also added an option to include the course or instructor name, if required for assignments.
- Our most recent feature addition was the ability to e-mail the quiz results to any address provided by the quiz taker.
- We do not set time limits for our learning quizzes; learners can take as long as they wish.
- The quiz results are sent to a database table that tracks the following:
- Quiz name
- User name (optional)
- Course name (optional)
- Passing score (usually 80%)
- Passing points
- Earned points
- Total points
- Score per cent
- Pass or fail
- Time taken in seconds
- Date and time (obtained from server)
- IP address (obtained from server for location and security purposes)
- The types of analyses we do on this data are:
- Most popular and least popular quizzes
- Average scores, highest scores and lowest score (crude measure or learning needs and competence)
- Time taken (crude measure of difficulty of each quiz)
- Daily, weekly and monthly traffic
- Location of learners
- We can NOT do item analyses with the data we collect.
- Final exams are created by importing module quizzes into distinct question categories; number of questions selected from each category are based on perceived importance of the topic.
- It is possible to set value points for each question; we usually don’t do so.
- We use music for the title page, instructions and results slide as well as audio sounds for slide transition and correct/wrong answers.
- We create the “perfect” template and question specs, and then duplicate the slides and enter the new information.
- We use the slide view for previewing correct formatting and layout.
- We publish the quizzes for HTML5 and post them on the Learning Nurse website.
Here some things we have learned from our Learning Nurse quizzes.
- Quizzes are very popular; we currently average about 400 completions per day.
- Research has shown that quizzes used as discovery learning are four time more effective for learning content and long-term retention is better.
- Quizzes are a good choice for dull or boring topics such as anatomy or medical terms.
- Background music is distracting to most, so we only use it for title and results pages. We have had complaints about music on the Instructions page.
- We have experimented with narrated quizzes; they have NOT worked for us.
- True/false questions work well for interactive learning; they are particularly effective when detailed feedback is provided.
- One size does not fit all. After some feedback, we now offer 25, 50, 75 and ALL questions quizzes. Nurses studying for licensure exam wanted to do all of the questions, rather than a smaller sample of them.
Here are a couple of additional resources on our quizzes.
Learning Nurse Users
Analyzing the Learning Nurse website statistics from January 1 to December 31, 2017 shows the following:
- 477,000 unique visitors (IPs), an increase of 28% over 2016
- 764,000 visits with 7.4 million page views and 2,497 GB of traffic
For 2016, the numbers were:
- 342,000 unique visitors (IPs)
- 568,000 visits with 6.1 million page views and 1,406 GB of traffic
The geographical locations of Learning Nurse visitors in 2017 were approximately as follows:
- 35% from the United States
- 11% from Canada
- 5% from Great Britain
- 5% from Australia
- 2% from India
- 2% from Philippines
- 1% from New Zealand
- 1% from Saudi Arabia
Other top countries were Thailand, United Arab Emirates, South Africa and Ireland.
The Learning Nurse website has a loyal following with many repeat visits. Statistics for 2017 show that:
- Some 67% of visitors come directly from a bookmark or URL entry; this is the same as the previous year
- 22% come to the website from a search engine – Google, Yahoo and Bing; “nursing quiz(zes)”, “learningnurse” and “nursing games” are the most common search terms
- Some 8% of traffic comes from links on other websites with Pinterest and Facebook being the top referrers; this percentage has decreased over the past few years
- About 48% of visitors this year have bookmarked the website.
Despite the fact that our visitors are increasing every month, we are still a ways to go to reach the approximately 6 million English speaking nurses on this planet.
Nursing Educational Games Results
Over the past two years of 2016 and 2017, we were able to collect the results on 262,388 educational nursing games. Here are some analyses of these results.
Let’s start by examining which nursing games were most, and least, popular. Here were the 20 most played games with N and Mean score for each.
- Anatomy & Physiology I (34,333 – 56%)
- Abdominal Pain (17,751 – 45%)
- Urinary System (10,506 – 56%)
- Nervous System (8,887 – 47%)
- Anatomy & Physiology II (8,632 – 56%)
- Integumentary System (7,364 – 51%)
- Diabetes I (7,219 – 58%)
- Metric Conversions (6,302 – 62%)
- Blood Components (5,417 – 54%)
- Patient Assessment I (5,054 – 54%)
- Pediatric Nursing (4,892 – 53%)
- Body Fluids Disorders (4,858 – 50%)
- Chest Pain (4,483 – 57%)
- Mental Health Disorders (4,321 – 58%)
- Diabetes II (4,296 – 66%)
- Seniors and Meds (4,243 – 66%)
- Tablet Dosage Calculations (4,012 – 58%)
- Lymphatic System (3,588 – 48%)
- Sexually Transmitted Diseases (3,445 – 58%)
- Bacteria (3,348 – 49%)
All of these games are of the SuperNurse (SN, multiple choice) type.
The 10 least played games were (with N and Mean scores):
- Microscopic Analyses (3 – 73%)
- Diagnostic Tests II (10 – 68%)
- Blood Tests II (16 – 65%)
- Stool (32 – 54%)
- Fluid Analyses Tests (44 – 72%)
- Diagnostic Tests I (50 – 68%)
- Nursing Informatics (54 – 59%)
- Meds Classifications (63 – 57%)
- Diseases II (70 – 73%)
- Diagnostic Tests (91 – 92%)
All of these games are StoryLine (SL) drag and drop games.
Next, an analysis of the games’ scores was done. Here are the averages for the top 20 scoring games with N and Mean score for each.
- Oral Dosages I (269 – 98%)
- Injection Dosages (208 – 96%)
- Medication Abbreviations (376 – 95%)
- IV Flow Rates (200 – 94%)
- Metric Conversions (249 – 94%)
- Word Roots I (229 – 94%)
- Oral Dosages II (181 – 93%)
- Diagnostic Tests (91 – 91%)
- Blood Tests (190 – 91%)
- Cardiology (2,638 – 83%)
- Tablet Dosages I (534 – 82%)
- Metric Measurements (174 – 82%)
- Diabetes Terms (1,047 – 82%)
- Brain Terms (865 – 81%)
- Body Directions (928 – 80%)
- Drug Abuse Terms (324 – 80%)
- Endocrine Terms (3,124 – 78%)
- Med Rounds I (379 – 78%)
- Anatomy Terms III (394 – 78%)
- Cancer Types (381 – 78%)
These nursing games are SL (drag and drop) except for Cardiology, Diabetes Terms, Brain Terms, Body Directions, Drug Abuse Terms, Endocrine Terms, Anatomy Terms III and Cancer Types which are word guess (WG) types.
Now here are the 10 games with the lowest scoring games with N and Means scores for each.
- Infusion Rate Calculations (1,967 – 21%)
- Birth Defects (184 – 31%)
- Nutrition Requirements (748 – 33%)
- Diarrhea (836 – 38%)
- Medication Practice (1,514 – 38%)
- Fluid Dosage Calculations (2,810 – 40%)
- Nutrition Disorders (752 – 42%)
- Skeletal System I (1334 – 44%)
- Abdominal Pain (17,751 – 45%)
- Cancer Risks (1,765 – 46%)
All of these games are SuperNurse (SN) except for Birth Defects which is a drag and drop (SL) game.
Beginning in mid-2016, we completely redid our quizzes to provide greater functionality and tracking capability. Over the period of 2016 and 2017 we were able to capture the results of 151,114 quizzes. Here are some analyses of the results.
The 20 most popular nursing quizzes were (with N and Mean Score):
- Anatomy Terms I (8,356 – 73%)
- Cardiovascular Terms (5,498 – 70%)
- Anatomy and Physiology (3,227 – 63%)
- Health Assessments (3,026 – 63%)
- Fluids and Electrolyte Disorders (2,860 – 61%)
- Digestive System Terms (2,734 – 75%)
- Diabetes I (2,700 – 64%)
- Disease Terms I (2,507 – 72%)
- Legal Risks (2,403 – 77%)
- Anatomy Terms II (2,335 – 76%)
- Respiratory Systems Terms (2,287 – 76%)
- Mental Health Disorders (2,274 – 68%)
- Cardiovascular Disorders (2,172 – 71%)
- Cardiovascular System (2,093 – 68%)
- Medical Terms I (2,089 – 70%)
- Pharmacology Terms (2,025 – 66%)
- Endocrine System Terms (2,012 – 73%)
- Tablet Dosage Calculations (1,897 – 90%)
- Nursing Ethics (1,755 – 71%)
- Body Structure Terms (1,743 – 66%)
The 10 least popular quizzes were:
- SI Units of Measurement (43 – 53%)
- Dizziness (52 – 65%)
- Diagnostic Tests (57 – 51%)
- Limb Pain (61 – 64%)
- Diarrhea (86 – 49%)
- Viruses (86 – 57%)
- Coughing (87 – 66%)
- Fatigue (96 – 74%)
- Headaches (104 – 65%)
- Helminths (105 –47%)
The quiz data was also analyzed to determine average / mean scores. The 20 quizzes with the highest average scores were:
- Injection Dosages Calculations (1,097 – 92%)
- Metric Conversions (1,153 – 91%)
- Fluid Dosages Calculations (1,535 – 90%)
- Tablet Dosages Calculations (2,145 – 89%)
- Infusion Rate Calculations (785 – 85%)
- Drug Classifications (1,422 – 83%)
- Digestive and Respiratory System (708 – 82%)
- Mobility and Falls (815 – 80%)
- End-of-Life Care (764 – 78%)
- Reproductive and Urinary System (464 – 78%)
- Nervous System Terms (1,428 – 77%)
- Sleep Disorders (240 – 77%)
- Legal Risks (2,403 – 77%)
- Safe Medication Principles (1,945 – 77%)
- Antidotes (231 – 77%)
- Hygiene & Infection (1,022 – 76%)
- Respiratory System Terms (2,287 – 76%)
- Medical Disorders (561 – 76%)
- Musculoskeletal System (1,500 – 75%)
- Ear Pain (302 – 75%)
The 10 quizzes with the lowest average scores were:
- Critical Diagnostic Values (111 – 40%)
- Cancer Risks (229 – 45%)
- Helminths (105 – 47%)
- Nutritional Disorders (364 – 49%)
- Diarrhea (86 – 49%)
- Antibacterial Drugs (149 – 50%)
- Dangerous Drug Combinations (210 – 50%)
- Diagnostic Tests (57 – 50%)
- Fungi (161 – 52%)
- Personal Protective (451 –52%)
It is up to the nursing profession to determine whether the lack of knowledge in these latter areas should be of concern.
About Learning Nurse
The Learning Nurse Resource Network (LearningNurse.org) was created by Steppingstones Partnership, Inc. and went online in April 2008. The purpose of this nursing professional development resource is to make available free, informal, accessible and convenient learning opportunities for nurses. Because of the shortage of nurses in many facilities, it is very difficult to get away for traditional courses and workshops. This is particularly true in smaller and rural areas where it nearly impossible to obtain leave and funding to attend professional development events.
Although Learning Nurse was originally designed for all levels of practicing nurses, it has become popular with nursing students, graduate students and nurses returning from leaves. Many nursing educational programs around the world now use our resources to supplement their regular programs. Nurses take time during breaks in their night shift to review and refresh their nursing knowledge and skills using our quizzes, educational games and e-learning modules.
That’s it for now. If you have any questions, comments or suggestions, please let me know. Thanks for your continuing interest and support.
Webmaster and Editor
Note: Permission is granted to reprint all or part of the information in this newsletter provided that the source – LearningNurse.org – is credited.