Since CNET started testing robot vacuums, in the early days of the first Roomba, we’ve worked hard to craft comprehensive testing procedures to evaluate every robot vacuum for the things that matter for buyers. Our lab experts have tested over 100 robot vacuums and counting, evaluating them for cleaning performance, navigation ability, object avoidance and more. Price and special features also play a role in our overall rating and buying advice.
Below, we’ve laid out the comprehensive set of lab tests we run on each and every robot vacuum that’s sent to the lab. Our tests are also always being tweaked and refined to generate the most consistent and reliable results, but the current testing procedures are accurate as of this writing. We’ve tested 24 robot vacuums under the new procedures, with more coming soon.
Awarding the highest performers
We weigh the results of our rigorous lab testing for cleaning performance, navigation and obstacle avoidance with our decades of editorial expertise reviewing robot vacuums. Then we evaluate the total consumer quality, considering the retail price and any extra features to assess the overall value of each robot vacuum.
There are a number of awards we may bestow on top performers. Editors’ Choice represents the best of the best products we’ve tested. These robot vacuums represent our top-tier choices for most people. We may offer an Editors’ Choice for a premium model and a different award for the best budget model.
We may also designate a Lab Award to a single product that achieved the highest performance on a specific, repeatable metric or benchmark meaningful to consumers. For example, the Dreame X50 won the Lab Award for best cleaning coverage, while the Mova V50 Ultra won for the best average dust pickup across flooring types. Finally, we will spotlight top-performing products in our best lists and make note of who they may be best for.
Don’t miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.
Our test room in Louisville lab
Each robot vacuum we consider for recommendation is put through its paces in our test lab in Louisville, Kentucky. The first trial is to figure out how well a robot covers the floor while cleaning, and how well it works to avoid obstacles. We built an industry-standard testing room, as specified by the International Electrotechnical Commission, just for this purpose. The IEC is an international standards body responsible for managing robot vacuum testing procedures, among other things, for vacuum manufacturers.
Inside this room are objects designed to simulate typical obstacles a robot vacuum encounters for navigation as it cleans. These obstacles include wall edges, table and chair legs, couches, loose cords, simulated animal poop, other furniture and so on. We test on bare tile and hardwood floors, as well as carpet.
Beyond that, we evaluate each robot vacuum’s ability to collect pet hair without getting clogged or leaving loose strands behind. We also consider its mopping capabilities and how well it navigates (fake) pet messes.
We’ll dive into more of each of the sections below.
Robot vacuum pickup and cleaning power
When it comes to vacuuming prowess, we want to know how effectively each robot cleans common crumbs and other debris. We also want to see how it fares against much smaller particles, such as dust, dirt and sand on all flooring types.
Sand and pet hair
We use play sand as an analog for finer particles, and we get pet hair from a groomer. In each case, we scatter a controlled amount across three test floor patches. We weigh each flooring section with a high-precision scale before and after each test run to see how much sand and other particles were removed. This lets us get individual pickup measurements for each floor type since we weigh the sand before adding it to the floor.
For pet hair, we don’t weigh the pet hair after the test, but we take before and after pictures of our test floor patches for subjective visual evaluation. The robot vacuums that remove the most pet hair from the carpeting and hard flooring are the ones we recommend for pet owners.
Floor types we test on
To test each robot vacuums ability on different floor types, we place sand and pet hair on three different floor types:
- Low-pile carpet: Typically shorter fibers and less plush than mid- or high-pile carpet. Robot vacuums have an easier time picking up from it (though not always). Low-pile and midpile carpet tend to be more challenging and our scores can vary from a high of 60% or more (excellent) to as low as 10%.
- Midpile carpet: Softer and more plush with taller fibers. It tends to be more challenging for robot vacuums (though again, not always). To ensure the sand is properly distributed, we use a brush to spread the sand out evenly. Robot vacuums especially differentiate themselves in midpile carpet performance, with our current best-scoring robot vacuum, the Mova V50 Ultra Complete, getting 47.54%.
- Hardwood floors: We use hardwood floors that can replicate any flat surface, such as tile or vinyl flooring. Generally, most robot vacuums score well on the hard flooring, with many of our top picks scoring above 80%.
How we measure pickup performance success
Before each test, we thoroughly empty the robot vacuum’s dust bin. We send it to clean the affected area, then finally measure the weight of whatever it picked up. That gives us a pickup percentage of the full amount.
From there, we repeat each four more times for a total of five tests and average the results. If there’s an obvious outlier, such as a test with an unusually high or low pickup percentage, we rerun the test cycles. We consider an unusual result that requires a retest to be approximately two standard deviations outside the average. We then calculate the individual average pickup for each flooring type and get an overall average.
This gives us a frame of reference for how effective a robot vacuum is at cleaning.
Our overall average pickup scores run the gamut, with the bottom performing robot vacuums scoring as low as 20%, to the best ones scoring above 60%.
Robot vacuum navigation skills: Heat mapping
Your robot vacuum will only clean your home as thoroughly as it can navigate it. The ideal cleaner will make easy work of finding its way from room to room and automatically avoiding obstacles along the way. All of this makes for proper, low-maintenance automated cleaning.
To test navigation efficiency, we record the robot vacuum in our test room with an overhead camera. Then we use video analysis software to assign it an average coverage percentage based on the total portion of the accessible room it’s observed to visit during three standard test runs. The heat map it generates is then analyzed to provide the result. The test room includes dummy furniture, low-hanging furniture, different flooring types (tile, hardwood and carpet), lamps, cables and other obstacles.
We also have boxes with wide holes cut out of them, between 3 and 4 inches tall, to test threshold clearance by simulating low-lying furniture. This helps us see if the robot vacuum is capable of getting under this furniture to clean. The ones that succeed are robot vacuums with thinner profiles or the ability to retract their sensor.
Heat map color legend:
- Blue/cyan: Very few passes — light coverage.
- Green: Occasional passes — low to moderate coverage.
- Yellow/orange: Frequent passes — high coverage.
Our two best performers can be seen below, the Mova V50 and Dreame X50 Ultra. Both offer a high percentage of cleaning coverage with very few missed spots. There’s some variation in how frequently they pass over certain parts of the room. The X50 Ultra focused heavily on the center by the simulated table arrangement, while the Mova V50 distributed its attention a bit more evenly. Notably, both were able to get under most pieces of furniture.
In contrast, our worst-scoring robot vacuum in this category was the Eufy E28, with numerous missed spots — especially around edges and corners — and light coverage in the outer parts of the room. It also missed both pieces of low clearance furniture. For you, that means this robot vacuum will provide less comprehensive cleaning coverage.
Object avoidance
We test object avoidance at the very end of our procedure by using six distinct common items placed in the environment to see if the robot vacuum is able to recognize and avoid them on its cleaning route. These items are placed at different degrees of approach, giving the robot vacuum more or less restrictive ways to navigate around the obstacle. These degrees of approach include pet waste at 360-degree, 180-degrees and 90-degrees using a lamp, pet toy and sock.
For the pet waste classifications, we distinguish between the fake pet waste based on the angle of the barriers around it that limit the possible approaches from the vacuum.
That means:
- The 90-degree trial is in the corner of the room, usually preventing the vacuum from accessing that portion of the room if it successfully avoids the obstacle.
- The 180-degree trial is along a wall, allowing more options to clean around it.
- The 360-degree trial is in the middle of the room and should be easy for the vacuum to clean around.
The more objects avoided, the better we consider the robot vacuum’s obstacle navigation. Currently, no robot vacuum has successfully avoided all six of our obstacles, with the best-performing ones avoiding five and the worst avoiding none of them. We also distinguish different failure states, noting that several robot vacuums suffer from a critical failure in obstacle avoidance, by sucking in the sock, as with the Roomba DustCompactor Combo 205, or running right over the pet waste, as with the Narwal Flow.
Noise level
We measure the noise level of each robot vacuum using a sound level meter to get dBA, which represents the average noise over time. We then use analytics to derive the LAeq, with an adjustment for human hearing. The data above reflects average noise levels in decibels recorded throughout the entire test while the unit was operating. A lower score indicates a quieter vacuum, which is generally preferable. The difference of 10dB from the top to the bottom of the chart is 10 times, meaning that the loudest vacuum is approximately 10 times as loud as the quietest vacuum on this chart. However, noise doesn’t weigh heavily in our scoring at this time, as all the robot vacuums fall within a similar range to the human ear. As we continue our testing of more vacuums, we’ll reevaluate our noise level testing.
Price, features and other factors
Price and features don’t require objective testing, but we do consider them in our overall evaluation and rating. Robot vacuum models can cost anywhere from $200 to nearly $2,000. We try to evaluate a robot vacuum according to its price range by comparing it to other models that cost about the same. That way, we can provide recommendations for a budget model and a high-end model with all the bells and whistles, as well as everything in between.
Features like mopping, self-emptying base stations and the usability of the app and software are all other things we consider. Several of these features, like base stations and mopping, are becoming more common across all prices. We’re currently in the process of crafting a test to evaluate, and we’re planning on implementing this test in the near future.
Correction, Jan. 9: An earlier version of this article misidentified the lab results for one of the robot vacuum models. We have updated it to reflect the correct model name, which is the Mova V50, our lab award winner for cleaning performance.
Read the full article here