All the vacuums on CNET’s best lists are tested and evaluated in our state-of-the-art Test Labs. CNET Testing Labs go beyond product specifications to test in real-world conditions with real-life messes. Our testing lab is set up specifically for evaluating robot vacuums, which is a grueling test environment seen nowhere else.
Here’s a breakdown of how we measure vacuum performance:
Differences in scoring
At CNET we diligently test as many products as possible to ensure that we are able to give you the best advice. When we put this to work on vacuums, there are inherently going to be differences in the testing methodologies. For cordless vacuums, handling and comfort are important factors as you physically hold the device in your hand to use it. For robot vacuums, that isn’t a contributing factor, but features are. So, this is why you’ll notice a difference in the charts below.
Methodology for cordless vacuums
| Subrating category | Weight | What we looked for |
|---|---|---|
| Performance | 30% | Performance score extrapolated from the average sand score and average rice score on hard flooring, low pile carpets and midpile carpets. |
| Value/Price | 25% | Retail price rating considering all other features compared to the value of features offered. |
| Features | 20% | Usefulness of extra features (i.e. self-emptying station, handheld conversion, additional tools, multiple batteries, NAV tech, mopping) |
| Handling & Comfort | 15% | Overall comfort and handling of the product. (i.e. heavy versus lightweight, comfortable grip handle). |
| Ease of use | 10% | Overall user experience with set up, functionality, smartphone app accessibility, smart home features. |
Methodology for robot vacuums
| Subrating category | Weight | What we looked for |
|---|---|---|
| Performance | 30% | Performance score extrapolated from the average sand score and average rice score on hard flooring, low pile carpets and midpile carpets. |
| Price | 25% | Retail price rating considering all other features compared to the value of features offered. |
| Features | 15% | Usefulness of extra features (i.e. self-emptying station, handheld conversion, additional tools, multiple batteries, nav tech, mopping) |
| Runtime | 20% | Navigation efficiency score (1-10), based on time taken to complete a full cleaning cycle inside CNET labs custom robot vacuum nav testing room. |
| Ease of use | 10% | Overall user experience with setup, functionality, smartphone app accessibility, smart home features. |
Robot vacuum testing
Test 1: Floor coverage ability
The first test is to figure out how well a robot covers the floor while it’s cleaning. We built an industry-standard testing room as specified by the International Electrotechnical Commission just for this purpose. The IEC is an international standards body responsible for managing robot vacuum testing procedures, among other things, for vacuum manufacturers.
Inside this room are objects designed to simulate typical obstacles a robot vac encounters for navigation as it cleans. These obstacles include wall edges, table and chair legs, couches, other furniture and so on, plus bare tile and hardwood floors, as well as carpet. We mount LED lights to the top of each vacuum cleaner. The dimensions of the lights correspond to the measured nozzle width of each particular robot vacuum we test.
As robots move through the room while cleaning, an overhead camera captures a long-exposure image of the entire room in low light. That photo will then have a light trail, created by the LEDs, that shows the exact areas where the robot traveled (and its nozzle position) during its runtime. We can also see areas of the floor where the vacuum may have missed or gotten stuck. You can see the navigation results of all the robot vacuums in our test group in the gallery below.
Test 2: Debris pick up performance
The second type of test reveals exactly how much physical debris a vacuum is able to pick up off of the floor. To mimic dirt of small particle size, we use a mixture of play sand and landscaping sand. For bigger particle soil, we use grains of uncooked black rice. Robots then run in straight line mode across three types of flooring (low-pile carpet, medium-pile carpet and hardwood bare floors).
We control for the specific nozzle width of each vacuum too. We constructed an adjustable tool to soil our test floors. It lets us lay down a strip of a precise area of soil to match the nozzle dimensions for every robot. The mass of soil isn’t chosen at random either. We measure a proportional amount that’s related to the flooring material, type of debris, plus each vacuum’s nozzle width.
We conduct three cleaning runs (at minimum) on each floor type. We also perform cleaning tests with sand and rice separately. That comes to at least 18 tests per robot vacuum. We weigh the robot’s dustbin both before and after each run. From there we can calculate the percentage of debris pickup for every cleaning run and the average amount of soil a machine manages to remove. Additionally we run anecdotal (visual) pet hair tests for each robot, on all three floor types.
Our rice-based, medium-size particle test didn’t show enough differentiation between each cleaner, which says they can all handle larger particles without trouble. For fur removal for pet owners, we judged anecdotally.
Cordless stick vacuums testing
Just like in robot tests, the width of the test bed is proportional to the vacuum’s nozzle width. We measure this width ourselves. We also use nozzle width, plus the flooring type, to calculate the soil density for each test, per IEC guidelines. We use the same soil types here as well; sand, rice and pet hair. We perform three runs (at minimum) on each floor type. We also test suction power with sand and rice separately. That comes to at least 18 tests per vacuum. We weigh the vacuum’s dust bin both before and after each run.
From there we can calculate the percentage of dirt and debris pickup for every run and the average amount of soil a vacuum manages to remove. Additionally, we run anecdotal (visual) pet hair tests for each vacuum, on all three floor types to help us select the best cordless vacuum.
Read the full article here