Crowdsourcing versus mobile network testing

The need for an objective evaluation of mobile network quality and performance drives two fundamentally different approaches, crowdsourcing and mobile network testing. The crowdsourcing market is highly saturated. More than 20 companies have a significant focus on crowdsourcing mobile network measurements. To get measurement results, they use different concepts that serve different purposes. This application card describes the structure of the crowdsourcing space and analyzes the pros and cons of both approaches.

Crowdsourcing versus mobile network testing
Crowdsourcing versus mobile network testing
/

Your task

Mobile network operators and regulators have a growing interest in getting deeper insights into network quality and competition in specific markets. This interest is driven by three key requirements:

  • Benchmarking – provides a view of the relative performance of different mobile networks
  • Operations optimization – improves operational efficiency for the network operator (including customer service, service assurance and network engineering)
  • Marketing/customer experience management (CEM) – assesses end-user experience of network quality

Crowdsourcing and mobile network testing (MNT) are two fundamentally different approaches to addressing these requirements. Mobile network operators and regulators need to evaluate these two approaches and map them against their needs. This application card investigates the pros and cons of both concepts.

Crowdsourcing solution

Crowdsourcing solutions can be grouped into three categories:

  • Syndication – allows unrelated apps to integrate a measurement SDK for monetary compensation (similar to in-app ad placements). Often end-users do not know that data is collected in the background and sent to cloud storage for further analysis, which can be a legal gray area.
  • Operator app – embeds a measurement SDK in an operator branded app
  • Own app – places a branded app in app stores specifically targeting users interested in network performance

Crowdsourcing market segmentation, based on company positioning

Crowdsourcing market segmentation, based on company positioning
Crowdsourcing market segmentation, based on company positioning

Figure 1: Segmentation of crowdsourcing solutions by requirement and application category (examples, non-exhaustive).

/
Figure 1: Segmentation of crowdsourcing solutions by requirement and application category (examples, non-exhaustive).

Crowdsourcing analysis

Pros Cons
  • Crowdsourcing data is collected continuously and autonomously without a dedicated measurement campaign.
  • Massive amount of data provides very high resolution analysis. You get data from wherever people are. But that does not ensure complete geographical coverage.
  • Mobile network coverage and quality statistics are available across a country or region.
  • Identification of sites, locations and operators with highest data traffic can offer insights for new site planning or capacity expansion.
  • Provides an understanding of how the available spectrum is used in different geographical areas.
  • Massive amount of data can be filtered and used for drill-down analysis according to various criteria (operators, location, time, etc.).
  • Conditions of crowdsourcing data collection are unpredictable and mostly unknown:
    • Temperature: smartphone in the user’s pocket is heated to body temperature; heating due to applications running in parallel
    • Location: smartphone in the basement or 1st or 10th or 20th floor of a house
    • Device dependency: older phones with limited UE capabilities in use for long time – network performance measurement might be affected by limited UE capabilities
    • Customer traffic plan: data volume might be expended and throughputs throttled by the network
  • Measurements are not reproducible (crowdsourcing analysis can be repeated or done on a newer data set, but it cannot be reproduced). If there is a dispute on a reported event, it cannot be reproduced for validation.
  • Crowdsourcing data cannot be used for network optimization. Limited or no information on root causes.
  • Crowdsourcing solutions mostly follow a proprietary approach with an undisclosed scoring method for the results. The scores or results from different solutions are not comparable with each other.
  • Crowdsourcing solutions typically limit their active tests to the following quality KPIs: UL/DL throughput, average latency, average jitter and packet loss.
  • Smartphone operating systems are increasingly restrictive and limit the available parameters that crowdsourcing apps (running in the background) may use.
  • Dedicated crowdsourcing apps typically show a negative bias (technically savvy subscribers in particular trigger these tests when they experience a network issue).

Mobile network testing solution

Drive and walk tests are dedicated mobile network test concepts. Dedicated smartphones (commercial UEs) and network scanners are used in a controlled test setup (temperature and RF shielding as far as possible) and perform a set of test cases reflecting end-user behavior. Voice services (telephony) and data services (data transfer, video streaming, social media and web browsing) are the most commonly used services and account for the vast majority of traffic volume in mobile networks.

ETSI TR 103 559 describes best practices for robust network QoS benchmark testing and scoring. The technical report provides a recommendation for how to test the quality of mobile networks based on end-user perception.

The Rohde & Schwarz network performance score implements the ETSI method.

Applications are tested in line with the following three categories:

  • Accessibility – service availability, e.g. dropped calls
  • Waiting time – call setup time
  • Media quality – picture quality or speech quality

The results are mapped to a point scale and an aggregated score is generated reflecting end-user perceived quality (a streamed video that cannot be started is more severe than slightly reduced video quality and gets a higher weighting).

For further details and explanations, go to:
www.rohde-schwarz.com/nps

Highway test campaign

Highway test campaign
Highway test campaign

Figure 2: Rohde & Schwarz network performance score (NPS) covers the full range of end-user applications.

/
Figure 2: Rohde & Schwarz network performance score (NPS) covers the full range of end-user applications.

Mobile network testing analysis

Pros Cons
  • Data is collected under defined conditions and in a controlled test setup (temperature, RF shielding). You always know under which conditions the data is collected (outdoor, indoor, measurement device position, temperature, etc.).
  • Measurement campaigns are planned with latest smartphones and UE capabilities to ensure that the performance measurement is not impacted by UE deficiencies. All applications that subscribers have running in the background contribute to the overall assessment of network quality.
  • Using defined and harmonized test methods makes mobile network testing comparable and reproducible
  • Measurement campaigns are clearly defined (routes, teams, times, measurement setups, etc.) and can be easily repeated after 3, 6 or 12 months to measure performance improvement over time.
  • The test scope is much wider compared to crowdsourcing. Professional mobile network testing solutions can also calculate advanced metrics such as mean opinion score (MOS) for voice and video content.
  • The network performance score using the ETSI method provides insights for network optimization due to the standardized measurement and scoring approach. All details are available for further analysis. When problems are reported, context is available (RF conditions, detailed L3 messages from the network and applications) for troubleshooting purposes.
  • Its comparability due to the harmonized test method means that mobile network testing can be used to compare Open RAN (O-RAN) with a single vendor RAN architecture.
  • Agreements with mobile device and chipset manufacturers are in place to overcome the limitation of restrictive smartphones operating systems.
  • Dedicated measurement campaigns have to be planned and executed which requires effort. They are time-consuming and expensive compared to crowdsourcing (although the crowdsourcing reports are also not available for free).
  • Network performance results are only available for areas where the data has been collected. A coverage map for a whole country cannot efficiently be created based on mobile network testing.

Recommendations

Analysis and comparison of the pros and cons of both approaches shows that crowdsourcing and mobile network testing are more complementary than competitive and that one can never replace the other.

Depending on the target or the requirement, one approach has advantages over the other. Here are some examples:

  • Network quality benchmarking/ranking of operator networks in a country: For a rough overview, crowdsourcing data might be sufficient. But if the aim is to make a fair comparison and draw solid conclusions, a comparable test method like ETSI TR 103 559 for mobile network testing is highly recommended.
  • Coverage map of a whole country: If a coverage map for a whole country is the goal, crowdsourcing is the best solution. Basic RF data such as reference signal received power (RSRP) is sufficient in most cases, although discussions are ongoing in the European Conference of Postal and Telecommunications Administrations (CEPT) to redefine the term “coverage”, in particular in light of 5G beamforming. Mobile network testing cannot cover a whole country with each and every street in every city, town, village at a reasonable cost.
  • Network optimization: If network optimization is to be based on data, the collected data must be reliable, collected under known, defined conditions and be reproducible. The data also needs to be available with context (RF conditions, detailed L3 messages from the network and applications) in order to match identified problems with RAN events so that root causes for these problems can be identified. Only mobile network testing fulfills these requirements, whereas crowdsourcing data does not provide this context.
  • Network usage analysis: To identify locations or geographical areas with high data and spectrum use, crowdsourcing is a viable solution because mass data can be analyzed and the reliability of the data is not critical for this purpose.
  • Crowdsourcing can be a good complement to a traditional drive test: Crowdsourcing can be used to detect areas with low coverage or network problems. Drive test teams can then be dispatched to collect data. Crowdsourcing can help justify national benchmarking campaigns (e.g. NPS campaigns), such as campaign and route planning to determine whether a campaign is really covering the relevant parts of the country.

Additional information

For more information on the mobile network testing solutions for benchmarking discussed in this application card, please contact your Rohde & Schwarz sales representative or visit our website:
www.rohde-schwarz.com/mnt/network-benchmarking

Download