Development of internet measurement principles for representation of measured provision of service (QoS-2)

ABSTRACT In modern world, technology plays a significant role. Upcoming services, demanding a specific level of service quality that should be guaranteed no matter what, will impose obligations to network performance and capacity. European strategy for broadband development prescribes a set of quality indicators that networks should correspond. But imposed obligations themselves don’t guarantee the persistent level of quality. European initiatives of geographical mapping of broadband access, which are developed in order to monitor the development of internet access services, propose guidance for gathering and representation of estimated QoS parameters at the so-called QoS-1 level, whereas monitoring of actual network performance on QoS-2 level and representation of real and objective internet quality indicators rests undefined. Besides ensuring that quality is described in meaningful and comparable manner the general measurement methodology that suits various purposes should be established. This research is aimed to develop principles of monitoring and objective representation of internet access service quality parameters in specific location on QoS-2 level, as well as to establish joint mechanism to obtain the quality of service data that would be appropriate for different needs. This research is directed to the mobile internet access service.


Introduction
European Electronic Communications Code states that Member states shall conduct a geographical survey of the reach of electronic communications networks capable of delivering broadband access. The aim of this requirement is to survey the current geographic reach of broadband networks in order to promote investments in the further network rollout, increase connectivity across Europe as well as provide information to all relevant authorities and citizens (European Parliament and the Council, Directive (EU) 2018/ 1972, 2018). Moreover, the results of geographical survey should be represented at the specified detail and territorial granularity. Based on the agreement among European member states, it is considered that, for fixed networks, the level of resolution should be the address, whereas for mobile networks, it should be (at least) a 100 × 100 m grid (or equivalent polygon) (BEREC, BoR (20) 42, 2020). Together with the aim of gigabit society and objective to provide access with broadband speeds at least 30 Mbps both indoor and outdoor to all European Union citizens by 2020 and at least 100 Mbps by 2025 (European Commission, COM/2016/0587, 2016European Parliament and the Council, Directive (EU) 2018/1972, collection of information on theoretical mobile coverage will not be enough. Anticipating the further mobile network development and emergence of critical QoS services, such mapping and quality monitoring becomes even more significant. Thereby, it is necessary to define common principles of quality assessment as well as establish mechanism of quality monitoring.

Related work
The general principles of internet quality assessment have been described in various documentations, recommendations and standards (BEREC, BoR (20) 42, 2020;ETSI, 2008;ITU, 2017). With the wider deployment of communication networks and growing demand for internet access service as a public utility, QoS requirements have been introduced (BEREC, BoR (20) 53, 2020; European Parliament and the Council, Directive (EU) 2018/1972, 2018). According to the European strategic objectives (European Commission, COM(2010)245, 2010European Commission, COM(2016) 587, 2016, all citizens of the Union should have access to the broadband speeds of not less than 30 Mbps by 2020 and not less than 100 Mbps, upgradable to Gigabit speed, by 2025. To achieve this objective, it is necessary to understand the actual penetration of broadband networks and their quality indicators. For this reason, Electronic Communications Code states the requirement for European member states to perform Geographic survey in their countries. Additionally, BEREC has specified the criteria of geographical network assessment in their guidelines (BEREC, BoR (20) 42, 2020). Among other things, guidelines specify the geographical scale in which internet service availability should be assessed, i.e. for fixed networks it is an address level, and for mobile networks -100 × 100 m grid. Service availability can be evaluated in three levels: QoS-1 or theoretical service availability based on operators' calculations; QoS-2 or actual service availability based on measurements that do not include end-user environment; QoS-3 or actual service availability based on measurements that include end-user environment (European Commission, 2014). While fixed internet access services are more stable and likely to ensure theoretical quality indicators in practice, those of mobile networks may differ vastly. In previous research studies, it was found out that speed indicators may change significantly in a different daytime period. In certain locations, this difference can be observed permanently, resulting in lower speed values during peak-hours. Therefore, theoretical quality information is not enough, and quality measurements should be performed. Lipenbergs et al. in their work have concluded that objective and comparable quality indicators can be assessed if end-user environment is excluded. Measurements should be performed in the same condition and similar manner in all locations and among different operators. According to Lipenbergs et al., reference point is of utmost importance to obtain objective and meaningful overall quality indicators. Taking that into account, it can be concluded that internet quality assessment on QoS-2 level gives the most comprehensive information about quality indicators. However, since speed indicators are likely to differ during the daytime period, it should be understood how to perform measurements in order to represent normally available speed values.

Practical measurements
In order to get quality indicators in each of 100 m sector throughout the country, the reference point in each of these sectors should be incorporated. Additionally, it should be defined what kind of traffic should be measured to objectively represent overall internet quality and how often the test measurements should be initiated in order to obtain meaningful results without unreasonable network occupation. European legislation as well as broadband development strategy states the value of broadband connection speed that should be achieved, but it does not refer to what kind of speed value it isminimum, average, achieved for a certain period of time or else how. In case this value is defined as minimum, the supervision became easier, since independently of daytime the speed value should correspond to certain minimum threshold. However, such measurements will not give a proper understanding of achievable speed values and broadband performance, especially in mobile networks, as the download speed of the latter can vary a lot during the day. Therefore, in this research it is considered that the normally available speed best represents day's average speed value and it could be considered as actual broadband performance at a specific location (Lipenbergs et al., Fall 2017;Lipenbergs et al., Spring 2017;Smirnova (Vagale) et al., 2018).

Location of measurement agent
Since internet access service in the existing networks is used for different kinds of services, testing the end-user generated traffic would not give comprehensible results. To get comparable and meaningful measurement data, tests should be performed with uniformed methodology; therefore, the use of injected traffic to the same reference point is needed (Lipenbergs et al., 2016a;Smirnova (Vagale) et al., 2019). Speed values can vary vastly during the day, and results got at different timeslots differ a lot. Thus to correspond to aforementioned principles measurement agent should be incorporated in every 100-m sector and located there for a long term. Such approach seems unrealistic as it would require enormous resources and would be expensive and difficult to implement. Even in the case of locating probes on end-user equipment, the measurements cannot be done permanently. Therefore, it is necessary to define the amount of measurements and test approach which would deliver the most objective results (ETSI, 2008). It is even more important in the case of test drives, which is the most probable way of how to perform internet quality monitoring.
For the overall and comparable quality, indicators' end-user environment should be excluded (Lipenbergs et al., 2016b;Smirnova (Vagale) et al., 2019). Measurements performed only by equipment with similar technical characteristics should be evaluated. It means that measurements should be done whether on device directly connected to mobile network, whether on probe connected to mobile router. Moreover, if performed on end-user equipment, those should be passive measurements not requiring end-user action to start the test. Otherwise, no testing frequency can be ensured. Taking into account the purpose of such measurements it would not be sufficient and representative if measurements would be performed without any system and periodicity.

Measurement amount and periodicity
Only general guidance regarding the temporal and spatial distribution of measurements is proposed. Basically, these concerns approach for different kinds of networks (fixed and mobile) and the variance in quality values that can be observed in those networks. In any case, it is presumed that the measurement results should objectively reflect the QoS, as perceived by the user (ETSI, 2008;ITU, 2017). The harmonized spatial distribution and general measurement approach are already proposed (BEREC, BoR (20) 42, 2020;ETSI, 2008), whereas none of recommendations are given regarding temporal distribution and measurement sample size at specific location.
The spatial distribution at least 100 m polygon should be enough to represent mobile internet quality in certain area. Additionally, if quality indicators vary significantly between the nearest 100-m polygons, it can serve as an indicator for mobile operators to assess probable reasons of this variance and consider the changes in antenna location or its parameters. To define temporal distribution and the minimum amount of the tests, long-term measurements were performed and evaluated. Measurements were done with the measurement system that is based on Visualware Inc. software. The system is a webbased measurement tool which is elaborated by customizing Visualware Inc. measurement modules. To measure different internet quality parameters, such as speed, latency, jitter and packet loss, MySpeed and MyVoip module algorithms were implemented (Visualware Inc.). Measurement server's switch is connected to four national internet exchange point (IXP) switches, thus allowing internet quality assessment between network termination point (4G LTE mobile routers) and IXP. Such system's architecture ensures indiscriminate physical and logical symmetry performing tests in different operator's networks. Tests were done in the same conditions in three mobile operators' networks. Tests were done in totally 44 different locations. In each location, tests were performed for at least 1 week, repeating 4 measurement sets each 5 min for 24 h. Totally more than 427,000 measurements were done. It gives an adequate number of measurements to evaluate deviation in the results and determine sufficient temporal distribution of measurements.

Measurement data analysis
By aggregating measurements, the time periods with the greatest fluctuation between actual speed values and the day's average value can be observed (Figure 1). It can be observed that around midday the actual download speed value is close to the day's average. To better evaluate the daytime with the lowest variance in actual and mean values, the amount of measurements with deviation less than 2.5 Mbps was calculated. Figure 2 shows that in the time period from 9:00 to 15:00, the highest frequency of measurements with deviation less than 2.5 Mbps can be observed.
The measurement results were evaluated based on a day type, assuming that internet usage habits could differ on holidays. No significant difference in speed values deviation on holidays and working days was observed.
After evaluating each measurement deviation from day's average value, it was concluded that even in time period with the smallest number of variations observed, every single measurement can differ from population's mean value (i.e. day's mean download speed) in a wide range. This indicates that in order to get accurate results and represent realistic download speed value, the multiple measurements should be performed. To verify this assumption, the variance of an hour's average speed value relatively to the day's average speed value was tested.
As it can be observed in Figure 3, by performing multiple measurements in 1-h time period the probability of having more accurate measurement results increases almost twice. Still there is a possibility to obtain results that does not properly represent the mean value in a specific location, but 95% of download speed measurement results put into the ∼10 Mbps deviation range, and 68%in 5 Mbps deviation range.  To ensure that measurement result has the smallest possible deviation from the day's average values, it is necessary to determine the minimum sample size. According to that and considering the available resources dedicated to this task, sample size should be chosen large enough to represent the population's mean value meanwhile small enough to not wasting resources.
In this study, the case of an expert performing tests to explore the quality parameters of mobile broadband connection is considered. In the case of test drives an expert, who is performing the measurements can be present at specific location for a short period of time, foreseen as not exceeding 1-h period. For this purpose, it was decided to test the variation of download speed characteristics by simulating the above-mentioned measurement conditions. Simulation code was developed in Algorithm 1. The simulation chooses daytime hour from predefined time ranges, sequentially chooses predefined number of measurements (starting from 1 to 19 with step 3) from the existing measurement data, grouped by operator, measurement location and week day, and aggregates speed parameter values. This way it is possible to estimate the extent of deviation between sample speed mean value and population's mean (i.e. day's average speed value), as well as the impact of daytime and sample size to the deviation decrease.
To investigate the impact of measurement time, tests at various daytimes were simulated and difference between sample's and day's mean speed value at time periods from 9:00 to 15:00 and at time period outside this time frame were compared. To observe the impact of a sample size, simulations with various sample sizes were done as in Figure 4.

Results
By simulating tests and analysing the results it was concluded that: (1) At time period from 9:00 to 15:00, the average sample's speed deviation from a day's mean download speed is in the range of 3 Mbps and has a trend to decrease by increasing the sample size. In general, at this timeslot the deviation of more than 60% of measurements is less than 5 Mbps. With the increase of sample size the probability for deviation to not exceed 5 Mbps increases. At this time period, the most frequent measurements were observed with 0-3 Mbps deviation. It was also observed that approximately in 40-50% of measurements speed deviation higher than 5 Mbps was observed on holidays (i.e. on Saturdays and Sundays) ( Table 1).
(2) At the rest of the time, sample's mean speed deviation from a day's mean is in the range of 10 Mbps. Deviation 5 Mbps or lower was observed in less than 40% of Algorithm 1. Python code for simulation of random measurements in one-hour time period basing on prepared existing data. results.append(dict(stats)) 21: print(results) #Output: mean difference in day's and hour's average speed for each location, operator and day of the week; 95% confidence interval of a mean speed difference; probability of speed difference to be more or less than 5 Mbps at certain daytime hour and for each sample size measurements. At this time period, the most frequent measurements were observed with 10 Mbps deviation relating to day's average speed value. On holidays (i.e. on Saturdays and Sundays) ∼25-35% of measurements with speed deviation higher than 5 Mbps were observed (Table 2).

Conclusion
The results of the research show that the most objective download speed measurement results, compared to day's average speed, can be achieved by performing speed tests at time period from 9:00 to 15:00. However, to avoid random influences, and ensure that ranges tested speed deviation is as narrow as possible, it is preferred to choose a sample size of more than 10 tests. At the same time, after test sample size n = 10, the minor decrease in the ranges of speed deviation is observed. Ten measurement sample size ensures that deviation of 66% of measurements is not exceeding 5 Mbps speed ranges, which ensures objective assessment of QoS indicators. With further increase of the sample size the proportion of tests, whose download speed deviation is lower than 5 Mbps, becomes flatter. Therefore, in order to obtain the realistic assessment of actually achievable download speed values but not wasting dedicated resources it is considered as optimal to perform measurement with a sample size 10-13 tests at certain location. Tests should be done on working days in a daytime period from 9:00 to 15:00.

Disclosure statement
No potential conflict of interest was reported by the author(s). Prof. Vjaceslavs Bobrovs is a director of RTU Institute of Telecommunications, Vice-dean for science of RTU Faculty of Electronics and Telecommunications, member of RTU Science council, and head of Transmission Systems Department at RTU TI. He has participated in several ERDF-funded and ESFfunded projects as well as Latvian National Research Programs. He has an excellent track record for managing and successful completion of this project. Prof. Bobrovs expertise in the physical optical layer, signal processing, performance evaluation, modeling, and characterization of fiber-optic transmission systems is crucial for the proposed project. Prof. Bobrovs has (co-)authored more than 140 scientific publications.

Notes on contributors
Prof. Girts Ivanovs is a professor at Institute of Telecommunications, Riga Technical university and has over 30 years of academic experience in telecommunications sector. His field of expertise is information and communication technologies, fiber-optic transmission systems and signal processing. Prof. Ivanovs has (co-)authored more than 100 scientific publications.