Skip to main content

Terrain-based adaption of propagation model loss parameters using non-linear square regression

A Publisher Correction to this article was published on 28 November 2021

This article has been updated


Reliable and real-time propagation loss modeling play a significant role in the efficient planning, development, and optimization of macrocellular communication networks in a given terrain. Thus, the need to adapt or tune an existing model to enhance its signal prediction accuracy in a specified terrain becomes imperative. In this paper, we proposed and applied a non-linear square regression method based on the Levenberg-Marquart (LM) algorithm to adapt and improve the empirical propagation loss estimation accuracy of the Egli model for two major cities in Nigeria. A comprehensive propagation loss measurement acquired over Long Term Evolution (LTE) mobile broadband networks operating at 2630 MHz for four different cities was collected using TEMS investigation tools to achieve the Egli model adaption. Results indicate that the adapted Egli model displays a high estimation accuracy over the Gauss-Newton (GN) algorithm leveraging the non-linear regression method employed to benchmark the propagation loss estimation. Using six standard statistical indicators, the adapted Egli model displayed lower estimation errors than the classical Egli model across the tested locations in the two cities investigated. Finally, the LM-adapted Egli model was compared with extensive measurements from another eNodeB in Port Harcourt different from the initial four eNodeBs investigated. The results indicate that the adapted model is suitable for deployment in related macrocellular environments.


Radio wave propagation in typical terrestrial terrains is a stochastic phenomenon whose characteristics vary in time due to atmospheric conditions and in space due to other clutter and environmental obstructions [1]. Primarily, this is the situation at the ultra-high frequency (UHF) radio propagation domain wherein a clutter of physical buildings, trees, hills, human activities, and vehicular movements, including the ever-constant changing and uncontrollable atmospheric conditions, dominate [2, 3]. The resultant effect of this stochastic phenomenon is large-scale attenuation and reduction of the propagated signal from radio waves between the transmitting and receiving channels [4]. Such reduction in strength or power of the propagated radio signal over a communication channel is known as propagation path loss [5,6,7].

Over the past few decades, several researchers have attempted to develop different models that can reliably estimate the signal propagation path loss over two or more radio communication terminals [8, 9]. Such reliable path loss estimations play a crucial role in the proper planning, development, and management of any wireless communication systems networks [10, 11]. The problem with the existing models is that none can be generalized to all environments, urban, rural, or suburban terrain. All existing models were developed to estimate path loss at a particular operating frequency in a specified environment [2]. One of such terrain-based models is the Egli propagation model developed from terrain-based measurements [12].

In order to adaptively modify an existing propagation model parameters to fit into practical field loss data, the use of least-squares (LS) regression has generally been the thriving method reported in several works [13,14,15]. For instance, an LS regression approach is studied in [13] to fine-tune the Hata model offset parameters to estimate field-measured propagation loss in a typical urban terrain. The work in [11] presents an adaptation of propagation model parameters toward achieving efficient cellular network planning using a robust LAD algorithm. A comparative investigation of least square methods for tuning the Erceg pathloss model was presented [14]. In [15,16,17,18,19], the authors explored the LS method to study and modify the offset parameters of COST-231 [20], Erceg [21], Hata [22], and Walficsh-Bertoni models [23] to enhance their predictive capabilities on-field loss data in different radio signal propagation terrains. The results indicate a need for a more generalized propagation loss model to predict pathloss in terrain-based environments accurately.

The study [24] also adopted a recursive algorithm corresponding with LS regression to calibrate the Okumura-Hata model estimation capacity for CDMA networks. Further information concerning pathloss measurements and modeling in CDMA networks have been reported [25,26,27]. In [28], an adapted LS algorithm, which the authors termed “Minimax LS algorithm,” was employed for automatic-based tuning of the Ericsson model to fit field measurements. However, the problem with the LS regression is that it approaches its abysmal performance in modeling and handling high stochastic and non-linear propagation loss datasets [28]. The limited performances of the existing models present a considerable gap in the literature, and the need to fill this gap is not out of place.

This work aims to examine and adapt the parameters of the terrain-based Egli model for suitable signal loss estimation using the non-linear square regression method. The study presents comprehensive propagation measurements over LTE mobile broadband networks at 2630 MHz for four different cities in Port Harcourt and Uyo in Nigeria. Measured pathloss was obtained using TEMS investigation tools, and the adaption of the Egli model to the empirical propagation loss data is proposed. The resultant model would be of great relevance to wireless cellular operators or network companies for enhanced network planning and efficient management of telecommunication networks in the two cities under consideration and any other related wireless propagation environment.

The rest of this paper is organized as follows. The methodology is presented in the “Methods” section. This section comprises the field measurement environments, tools, and procedures. Additionally, the classical Egli propagation loss model, adaptation of the classical Egli propagation loss model, and statistical performance metrics are discussed in this section. The results and helpful discussions are presented in the “Results and discussions” section. The results reported include measured propagation loss estimation and error distribution statistics using the classical Egli model and adapted Egli model for the cells under consideration. The measured propagation loss estimation and error distribution statistics using the classical Egli and adapted Egli models for the cells tested are broached. The estimated coefficients and statistics for the cells being tested are highlighted briefly. Additionally, the computed first-order estimates statistics for the cells considered are outlined. The calculated first-order estimates statistics for the cells investigated are presented. The LM-adapted Egli model was validated. Finally, the “Conclusions” section gives a concise conclusion to the paper.


The block diagram shown in Fig. 1 provides the methodology upon which this work is guided. It starts by acquiring the eNodeB transmission parameters via a detailed site survey. After conducting the site survey, actual field measurements involving signal data were initiated. This was followed by the definition of the propagation loss model in focus, and the steps followed to adapt the model based on the field loss data using the non-linear square regression method.

Fig. 1
figure 1

Block diagram showing the methodology of pathloss measurements and modeling

eNodeB transmission parameters

A site survey of the eNodeB transmission parameters of the commercial LTE network used in this work was conducted to obtain robust results. The parameters considered include the eNodeB total reference signal transmit power (Ptot), carrier transmission frequency (Fcf), antenna gain of the transmitter (Gt), the antenna gain of the receiver (Gr), transmit antenna height (hB), cable loss (Cl), and feeder loss (Fl).

Field measurement environments, tools, and procedure

This section presents the measurements environment, tools and procedure, and RF network data measured.

Measurement environment

Field measurements were conducted in two close medium-sized cities, Uyo and Portharcourt, located in the south-south zone of Nigeria. Uyo is the capital city of Akwa Ibom State, situated on latitudes 04° 59′ N and longitudes 07° 53′ E. It has a population density of 1200/km2 (3100/sq mi), with almost yearly warm and overcast long wet and short dry seasons. Port Harcourt is the capital city of Rivers State, located on latitude 4° 051′ N and longitude 7° 0′ E. It is the fifth largest congested city in Nigeria, with a population density of 1900/km2 and a weather condition similar to Uyo. Both cities are built up with a mixture of commercial/residential building clusters and sparsely human/vehicular traffics. Also, the topography of both cities ranges from flat to horizontal plains. For more description of the cities investigated, a schematic map showing the tested eNodeBs is given in Figs. 2 and 3, respectively.

Fig. 2
figure 2

A map showing eNodeBs locations in Uyo City Nigeria

Fig. 3
figure 3

A map showing eNodeBs locations in Port Harcourt City Nigeria

Measurement tools and procedure

The tools explored for the data collection comprise TEMS investigation drive test software installed on Sony Ericson handsets and connected to an HP Laptop similarly to [29]. Other supporting tools include the GPS, inverter and connecting cables, and the drive test commercial vehicle. The TEMS investigation drive test tool is a real-time professional telecom service quality monitoring software. TEMS possesses the capacity to measure, record, and display different desired RF spatial and temporal datasets in log files along the drive test routes. In this investigation, the test equipment was locked to the operating frequency of 2630 MHz of the tested eNodeBs. This helps to mitigate the impact of co-channel interference from neighboring cells operating at different frequencies. Additionally, the test vehicle was driven, considering the actual road traffic conditions at a relatively medium speed of up to 30 km/h with uniformity, thereby reducing possible Doppler effects. The post-processing of measured log data files was analyzed using Map Info, MS Excel, MATLAB analytic tools.

RF network data measurements

The work concentrated on the Reference Signal Received Power (RSRP) measurements using LTE radio networks [30, 31]. Technically, the measured RSRP is a special LTE network indicator for assessing signal power and coverage level [32]. With the aid of the experimental drive test tools, the RSRP data sets in the log were collected around four randomly selected eNodeB transmission sites, with two from each city considered. The four eNodeB transmission sites belong to well-known GSM/UMTS/LTE telecom service providers operating in the two cities investigated. In terms of the propagation loss, the RSRP is defined by [33] in eq (1), and the total reference signal transmit power is defined in (2).

$$ \mathrm{RSRP}\ \left(\mathrm{dB}\mathrm{m}\right)={P}_{tot}\left(\mathrm{dB}\right)-\mathrm{Path}\ \mathrm{Loss}\ \left(\mathrm{dB}\right) $$
$$ {P}_{tot}\left(\mathrm{dB}\right)={G}_t+{G}_r-{C}_l-{F}_l-\left(10\log {N}_{rb}\right)-10\log (12) $$

where Ptot defines the eNodeB total reference signal transmit power, Gr is the receiver antenna, Gt is eNodeB antenna gain, and Nrb indicates the number of resource blocks. Cl and Fl = express the cable loss and feeder loss, respectively. Thus, in terms of propagation loss, the expression in (1) can be written as (3) and further expressed in (4). The basic parameters used for simulation are defined in Table 1.

$$ \mathrm{PathLoss}\ \left(\mathrm{dB}\right)={P}_{tot}\left(\mathrm{dB}\right)-\mathrm{RSRP}\ \left(\mathrm{dB}\mathrm{m}\right) $$
$$ \mathrm{Path}\ \mathrm{Loss}\ \left(\mathrm{dB}\right)={P}_{tot}\left(\mathrm{dB}\mathrm{m}\right)+{G}_t+{G}_r-{C}_l-{F}_l-\left(10\log {N}_{rb}\right)-10\log (12)-\mathrm{RSRP}\ \left(\mathrm{dB}\mathrm{m}\right) $$
Table 1 Propagation loss computation parameters

The classical Egli propagation loss model

Consider a propagated signal from a transmitter to the receiver. In free space, the received signal, Pr as a function of the transmission frequency ft, transmit power Ptr, over a communication distance, d is given by [34] in eq (5):

$$ \frac{P_r}{P_{tr}}=\frac{G_{tr}{G}_r{\lambda}^2}{4{\pi}^2{d}_c^2} $$

where \( \lambda =\frac{c}{f_t} \)

The propagation loss in dB can be obtained from (1) as given in (6) and its simplified form in (7).

$$ Pl\ \left(\mathrm{dB}\right)=10{\log}_{10}\frac{P_r}{P_{tr}}=-10{\log}_{10}\frac{G_{tr}{G}_r{\lambda}^2}{4{\pi}^2{d}_c^2} $$
$$ Pl\ \left(\mathrm{dB}\right)=32+20{\log}_{10}\left({f}_{t(MHz)}\right)+20{\log}_{10}\left({d}_c\right)-{G}_{tr}-{G}_r $$

The expression in (5) is the popular free space path loss model [34]. In order to cater to the effect of receiver antenna height Hr and transmitter height Htr, and typical terrain obstacles, which the authors in [34] discounted in formulating (7), Egli modified the free space model based on extensive practical signal measurement conducted at VHF/UHF transmissions in the USA, yielding eq (8):

$$ {P}_r={G}_{tr}{G}_r{\left(\frac{H_{tr}{H}_r}{d_c^2}\right)}^2{\beta}^2{P}_{tr} $$

where \( \beta =\frac{40}{f_{t(MHz)}} \)

Similar to eq (6), the propagation loss in dB can be obtained from (8) as given in (9):

$$ Pl\left(\mathrm{dB}\right)=76.3+20{\log}_{10}\left({f}_{t(MHz)}\right)-20{\log}_{10}\left({H}_{tr}\right)-10{\log}_{10}\left({H}_r\right)+40{\log}_{10}\left({d}_c\right) $$

where the parameters, Gtr, Gr, Htr, and Hr have been defined in the preceding equations.

The expression in (9) is the classical Egli model, and it shows that doubling the communication distance and transmission frequency increases the propagation loss by a factor of 4 and 2, respectively. Based on (9), Egli only considered 40 MHz to 1GHz transmission frequency range and communication distances from 1 km to 50 km [12]. This limited range of operation of the model poses significant limitations that need to be resolved. In addition, the model was developed by assuming a terrain profile similar to plane earth with limited vegetation, irregular topography, and severe building obstructions. Therefore, the need for Egli’s model adaptation becomes very important.

Adaptation of the classical Egli propagation loss model

First, to modify the classical Egli model, we introduce some constant free adaption parameters into (9), resulting in eq (10):

$$ {Pl}_{Egli}\left(\mathrm{dB}\right)={A}_1+{A}_2{\log}_{10}\left({f}_{t(MHz)}\right)-{A}_3{\log}_{10}\left({H}_{tr}\right)-{A}_4{\log}_{10}\left({H}_r\right)+{A}_5{\log}_{10}\left({d}_c\right) $$

Now, let Ymdefine the measured signal propagation loss and \( {\hat{Y}}_m \) be the Egli model with A1,A2, A3, A4, andA5 being the adaptation parameters to be obtained based on the field signal propagation measurements in a non-linear square sense. It is given by eq (11):

$$ H=\sum \limits_{m=1}^n{\left({Y}_m-{\hat{Y}}_m\right)}^2=\sum \limits_{m=1}^n{\left({Y}_m-{A}_1-{A}_2{\log}_{10}\left({f}_{t(MHz)}\right)+{A}_3{\log}_{10}\left({H}_{tr}\right)-{A}_4{\log}_{10}\left({H}_r\right)+{A}_5{\log}_{10}\left({d}_c\right)\right)}^2 $$

where n indicates measured signal propagation data number. From (11), the challenge of determining the values of A1,A2,A3, A4, and A5 based on field measurement can be transformed into an optimisation problem as given by (12). The evolving partial derivatives are given in eq.(13) to (17). Details of the mathematical procedures are available [35,36,37]:

$$ \underset{A_1,{A}_2,..,{A}_n}{\min }H\left({A}_1,{A}_2,..,{A}_n\right)=\underset{A_1,{A}_2,..,{A}_n}{\min}\sum \limits_{m=1}^n{\left({Y}_m-{\hat{Y}}_m\right)}^2 $$
$$ \frac{\partial H}{\partial {A}_1}=2\sum \limits_{i=1}^n\;\left({Y}_m-{\hat{Y}}_m\;\right)\;\frac{\partial {\hat{Y}}_m}{\partial {A}_1}=0 $$
$$ \frac{\partial H}{\partial {A}_2}=2\sum \limits_{i=1}^n\;\left({Y}_m-{\hat{Y}}_m\;\right)\;\frac{\partial {\hat{Y}}_m}{\partial {A}_2}=0 $$
$$ \frac{\partial H}{\partial {A}_3}=2\sum \limits_{i=1}^n\;\left({Y}_m-{\hat{Y}}_m\;\right)\;\frac{\partial {\hat{Y}}_m}{\partial {A}_3}=0 $$
$$ \frac{\partial H}{\partial {A}_4}=2\sum \limits_{i=1}^n\;\left({Y}_m-{\hat{Y}}_m\;\right)\;\frac{\partial {\hat{Y}}_m}{\partial {A}_4}=0 $$
$$ \frac{\partial H}{\partial {A}_5}=2\sum \limits_{i=1}^n\;\left({Y}_m-{\hat{Y}}_m\;\right)\;\frac{\partial {\hat{Y}}_m}{\partial {A}_5}=0 $$

By setting \( \frac{\partial H}{\partial {A}_1}={f}_1\left({A}_1,{A}_2,{A}_3\right)\frac{\partial H}{\partial {A}_2}={f}_2\left({A}_1,{A}_2,{A}_3\right) \), and \( \frac{\partial H}{\partial {A}_3}={f}_3\left({A}_1,{A}_2,{A}_3\right) \)

The expressions in (13) to (17) are converted as given in (18):

$$ \left\{\begin{array}{l}{f}_1\left({A}_1,{A}_2,{A}_3,{A}_4,{A}_5\right)=0\\ {}{f}_2\left({A}_1,{A}_2,{A}_3,{A}_4,{A}_5\right)=0\\ {}{f}_3\left({A}_1,{A}_2,{A}_3,{A}_4,{A}_5\right)=0\\ {}{f}_3\left({A}_1,{A}_2,{A}_3,{A}_4,{A}_5\right)=0\\ {}{f}_3\left({A}_1,{A}_2,{A}_3,{A}_4,{A}_5\right)=0\end{array}\right. $$

Equation (18) expresses the non-linear equation; solving for the values of A1, A2, A3, A4, and A5 by employing analytical techniques is generally a complex task. A commonly used method of solving the above complex equation is the Gauss-Newton method [38]. Still, its application during the iterative implementation process requires a full rank matrix, thus becoming a significant limitation of the algorithm.

In this work, we engaged the Levenberg-Marquart method [39,40,41] to upturn the limitation of the Gaussian Newton algorithm in resolving the non-linear equation in (18). Thus, by employing the Levenberg-Marquart algorithm, eq (18) is transmuted as given in (19):

$$ \underset{A\in {\Re}^3}{\min }f(A)=\frac{1}{2}{\left\Vert f(A)\right\Vert}^2=\frac{1}{2}\sum \limits_{i=1}^5{f}_i^2(A),A=\left({A}_1,{A}_2,{A}_3,{A}_4,{A}_5\right) $$

The Levenberg-Marquart method can be resolved with the following eq (20)–(21):

$$ \left(J{\left({x}_q\right)}^TJ\left({x}_q\right)+{\mu}_qI.\right)\Delta {x}_q=-J{\left({x}_q\right)}^Tf\left({x}_q\right),\mu \ge 0 $$
$$ \Rightarrow \Delta {x}_q={\left(J{\left({x}_q\right)}^TJ\left({x}_q\right)+{\mu}_qI.\right)}^{-1}\left(-J{\left({x}_q\right)}^Tf\left({x}_q\right)\right)\mu \ge 0 $$

J(xq) = fI(A)= Jacobian Matrix


Im × m and μ express the damping term and the identity matrix introduced by Leverberg-Marquart into the classical Gauss-Newton algorithm [38] to improve its performance.

Accordingly, utilizing the Levenberg-Marquart method, the parameters A = (A1, A2, A3, A4, A5) can be obtained iteratively using eq (22):

$$ {x}_{q+1}={x}_q+{\left(J{\left({x}_q\right)}^TJ\left({x}_q\right)+{\mu}_qI.\right)}^{-1}\left(-J{\left({x}_q\right)}^Tf\left({x}_q\right)\right),\mu \ge 0 $$

Thus, for μ = 0, the expression in eq (20) becomes the classical Gauss-Newton algorithm given by eq (23):

$$ \left(J{\left({x}_q\right)}^TJ\left({x}_q\right)\right)\Delta {x}_q=-J{\left({x}_q\right)}^Tf\left({x}_q\right) $$

Similarly, eq (22) is simplified as given in (24):

$$ {x}_{q+1}={x}_q+{\left(J{\left({x}_q\right)}^TJ\left({x}_q\right)\right)}^{-1}\left(-J{\left({x}_q\right)}^Tf\left({x}_q\right)\right) $$

To implement the LM algorithm, the following steps are engaged intuitively:

LM algorithm implementation steps

  1. I.

    Initialise guess parameters,xo for x at iteration q = 0, 1, 2, …

  2. II.

    Select the Lagrange multiplierλqfor each step q

  3. III.

    Calculate theΔxqwith its expression in eq. (21)

  4. IV.

    Calculate xq + 1 = xq + Δxq

  5. V.

    Evaluate Δxq = xq + 1 − xqat the initial parameter,xo

  6. VI.

    For smaller Δxqvalues, check the rate of convergence

  7. VII.

    If convergence rate is acceptable, stop the calculation or else go a step (IV).

Performance statistical metrics

To quantitatively examine the performance of the non-linear regression approach, six standard statistical metrics are engaged. These include the root mean square error (RMSE), maximum absolute error (MAE), mean absolute percentage error (MAPE), mean absolute error (MAE), standard deviation error (STD), and mean percentage residual error (MPRE). Detailed information about these statistical metrics can be found in [35, 42,43,44,45].

Results and discussions

The results of the study and valuable discussions are presented in this section. As shown in Figs. 4, 5, 6, and 7, the top parts of the graphs reveal how the classical Egli model in eq (9) predicts the acquired propagation loss values relative to the measured distances across the study site locations. It is conspicuously clear from the graphs in Figs. 4, 5, 6, and 7 that the classical Egli model over-predicted the measured propagation loss with considerably high RMSE, MAE, MAPE, MAE, STD, and MPRE values in sites 1 to 4. Such significant error differences may be attributed to the physical terrain and topographical differences between locations where the measurement loss data is conducted and the terrain characterization where the Egli model was developed. Thus, the need to adapt (fine-tune) the Egli model to fit the measured propagation loss data is self-evident.

Fig. 4
figure 4

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_1

Fig. 5
figure 5

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_2

Fig. 6
figure 6

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_3

Fig. 7
figure 7

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_4

To adapt the Egli model to the measured propagation loss data acquired over the two study locations using the non-linear square regression method, we employ the robust Levenberg-Marquart algorithm and the commonly used Gauss-Newton algorithm. Table 2 displays the obtained adaptive coefficients of the Egli model and their descriptive statistical values after employing the non-linear square regression method for its adaptation across the propagation loss measurement locations. Furthermore, Table 2 presents the estimation errors using the Egli model before and after the adaptation to measured field propagation loss data in the study locations. Moreover, Figs. 4, 5, 6, and 7 are plotted graphs showing the estimation performance of the Egli model on the measured propagation loss before and after its adaptation in the four study locations using the Levenberg-Marquart algorithm and the commonly used Gauss-Newton algorithm.

Table 2 Estimated coefficients and statistics for Cell_1 to Cell_4

Based on the obtained adapted propagation model parameters summarized in Table 2 using the LM algorithm, the Egli model loss PlEgli(dB) for cell_1 to cell_4 can be written as (25)–(28):

$$ {Pl}_{Egli}(dB)=72.24+6.16{\log}_{10}\left({f}_{t(MHz)}\right)+26.20{\log}_{10}\left({H}_{tr}\right)-10.71{\log}_{10}\left({H}_r\right)+39.34{\log}_{10}\left({d}_c\right) $$
$$ {Pl}_{Egli}(dB)=73.25+9.60{\log}_{10}\left({f}_{t(MHz)}\right)-24.49{\log}_{10}\left({H}_{tr}\right)-10.53{\log}_{10}\left({H}_r\right)+30.48{\log}_{10}\left({d}_c\right) $$
$$ {Pl}_{Egli}(dB)=73.98+12.07{\log}_{10}\left({f}_{t(MHz)}\right)+23.49{\log}_{10}\left({H}_{tr}\right)-10.40{\log}_{10}\left({H}_r\right)+28.69{\log}_{10}\left({d}_c\right) $$
$$ {Pl}_{Egli}(dB)=73.35+9.93{\log}_{10}\left({f}_{t(MHz)}\right)-24.35{\log}_{10}\left({H}_{tr}\right)-10.51{\log}_{10}\left({H}_r\right)+29.39{\log}_{10}\left({d}_c\right) $$

Taking the mean of the first two eq (25)–(26) and the last two eq (27)–(28) leads to the adapted Egli models obtained for Port Harcourt City and Uyo City in (29) and (30), respectively:

$$ {Pl}_{Egli}(dB)=29.75+35.16{\log}_{10}\left({f}_{t(MHz)}\right)+{A}_3{\log}_{10}\left({H}_{tr}\right)-{A}_4{\log}_{10}\left({H}_r\right)+9.42{\log}_{10}\left({d}_c\right) $$
$$ {Pl}_{Egli}(dB)=32.16+26.90{\log}_{10}\left({f}_{t(MHz)}\right)+{A}_3{\log}_{10}\left({H}_{tr}\right)-{A}_4{\log}_{10}\left({H}_r\right)+11.54{\log}_{10}\left({d}_c\right) $$

The above expressions also reveal that the mean rate of propagated loss attenuation for Port Harcourt City and Uyo City are 3.5 and 2.9, respectively. As expected, the values also clearly reveal that the signal loss attenuation rate in Port Harcourt is about 30% higher than the one obtained for Uyo. The higher signal loss attenuation rate may be ascribed to higher cluttered buildings, other obstructions, and congested human/vehicular traffic in Port Harcourt than Uyo city.

Additionally, Figs. 8, 9, 10, and 11 provided the residual error distribution quantitatively fits of the LM and GN regression methods along with the measured propagation loss data. In terms of mean percentage residual error indicated with letter W in the graphs, the LM method attained 3.53, 3.50, 3.26, and 2.04, respectively, whereas the GN method attained more flawed values of 4.77 5.56, 11.63, and 15.09. Furthermore, plots showing normal distribution fits are shown in each figure to reveal the residual error distribution spreads using the LM and GN methods. Again, lower error spreads with LM over the GN method in each plot show a better Egli model adaptation efficiency. The enhanced estimation performance of the LM-based adapted Egli model relative to the GN measured propagation loss data can be attributed to its ability to tune multiple free parameters and achieve optimal solutions regardless of the nature of the initial guess parameters selected.

Fig. 8
figure 8

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_1

Fig. 9
figure 9

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_2

Fig. 10
figure 10

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_3

Fig. 11
figure 11

Measured propagation loss estimation and their error distribution statistics using classical Egli model and adapted Egli model for Cell_4

Estimation error analysis

The estimation errors are obtained using six standard statistical indicators. These comprise the root mean square error (RMSE), maximum absolute error (MAE), mean absolute percentage error (MAPE), mean absolute error (MAE), standard deviation error (STD), and mean percentage residual error (MPRE), as briefed in the “Adaptation of the classical Egli propagation loss model” section. For convenience, we also used W to refer to the mean percentage residual error (MPRE) in the results, as shown in Figs. 8, 9, 10, and 11. The computed first-order estimates statistics are presented in Tables 3, 4, 5, and 6. Specifically, the computed first-order estimates statistics for Cell_1 are presented in Table 3. The calculated first-order estimates statistics for Cell_2 are given in Table 4. The calculated first-order estimates statistics for Cell_3 are shown in Table 5, and the computed first-order estimates statistics for Cell_4 are displayed in Table 6.

Table 3 Computed first order estimates statistics for Cell_1
Table 4 Computed first order estimates statistics for Cell_2
Table 5 Computed first order estimates statistics for Cell_3
Table 6 Computed first order estimates statistics for Cell_4

The summarized fitting performance using LM and GN regression methods compared with the original Egli model on the measured loss data for each eNodeB cell are displayed in Tables 3, 4, 5, and 6. With MAE and RMSE, the LM method attained 0.18, 0.37, 2.32, 3.41 and 5.33, 2.12, 4.23, 4.56 values compared to GN-based adapted Egli model that attained 3.80, 3.32, 3.56, 5.44 and 5.12, 7.13, 8.22, 10.21, respectively. Improved fitting performances are recorded for the adapted Egli model using STD, MAPE, and MPRE statistical metrics. It is worth noting that lower computed error values with the indicators depict a better estimation accuracy.

Finally, field measurements taken from four selected eNodeB transmission sites located in Uyo and Port Harcourt in Nigeria were used in this study. The adapted Egli model was derived based on these measurements. In order to test the validity of the model, we compared the adapted Egli model with another set of measurements taken from another cell different from the initial four eNodeBs tested. The height of the new eNodeB investigated is 32 m, and other parameters of the eNodeB were observed. The results of our comparison are shown in Fig. 12. Other valuable results include A1=7.39, A2=9.18, A3=-2.60, A4=0.57, A5=41.11. MAE=3.38, MAPE=2.15, STD=2.80, RMSE=4.40, and PE=7.63.

Fig. 12
figure 12

Comparison of measured pathloss from the new eNodeB and the LM-based adapted Egli model


A detailed investigation to improve real-time propagation path loss estimation using two closed medium-sized cities in Nigeria was presented in this paper. By employing a non-linear square regression method based on the Levenberg-Marquart algorithm, Egli’s model was adapted to get an improved fitting for propagation loss in the investigated cities. Additionally, the performances of the adapted Egli model and the classical Egli model based on the measured propagation loss data have been demonstrated using six different statistical indicators for comparative analysis. The adapted Egli model displayed lower estimation errors than the classical model across the two cities studied. The results reveal that the adjusted Egli model parameters better fit the two cities investigated and can be deployed to efficiently plan macrocellular communication networks in similar propagation environments. Finally, the work in this paper could provide valuable guidelines and references in dimensioning or optimizing cellular network deployment. Future work would focus on optimizing the parameters of the adapted Egli model for improved performance and providing correction factors to ease its applicability in other related propagation environments for emerging wireless communication systems.

Availability of data and materials

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Change history



Code division multiple access


Evolved node base station




Global Positioning System


Global System of Mobile Communication


Least absolute deviation




Long-term evolution


Least squares


Mean absolute error


Mean absolute percentage error


Mean percentage residual error


Percentage error


Root mean square error


Reference signal received power


Signal error ratio


Standard deviation


Test Mobile System


Ultra high frequency


Universal Mobile Telecommunications System


Very high frequency


  1. Tataria H, Haneda K, Molisch AF, Shafi M, Tufvesson F (2021) Standardization of propagation models for terrestrial cellular systems: a historical perspective. Int J Wirel Inf Networks 28(1):20–44.

    Article  Google Scholar 

  2. Ebhota VC, Isabona J, Srivastava VM (2019) Environment-adaptation based hybrid neural network predictor for signal propagation loss prediction in cluttered and open urban microcells. Wirel. Pers. Commun. 104(3):935–948.

    Article  Google Scholar 

  3. da Costa FM, Ramirez LAR, Dias MHC (2018) Analysis of ITU-R VHF/UHF propagation prediction methods performance on irregular terrains covered by forest. IET Microwaves Antennas Propag 12(8):1450–1455.

    Article  Google Scholar 

  4. Li X (2006) RSS-based location estimation with unknown pathloss model. IEEE Trans Wirel Commun 5(12):3626–3633.

    Article  Google Scholar 

  5. Isabona J, Srivastava VM (2017) Coverage and link quality trends in suburban mobile broadband HSPA network environments. Wirel. Pers. Commun. 95(4):3955–3968.

    Article  Google Scholar 

  6. Nadir Z (2012) “Empirical pathloss characterization for Oman,” in 2012 Computing, Communications and Applications Conference, pp 133–137.

    Book  Google Scholar 

  7. Nadir Z, Ahmad MI (2010) Pathloss determination using Okumura-Hata model and cubic regression for missing data for Oman. Proc. Int. MultiConference Eng Comput Sci 2010:804–807

    Google Scholar 

  8. Wang CX, Bian J, Sun J, Zhang W, Zhang M (2018) A survey of 5 g channel measurements and models. IEEE Commun Surv Tutorials 20(4):3142–3168.

    Article  Google Scholar 

  9. Imoize AL, Ibhaze AE, Atayero AA, Kavitha KVN (2021) “Standard propagation channel models for MIMO communication systems,” Wirel. Commun. Mob. Comput 2021(8838792):36.

    Article  Google Scholar 

  10. Blaszczyszyn B, Karray MK (2012) Linear-regression estimation of the propagation-loss parameters using mobiles’ measurements in wireless cellular networks, in 2012 10th International Symposium on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks (WiOpt), pp 54–59

    Google Scholar 

  11. Hu Y, Leus G (2015) Self-estimation of path-loss exponent in wireless networks and applications. IEEE Trans Veh Technol 64(11):5091–5102.

    Article  Google Scholar 

  12. Egli JJ (1957) Radio propagation above 40 MC over irregular terrain. Proc. IRE 45(10):1383–1391.

    Article  Google Scholar 

  13. AarnæS E, Holm S (2004) Tuning of empirical radio propagation models effect of location accuracy. Wirel Pers Commun 30(2):267–281.

    Article  Google Scholar 

  14. Nathaniel Chimaobi N (2017) Comparative study of least square methods for tuning erceg pathloss model. Am. J. Softw. Eng. Appl 6(3):61.

    Article  Google Scholar 

  15. Isabona J, Ojuh D (2020) Adaptation of propagation model parameters toward efficient cellular network planning using robust LAD algorithm. Int J Wirel Microw Technol 10(5):13–24.

    Article  Google Scholar 

  16. Castro BSL, Pinheiro MR, Cavalcante GPS, Gomes IR, De O, Carneiro O (2011) Comparison between known propagation models using least squares tuning algorithm on 5.8 GHz in Amazon region cities. J Microwaves Optoelectron 10(1):106–113.

    Article  Google Scholar 

  17. Castro-Hernandez D, Paranjape R (2016) Local tuning of a site-specific propagation path loss model for microcell environments. Wirel Pers Commun 91(2):709–728.

    Article  Google Scholar 

  18. Yang M, Shi W (2008) “A linear least square method of propagation model tuning for 3G radio network planning,” in 2008 Fourth International Conference on Natural Computation, vol 5, pp 150–154.

    Book  Google Scholar 

  19. Mohammed AK, Jaafar AA (2012) Performance evaluation of path loss in mobile channel for Karada District in Baghdad City. Eng Technol J 30(17):3023–3038

    Google Scholar 

  20. Drozdova VG, Akhpashev RV (2017) “Ordinary least squares in COST 231 Hata key parameters optimization base on experimental data,” in 2017. Int Multi Conference Eng Comput Inform Sci (SIBIRCON):236–238.

  21. Erceg V, Greenstein LJ, Tjandra SY, Parkoff SR, Gupta A, Kulic B, Julius AA, Bianchi R (1999) An empirically based path loss model for wireless channels in suburban environments. IEEE J. Sel. Areas Commun. 17(17):1205–1211.

    Article  Google Scholar 

  22. Hata M (1980) Empirical formula for propagation loss in land mobile radio services. IEEE Trans. Veh. Technol. 29(3):317–325.

    Article  Google Scholar 

  23. Walfisch J, Bertoni HL (1988) A theoretical model of UHF propagation in urban environments. IEEE Trans. Antennas Propag. 36(12):1788–1796.

    Article  Google Scholar 

  24. Liming X, Dacheng Y (2003) “A recursive algorithm for radio propagation model calibration based on CDMA forward pilot channel,” in 14th IEEE Proceedings on Personal, Indoor and Mobile Radio Communications, 2003. PIMRC 1:970–972.

    Article  Google Scholar 

  25. Pontes J, Porebska M, Fugen T, Wiesbeck W (2006) “Base station antenna synthesis for high sites CDMA networks,” in 2006 First European Conference on Antennas and Propagation, pp 1–4.

    Book  Google Scholar 

  26. Akhoondzadeh-Asl L, Noori N (2007) “Modification and tuning of the Universal Okumura-Hata model for radio wave propagation predictions,” in 2007 Asia-Pacific Microwave Conference, pp 1–4.

    Book  Google Scholar 

  27. Arias D, Rodriguez G (2015) “Performance of Advanced Metering Infrastructure using cellular communication based on uplink CDMA,” in Proceedings of 2015 International Conference on Intelligent Computing and Internet of Things, pp 111–116.

    Book  Google Scholar 

  28. Zrni VP, Akademija KA, Resanovi R (2001) Minimax LS algorithm for automatic propagation model tuning

    Google Scholar 

  29. Imoize AL, Ibhaze AE, Nwosu PO, Ajose SO (2019) Determination of best-fit propagation models for pathloss prediction of a 4G LTE network in suburban and urban areas of Lagos, Nigeria. West Indian J. Eng. 41(2):13–21.

    Article  Google Scholar 

  30. Imoize AL, Adegbite OD (2018) Measurements-based performance analysis of a 4G LTE network in and around shopping malls and campus environments in Lagos Nigeria. Arid Zo. J. Eng. Technol. Environ. 14(2):208–225

    Google Scholar 

  31. Imoize AL, Orolu K, Atayero AA-A (2020) Analysis of key performance indicators of a 4G LTE network based on experimental data obtained from a densely populated smart city. Data Br. 29(105304):1–17.

    Article  Google Scholar 

  32. Imoize AL, Oseni AI (2019) Investigation and pathloss modeling of fourth generation long term evolution network along major highways in Lagos Nigeria. Ife J. Sci. 21(1):39–60.

    Article  Google Scholar 

  33. Zakaria YA, Hamad EKI, Elhamid ASA, El-Khatib KM (2021) Developed channel propagation models and path loss measurements for wireless communication systems using regression analysis techniques. Bull. Natl. Res. Cent 45(1):54.

    Article  Google Scholar 

  34. Rappaport TS (2002) Wireless Communications: Principles and Applications, 2nd ed. Prentice Hall, Upper Saddle River

    Google Scholar 

  35. Bird J (2010) Engineering Mathematics, 5th ed. Newness.

    Book  Google Scholar 

  36. Folland GB (2020) Introduction to partial differential equations. Princeton university press.

    Book  MATH  Google Scholar 

  37. Ames WF (2014) Numerical methods for partial differential equations. Academic press

    Google Scholar 

  38. Spedicato E, Vespucci MT (1988) Numerical experiments with variations of the Gauss-Newton algorithm for nonlinear least squares. J. Optim. Theory Appl. 57(2):323–339.

    MathSciNet  Article  MATH  Google Scholar 

  39. Jang J-SR, Mizutani E (1996) “Levenberg-Marquardt method for ANFIS learning,” in Proceedings of North American Fuzzy Information Processing, pp 87–91.

    Book  Google Scholar 

  40. Fan J (2012) The modified Levenberg-Marquardt method for nonlinear equations with cubic convergence. Math. Comput. 81(277):447–466.

    MathSciNet  Article  MATH  Google Scholar 

  41. Yamashita N, Fukushima M (2001) “On the rate of convergence of the Levenberg-Marquardt method,” in Topics in numerical analysis. Springer, pp 239–249

    MATH  Google Scholar 

  42. Wu L, He D, Ai B, Wang J, Qi H, Guan K, Zhong Z (2020) Artificial neural network based path loss prediction for wireless communication network. IEEE Access 8:199523–199538.

    Article  Google Scholar 

  43. Popoola SI, Jefia A, Atayero AA, Kingsley O, Faruk N, Oseni OF, Abolade RO (2019) Determination of neural network parameters for path loss prediction in very high frequency wireless channel. IEEE Access 7:150462–150483.

    Article  Google Scholar 

  44. Yi H et al (2020) Channel characterization for vehicle-to-infrastructure communications at the terahertz band. IEEE Int. Conf. Commun. Work. ICC Work:2020.

  45. Okakwu IK, Oluwasogo ES, Ibhaze AE, Imoize AL (2019) A comparative study of time series analysis for forecasting energy demand in Nigeria. Niger. J. Technol 38(2):465.

    Article  Google Scholar 

Download references


The work of Agbotiname Lucky Imoize is partially supported by the Nigerian Petroleum Technology Development Fund (PTDF) and the German Academic Exchange Service (DAAD) through the Nigerian-German Postgraduate Program under Grant 57473408.


This study had no direct funding from any resource.

Author information




The manuscript was written through the contributions of both authors. Conceptualization, J.I.; methodology, J.I., and A.I.; writing—original draft preparation, J.I.; writing—review and editing, J.I., and A.I.; supervision, J.I., and A.I.; project administration, J.I., and A.I.; funding acquisition, A.I. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Agbotiname Lucky Imoize.

Ethics declarations

Competing interests

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Isabona, J., Imoize, A.L. Terrain-based adaption of propagation model loss parameters using non-linear square regression. J. Eng. Appl. Sci. 68, 33 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI:


  • Radio wave propagation
  • Terrestrial terrains
  • Egli model adaptation
  • Propagation loss
  • Non-linear square regression
  • Levenberg-Marquart algorithm
  • Efficient network planning