How Wialon processes data

At first glance, it may seem that our partners don't mind how Wialon works with messages. They install trackers and sensors on their clients' vehicles, connect them to Wialon, devices transmit information, our platform displays it – everything works!

In practice, questions from end-users are inevitable. “All month I have been tracking data in the mobile app, today I’ve generated a report, and the results are different!”, “The sensors sent new information – why can’t I see it in the report?”.

We believe that understanding the algorithms of our platform simplifies the end-user's work with it.

That is why, for example, Oleg Zharkovsky, leader of the Wialon training team, and Maria Poklonskaya, business analyst at Gurtam, held a meetup Three methods of data processing in Wialon.

In this article, we have collected and structured the most important and useful information from the meetup.

What data we are talking about

Let's start with the terms: what do we mean by data when we speak about Wialon?

Data is a message that is stored and transmitted by a device installed on a unit. Such messages include:

• time;
• information from the GPS module: coordinates, speed, altitude, number of satellites;
• parameters from external and internal sensors: fuel level from the FLS, the number of wheel rotations from the odometer, and so on.

Wialon receives this data, makes changes if necessary, and processes it.

Steps in Wialon's data processing

Let's talk in more detail about each step of data processing in Wialon.

Time in Wialon

For Wialon, time is extremely important, since Wialon uses time to sort messages.

How it works: the tracker generates a message using the Unix time format (the number of seconds elapsed since midnight on January 1, 1970) and sends it to the server.

How time affects data

• Wialon truncates a timestamp to a second but does not indicate milliseconds. At the same time, popular trackers like Teltonika, Ruptela, Galileosky can generate more than one message per second. With a large stream of messages, this can lead to confusion, since Wialon has no other criteria for sorting messages, except for time.

The Wialon development team is thinking about adding three more characters to the time field. But the implementation and testing of this refinement is not a quick fix, since data processing in seconds and milliseconds should be as reliable as possible.

• The time taken by the tracker to save a message is less than or equal to the time the tracker spends to send it from its memory, and all of this is less than or equal to the time it arrives at the server.

It looks like this:

Wialon's formula for sending and receiving a message

In practice, the user may encounter a situation that we call a message from the future: the server receives a message with a timestamp that has not arrived yet. Here is why this happens:

• Device configuration. If the UTC+0 time zone is not set in the tracker, then the tracker can send a message to the server with an incorrect timestamp. As a result, Wialon may receive a message with a timestamp “from the future” and will have to change it. The solution is simple – set the tracker time to UTC+0.
• Time synchronization. The situation is typical for Wialon Local: when synchronization is not configured, the server time may deviate by seconds or even minutes. To avoid this, the server administrator must configure regular time synchronization using the NTP (Network Time Protocol).

Sensor parameters

Parameters are values in messages that describe the state of the monitored unit and the devices installed on it.

What affects work with parameters

• Unreadable data. The information that Wialon receives is not always clear to the user.

When the user sees the value temperature13, he understands: we are talking about temperature. But the temperature of what: engine, fuel, or air? Celsius or Fahrenheit?

Each parameter can cause a wide array of such questions. Every hardware manufacturer has their own idea of how to record values.

Wialon, for its part, unifies parameters that are common to all devices. For example, the speed parameter is always speed; lat, lon stand for coordinates, and so on.

• No unified data standard. Wialon currently works with 2,300 types of devices, and the number is growing. Such flexibility has its drawbacks: it is impossible to automatically standardize parameter values with so many devices and scenarios of their use. 

To decrypt messages from GPS tracking devices, you can go to the Integrated devices section on the Hardware tab.

• In the Search hardware field enter the model you are looking for.
• Next, select the Parameters tab.
• You will get a list of parameter names, their descriptions, and units of measurement (if applicable).

If you could not find the information you are looking for, then write to the hardware specialists team at hw@gurtam.com, and they will help you.

Message validity filtration

Wialon deletes the data that falls into the category of inaccurate according to the validity settings. All settings are made by the user, otherwise, Wialon will not delete anything.

Let's briefly recap how message validity filtration works:

• Go to the Advanced tab in the unit properties.
• Open the message validity filtration section.
• Choose your filtering conditions. For example, filtering by speed (the most common criterion) or the minimum number of satellites.
• For trackers which can determine the message validity themselves, select the Skip invalid messages option.

Message validity filtration setup

Sensors

Although sensors also work with parameters, they greatly simplify their display for the user:

• Sensors determine the logic of parameter processing. Each of the sensor types has algorithms associated with it in Wialon. For example, reports for temperature and weight have separate columns that Wialon automatically fills in depending on the sensor type.
• Sensors improve the current functionality. A common use case from our partners is connecting a relative odometer or mileage sensor to give the end-user more accurate mileage and trip data than GPS.
• Sensors provide access to new functionality. Here we've got a lot of examples: the FLS detects fuel thefts, the ignition sensor counts the engine hours, and the passenger sensor assigns passengers. Without creating these sensors, such functionality is not available.

How sensors process data

• Data display with mathematical formulas. In the field where the parameter is specified, the user can indicate a formula. For example, it can be a simple one that converts meters to kilometers using a factor. More complex variants are also available: if the user wants to see acceleration, he enters a formula by which Wialon divides the speed difference between two messages by the time difference between them.

• Table calculation. It allows the users to display simplified dependencies using only a few of the points that the integrator determines in practice.

For example, calibrating the tank when using a fuel level sensor helps to determine the correspondence of the fuel level in the tank to the volume of this fuel.

Also, using calculation, the user compares the sensor value with the range of the original parameter.

Here is an example: if the voltage is more than 23, then the ignition sensor is on, that is, equal to 1, and if less, then the sensor is off and outputs 0.

• Validation. Using validation, the user can link the values of several sensors to each other. For this, not only mathematical but also logical operations are used. For example, testing for inequality to zero.

Three main types of data processing

Now when we know what we mean by data in Wialon and what are the bottlenecks with its preprocessing, let’s proceed to the final step, i.e. data processing.

There are three major data processing methods in Wialon that users turn to most often. Two of them – online and analytics – have been around for a while. The third method – events –  has appeared recently.

The methods work with the same messages.

Online

The algorithm of the online data processing method is the simplest and most transparent one: it processes every new message in real-time.

• Monitoring tab. The tab features a list of units with a lot of corresponding indicators. The user clicks on the unit’s icon and gets extended data, for example, FLS values. When the values change, Wialon updates the data.
• Dashboard. Information such as geofences and units in it or the state of movement comes in the form of diagrams with online data.
• Notifications. The method allows end-users to calculate the duration of the unit state.

Despite the fact that the online method seems to be a clear and transparent way of data processing, in practice, everything may be not that simple. Here is a more detailed example from the meetup.

Analytics: messages, tracks, and reports

Analytics can be opposed to online in a certain way: the online method processes every incoming message here and now, while analytics works with a complete period.

For example, if a user builds a report for the last hour, and then the tracker sends new data regarding that hour after the report generation, then the report will not display the new data until the user generates a new report.

Analytics tools

• Messages. The tab displays the received data and calculates the mileage and duration of the trip.
• Tracks. The tab considers the trip detector and displays gas stations and other markers.
• Reports. This is an extremely flexible tool for displaying a wide range of information.

How analytics works

Wialon downloads a set of data for a specified period and then works with it in accordance with the preset logic. At the same time, the logic for each analytics tool will be different.

For example, the Trips report focuses on the trip detector, the Fuel fillings report works with the values from the FLS, and the Chronology report displays both trips and fuel fillings.

Events

We have added events to the online and analytics methods to combine the key advantages of the other two methods.

Events are as efficient as online: they take into account every new message, processing it in the background.

They work with any volume of messages like analytics: events take into account the previous messages and process them again if necessary. Unlike online, events display information in case of a lack of data. If the last message contains no data on a certain parameter, the event will take this information from the last message which contains it. Online doesn't look that far and puts a dash instead.

As a result, when the user requests information, the data has already been processed and can be immediately displayed.

At the moment, this algorithm works with six types of data:

• trips;
• sensors;
• ignition;
• speeding;
• counters;
• fuel.

Where events are used

• Wialon app for iOS and Android. Events are very popular on smartphones, as they provide quick access to the most relevant data for any period.
• Fuel filling and fuel theft notifications. Events stand behind these notifications.
• Dashboard. Events display part of the data: connection state (connected/not connected), motion state (moving/not moving, etc.).
• Speeding report. Although reports naturally belong to the analytics method, this type of report is an event method tool.
• Tooltips and extended information. When configured correctly, events help display the latest sensor data. This way, Wialon can work with trackers that report their status only at the moment of change, instead of constantly sending monotonous messages.

For example, while the vehicle is moving, the tracker can send the message Engine is running over and over again. But it is much more efficient to send this message once when the engine is turned on, and then the message Engine is not running when the state has changed and the engine is turned off. Events provide insight into what happened during the period with just a single message.

Discrepancies between data processing methods

Ideally, there should be no discrepancy between events and analytics. But in reality, it happens. Most often the discrepancy is caused by differences in algorithms. But in this case, we rather see differences in data display than different data itself.

If we talk about discrepancies between values, it happens for a number of reasons.

• A large amount of data. The event method algorithm, which processes data only in chronological order, sorts messages first and then starts to process them from the moment when new data was received.

In case of a serious difference that does not disappear after some time, the user can always contact our technical support. These tasks have a high priority, and the team quickly and thoroughly works with each discrepancy case.

How to avoid the problem: configure the chronological order of messages from the tracker. This will ensure faster data processing since events won’t recalculate old data several times.

• Changes in settings. Let's imagine that the user was tracking data in the form of events in the mobile application for the whole month, and every day he saw the same values. On the last day of the month, the user changed the settings, for example, of the fuel level sensors, and built a monthly report. The data in the report may differ from what he had been observing. This happened because the report applied updated settings to the data received for the entire month, while events automatically recalculated the data only for the last 24 hours.

How to avoid the problem: request an events recalculation for the whole period from the technical support, support@gurtam.com. 

The bottom line

We hope that the article has helped you learn more about data processing in Wialon. With this know-how, you can better understand potential discrepancies in the results and select the settings for your customers more accurately.

We regularly release useful videos on our YouTube channel – subscribe, watch meetups and webinars and send us your comments, we’ll take them into account in our work. And don’t forget to follow the blog on gurtam.com to stay up to date with our news, analytical materials, and detailed instructions. 

Do you know about our Wialon Help Center? We collect the most useful information there – check it out!