At first glance, it may seem that our partners don't mind how Wialon works with messages. They install trackers and sensors on their clients' vehicles, connect them to Wialon, devices transmit information, our platform displays it – everything works!
In practice, questions from end-users are inevitable. “All month I have been tracking data in the mobile app, today I’ve generated a report, and the results are different!”, “The sensors sent new information – why can’t I see it in the report?”.
We believe that understanding the algorithms of our platform simplifies the end-user's work with it.
That is why, for example, Oleg Zharkovsky, leader of the Wialon training team, and Maria Poklonskaya, business analyst at Gurtam, held a meetup Three methods of data processing in Wialon.
In this article, we have collected and structured the most important and useful information from the meetup.
Let's start with the terms: what do we mean by data when we speak about Wialon?
Data is a message that is stored and transmitted by a device installed on a unit. Such messages include:
Wialon receives this data, makes changes if necessary, and processes it.
Let's talk in more detail about each step of data processing in Wialon.
For Wialon, time is extremely important, since Wialon uses time to sort messages.
How it works: the tracker generates a message using the Unix time format (the number of seconds elapsed since midnight on January 1, 1970) and sends it to the server.
Wialon truncates a timestamp to a second but does not indicate milliseconds. At the same time, popular trackers like Teltonika, Ruptela, Galileosky can generate more than one message per second. With a large stream of messages, this can lead to confusion, since Wialon has no other criteria for sorting messages, except for time.
The Wialon development team is thinking about adding three more characters to the time field. But the implementation and testing of this refinement is not a quick fix, since data processing in seconds and milliseconds should be as reliable as possible.
The time taken by the tracker to save a message is less than or equal to the time the tracker spends to send it from its memory, and all of this is less than or equal to the time it arrives at the server.
It looks like this:
In practice, the user may encounter a situation that we call a message from the future: the server receives a message with a timestamp that has not arrived yet. Here is why this happens:
Parameters are values in messages that describe the state of the monitored unit and the devices installed on it.
Unreadable data. The information that Wialon receives is not always clear to the user.
When the user sees the value temperature13, he understands: we are talking about temperature. But the temperature of what: engine, fuel, or air? Celsius or Fahrenheit?
Each parameter can cause a wide array of such questions. Every hardware manufacturer has their own idea of how to record values.
Wialon, for its part, unifies parameters that are common to all devices. For example, the speed parameter is always speed; lat, lon stand for coordinates, and so on.
No unified data standard. Wialon currently works with 2,300 types of devices, and the number is growing. Such flexibility has its drawbacks: it is impossible to automatically standardize parameter values with so many devices and scenarios of their use.
To decrypt messages from GPS tracking devices, you can go to the Integrated devices section on the Hardware tab.
If you could not find the information you are looking for, then write to the hardware specialists team at firstname.lastname@example.org, and they will help you.
Wialon deletes the data that falls into the category of inaccurate according to the validity settings. All settings are made by the user, otherwise, Wialon will not delete anything.
Let's briefly recap how message validity filtration works:
Message validity filtration setup
Although sensors also work with parameters, they greatly simplify their display for the user:
Data display with mathematical formulas. In the field where the parameter is specified, the user can indicate a formula. For example, it can be a simple one that converts meters to kilometers using a factor. More complex variants are also available: if the user wants to see acceleration, he enters a formula by which Wialon divides the speed difference between two messages by the time difference between them.
Table calculation. It allows the users to display simplified dependencies using only a few of the points that the integrator determines in practice.
For example, calibrating the tank when using a fuel level sensor helps to determine the correspondence of the fuel level in the tank to the volume of this fuel.
Also, using calculation, the user compares the sensor value with the range of the original parameter.
Here is an example: if the voltage is more than 23, then the ignition sensor is on, that is, equal to 1, and if less, then the sensor is off and outputs 0.
Now when we know what we mean by data in Wialon and what are the bottlenecks with its preprocessing, let’s proceed to the final step, i.e. data processing.
There are three major data processing methods in Wialon that users turn to most often. Two of them – online and analytics – have been around for a while. The third method – events – has appeared recently.
The methods work with the same messages.
The algorithm of the online data processing method is the simplest and most transparent one: it processes every new message in real-time.
Despite the fact that the online method seems to be a clear and transparent way of data processing, in practice, everything may be not that simple. Here is a more detailed example from the meetup.
Analytics can be opposed to online in a certain way: the online method processes every incoming message here and now, while analytics works with a complete period.
For example, if a user builds a report for the last hour, and then the tracker sends new data regarding that hour after the report generation, then the report will not display the new data until the user generates a new report.
Wialon downloads a set of data for a specified period and then works with it in accordance with the preset logic. At the same time, the logic for each analytics tool will be different.
We have added events to the online and analytics methods to combine the key advantages of the other two methods.
Events are as efficient as online: they take into account every new message, processing it in the background.
They work with any volume of messages like analytics: events take into account the previous messages and process them again if necessary. Unlike online, events display information in case of a lack of data. If the last message contains no data on a certain parameter, the event will take this information from the last message which contains it. Online doesn't look that far and puts a dash instead.
As a result, when the user requests information, the data has already been processed and can be immediately displayed.
At the moment, this algorithm works with six types of data:
For example, while the vehicle is moving, the tracker can send the message Engine is running over and over again. But it is much more efficient to send this message once when the engine is turned on, and then the message Engine is not running when the state has changed and the engine is turned off. Events provide insight into what happened during the period with just a single message.
Ideally, there should be no discrepancy between events and analytics. But in reality, it happens. Most often the discrepancy is caused by differences in algorithms. But in this case, we rather see differences in data display than different data itself.
If we talk about discrepancies between values, it happens for a number of reasons.
A large amount of data. The event method algorithm, which processes data only in chronological order, sorts messages first and then starts to process them from the moment when new data was received.
In case of a serious difference that does not disappear after some time, the user can always contact our technical support. These tasks have a high priority, and the team quickly and thoroughly works with each discrepancy case.
How to avoid the problem: configure the chronological order of messages from the tracker. This will ensure faster data processing since events won’t recalculate old data several times.
Changes in settings. Let's imagine that the user was tracking data in the form of events in the mobile application for the whole month, and every day he saw the same values. On the last day of the month, the user changed the settings, for example, of the fuel level sensors, and built a monthly report. The data in the report may differ from what he had been observing. This happened because the report applied updated settings to the data received for the entire month, while events automatically recalculated the data only for the last 24 hours.
How to avoid the problem: request an events recalculation for the whole period from the technical support, email@example.com.
We hope that the article has helped you learn more about data processing in Wialon. With this know-how, you can better understand potential discrepancies in the results and select the settings for your customers more accurately.
We regularly release useful videos on our YouTube channel – subscribe, watch meetups and webinars and send us your comments, we’ll take them into account in our work. And don’t forget to follow the blog on gurtam.com to stay up to date with our news, analytical materials, and detailed instructions.
Do you know about our Wialon Help Center? We collect the most useful information there – check it out!