Big Data technologies: oil Supply chain analytics and optimisation

The enormous data challenge

As the global supply chain business has advanced from paper ledgers to stand alone custom built systems through to paperless networked systems, the operational expense of human data entry and consequential errors has fallen. However, a new problem has emerged for forward thinking organisations and it is one that will be with us for a long time, big data.

The challenge for global supply chain logistics businesses, whether land based or ocean going, is one of effectively managing and extracting the value of data and its four V’s: volume, variety, velocity and veracity. The supply chain business involves data of almost every data format type:

  • Qualitative data and documentation
    (text files and contracts)
  • Geospatial data (Asset location tracking)
  • Quantitative tabular data with either extensive or
    minimal meta-data (e.g. quantity and quality assessment)
  • Digital image data (e.g. title transfer, BL, waybills through
    to satellite imagery)
  • Digital audio and video data files (e.g. voice recordings)

The advancement of technology has led to an explosion
of available data around each of these types.

On a micro scale, the IIoT (Industrial Internet of Things) has leveraged mobile data networks through the placement of sensors on remote or mobile assets and now on a macro scale through satellite imagery, there has been an explosion of the volume of data available to global supply chain organisations. For example, liquitrax (see, a solution by everis, has placed IIOT sensor-enabled Coriolis meters on trucks to   capture and transmit, in real time, the location of, quantity and quality of crude oil extracted from US shale plays. Whilst this is primarily aimed at accounting for losses between load and discharge, it also captures a wealth of other data that can be analysed and is equally applicable to the truck based movement of other liquids and gases.

How to extract value from this data overload

As this volume of data grows, deriving value from it becomes ever harder. The fine margins involved in logistics optimisation make the ability to tune out the data noise and turn these mountains of data into ‘monetisable’ information, a key differentiator.

The development of big data technologies, parallel processing and the introduction of machine learning algorithms is delivering enormous and surprising value in areas previously not considered by many organisations. Machine learning describes the ability of a computer system to learn or change without being programmed explicitly to do so when exposed to new data sets, whether they be raw, derived or combined.

Delivering predictive analytics with this kind of artificial intelligence sounds expensive and overwhelming, but this isn’t the case with the right approach.

Start small and be laser focused 

Often companies feel that they should be doing something in this space but are unsure of how to get started or of the business use case that will help them to do so.

This is where subject matter expertise (SME), not just technical knowhow, is required. This is something everis has helped with many times, to establish a short, time-boxed Big Data proof of value (PoV) to show value swiftly and garner additional support and sponsorship at executive levels.

The application of a technology agnostic and open-source based approach keeps costs down and ensures compliancy with corporate IT standards. Delivering this though a secure cloud based strategy ensures minimal impact on existing systems whilst the PoV is ongoing.

A data scientist and SME work on the initial use case whilst a data analyst cleans and ingests the data into the secure cloud environment. This data discovery phase yields valuable insights and statistical patterns emerge far sooner than one would normally imagine.

Experience shows us that the insights extracted accelerate as we add more data to the data lake based on our initial findings.

A study by Forrester Research found that most companies estimate they are analysing less than 15% of the data they have access to. This may be due to data silos only being available to certain parts of or systems within the company, data format incompatibility issues, or simply a lack of analytical tools. Beyond the obvious, it takes time and effort to know which additional data sources are valuable and which are not. Machine learning will answer that question for you and, whether the data is structured or unstructured, it can be consumed and analysed.

Finding the relevant documents before you ask

Studies have shown that on average workers spend 108 minutes per day searching for information and 24 minutes handling paperwork. In a business such as global shipping and logistics, where in addition to contracts and invoices, documents around insurance, inspection and customs are of critical importance and are central to the business process flow these numbers are most likely grossly underestimated.

There are many document management systems that will provide basic content management functionality. In the same way that machine learning identifies data that has value when combined with other data sets, context sensitive predictive search does the thinking and intelligent searching to extract the relevant documents when you need them.

It is important to note that the approach recommended in this article will not render previous IT system deployments worthless. In fact, the opposite is true. This approach will allow far more business value to be extracted from previous IT investments as they will become interconnected and the data is set free from its siloed housing. These would be incorporated as further source systems for the data analytics machine-learning algorithms and be used in combination or derivation with that from other previously unconnected systems.

Harness the power of predictive analytics

Based on a successful PoV, the technology deployed can be scaled up and used in a production environment allowing the data science to drive pre-processed ‘business real time’ results for predictive analytics. These analytics are akin to the user experience we enjoy in very common internet shopping experiences whereby our use of the platform directly generates suggestions of additional topics of interest and alerting built on these preferences becomes possible.

Whilst the data science of statistical modelling generates, isolates and informs based on mathematical models, the depth of documentation and digital imagery to support the hypothesis can be semantically linked and simply retrieved to ensure contextual support – relevant documentation is served up to the user from a content management service. One example of this would be the analysis of predicted weather patterns, current maritime choke point congestion and expected port traffic to assess impacts on contractual laydays and demurrage tolerances.


As with all such IT initiatives, the worry is one of what it will cost, how many bright shiny tin boxes will be needed and the software licences to make them work. The development of the open source software movement means that the vast majority of the software tools required to make these approaches deliver business value, are available free of licence cost and the annual, double digit percentage costs of support and maintenance that come with licenced software packages.
As these solutions are cloud based, the use of virtual hardware allows tin boxes to be brought up securely based in the cloud at a fraction of the traditional approach of having them delivered to your premises.

This approach allows a Proof of Value project to be undertaken for a surprisingly low initial investment and has encouraged more and more forward thinking organisations to dip their toe into the big data lake rather than diving in head first and hoping for the best.

About everis

everis is an NTT DATA company that specialises in consulting, transformation, technology and operations for various sectors, including the oil, gas and logistics sector.

everis has a great depth of expertise in this area and has provided several IIOT and Big Data solutions for companies in the supply chain arena. For further information contact Iain Chalmers
on  [email protected] or +44 (0)207 299 7890.

2017-06-01T09:40:21+00:00 April 13th, 2017|Ask the Experts|