INT

Empowering Visualization

CONTACT US SUPPORT

ivaap

How to Extend the IVAAP Data Model with the Backend SDK

The IVAAP Data Backend SDK’s main purpose is to facilitate the integration of INT customers’ data into IVAAP. A typical conversation between INT’s technical team and a prospective customer might go like this:

Can I visualize my data in IVAAP?

Yes, the backend SDK is designed to make this possible, without having to change the UI or even write web services.

What if my data is stored in a proprietary data store?

The SDK doesn’t make any assumptions on how your data is stored. Your data can be hidden away behind REST services, in a SQL database, in files, etc.

What are the steps to integrate my data?

The first step is to map your data model with the IVAAP data model. The second step is to identify, for each entity of that data model, how records are identified uniquely. The third step is to plug your data source. The fourth and final step is to implement, for each entity, the matching finders.

What if I want to extend the IVAAP data model?

My typical answer to this last question is “it depends”. The SDK has various hooks to make this possible. Picking the right “hook” depends on the complexity of what’s being added.

Using Properties

The IVAAP data model was created after researching the commonalities between the industry data models. However, when we built it, we kept in mind that each data store carries its own set of information, information that is useful for users to consume. This is why we made properties a part of the data model itself. From an IVAAP SDK point of view. Properties is a set of name-value pairs that you can associate with specific entities within the IVAAP Data Model.

For example, if a well dataset is backed by an .LAS file in Amazon S3 or Microsoft Azure Blob Storage, knowing the location of that file is a valuable piece of information as part of a QC workflow. But not all data stores are backed by files, a file location is not necessarily relevant to a user accessing data from PPDM. As a result, the set of properties shown when opening a dataset backed by Azure will typically be different from a set coming from PPDM, even for the same well.

 

properties dialog example
An example of properties dialog, showing multiple properties of a seismic dataset stored in Amazon S3.

 

 

Calling the properties a set of name-value pairs does not do justice to its flexibility. While a simple name+value is the most common use, you can create a tree of properties and attach additional attributes to each name-value pair. The most common additional attribute is a unit, qualifying the value to make this property more of a measurement.  Another attribute allows name-value pairs to be invisible to users. The purpose of invisible properties is to facilitate the integration with other systems than the IVAAP client. For example, while a typical user might be interested in the size of a file, this size should be rounded and expressed in KB, MB or GB. An external software consuming the IVAAP Properties REST services would need the exact number of bytes.

One of the benefits of using properties to carry information is that it’s simple to implement your own property finder, and it requires no additional work. No custom REST services to write, and no widget to implement to visualize these properties. The IVAAP HTML5 client is designed to consume the IVAAP services of the data backend, and show these properties in the UI.

Adding Your Own Tables and Documents

One of the limitations of properties is they don’t provide much interaction. Users can only view properties. The simplest way to extend the IVAAP model in a way that users can interact with that data is to add tables. For example, the monthly production of a well is an easy table  to make accessible as a node under a well. Once the production of a well is accessible as a table, users have multiple options to graph this production: as a 2D Plot, as a pie chart, as an histogram, etc. And this chart can be saved as part of a dashboard or a template, making IVAAP a portal.

The IVAAP Data Backend SDK has been designed to make the addition of tables a simple task. Just like for properties, the HTML5 Viewer of IVAAP doesn’t need to be customized to discover and display these tables. It’s the services of the data backend that direct the viewer on how to build the data tree shown to users. And while the data backend might advertise many reports, only non-empty reports will be shown as nodes by the viewer. 

 

tabular reports
An example of tabular reports related to a well.

 

 

In the many customization projects that I’ve been involved in, the tabular features of IVAAP have been used the most. I have seen dozens of reports under wells. The IVAAP Data Backend makes no assumptions about where this production data is stored relative to where the well is stored. For example, you can mix the schematics from Peloton WellView with the production reports from Peloton ProdView. From a user point of view, the source of the data is invisible, IVAAP combines the data from several sources in a transparent way. Extending the IVAAP data model doesn’t just mean exposing more data from your data source, it also means enriching your data model with data from other sources.

Data enrichment is sometimes achieved just by making accessible the documents associated with a well. For example, for Staatsolie’s portal, the IVAAP UI was giving direct access to the documentation of a well, stored in Schlumberger’s ESearch.

 

PDF document related to a well
An example of PDF document related to a well.

 

 

Adding Your Own Entities and Services

When data cannot be expressed as properties, tables or documents, the next step is to plug your own model. The API of the Backend SDK makes it possible to plug your own entities under any existing entity of the built-in data model. In this use case, not only code to access data needs to be developed, but also code to expose this data to the viewer. The IVAAP data model is mature, so this is a rare occurrence.

There are hundreds of services implemented with the IVAAP Data Backend SDK, developers who embark on a journey involving adding their own data types can be reassured by the fact that the path they follow is the same path the INT developers follow every day as we augment the IVAAP data model. INT makes use of its own SDK every day.

 

IVAAP Data Backend SDK Homepage
Home page of the website dedicated to the IVAAP Data Backend SDK.

 

 

Whether IVAAP customers need to pepper the IVAAP UI with proprietary properties or their own data types, these customers have options. The SDK is designed to make extensions straightforward, not just for INT’s own developers, but for INT customers as well. You do not need to contract INT’s services to roll your own extensions. You can, but you don’t have to. When IVAAP gets deployed, we don’t just give you the keys to IVAAP as an application, we also give you the keys to IVAAP as a platform, where you can independently control its capabilities.

For more information on IVAAP, please visit www.int.com/products/ivaap/

 


Deploying IVAAP Services to Google App Engine

One of the productivity features of the IVAAP Data Backend SDK is that the services developed with this SDK are container-agnostic. Practically, it means that a REST service developed on your PC using your favorite IDE and deployed locally to Apache Tomcat will run without changes on IVAAP’s Play cluster.

While the Data Backend SDK is traditionally used to serve data, it is also a good candidate when it comes to developing non-data-related services. For example, as part of IVAAP 2.8, we worked on a gridding service. In a nutshell, this service computes a grid surface based upon the positions of a top across the wells of a project. When we tested this service, we didn’t deploy it to IVAAP’s cluster; it was deployed as a standalone application, as a servlet, on a virtual machine (VM).

Deploying Apache Tomcat on a virtual machine is “old school”. Our customers are rapidly moving to the cloud, and while VMs are often a practical choice, other options are sometimes available. One of these options is Google App Engine. Google App Engine is a bit of a pioneer of cloud-based deployments. It was the first product that allowed servlet deployments that scale automatically, without having to worry about the underlying infrastructure of virtual machines. This “infinite” scalability comes with quite a few constraints, and I was curious to find out whether services developed with the IVAAP Data Backend SDK could live within these constraints (spoiler alert: it can).

Synchronous Servlet Support

The first constraint was the lack of support for asynchronous servlets. Google App Engine doesn’t support asynchronous servlets and the IVAAP servlet shipped with the SDK is strictly asynchronous. Supporting the synchronous requirements of Google App Engine didn’t take much time. The main change was to modify the concrete implementation of
com.interactive.ivaap.server.servlets.async.AbstractServiceRequest.waitForResponse
and wait on a java.util.concurrent.CountDownLatch instead of calling javax.servlet.startAsync().

Local File Access

The second constraint was the lack of a local file system. Google App Engine doesn’t let developers access the local files of the virtual machine where an application is deployed. The IVAAP Data Backend SDK typically doesn’t make much use of the local file system, except at startup when it reads its service configuration. To authorize users, the services developed with the IVAAP Data Backend SDK need to know how to validate Bearer tokens, and this validation requires the knowledge of the host name of the IVAAP Admin Backend. The Admin Backend exposes REST services for the validation of Bearer tokens. To support Google App Engine, I had to make the discovery of these configuration files pluggable so that they can be read from the WEB-INF directory of the servlet instead of a directory external to that servlet.

Persistence Mechanism

The third constraint was the lack of persistence. Google App Engine doesn’t provide a way to “remember” information between two HTTP calls. To effectively support computing services, a REST API cannot make an HTTP client “wait” for the completion of this computing. The computation might take minutes, even hours. The REST API of a computing service has to give a “ticket” number back to the client when a process starts, and provide a way for this client to observe the progress of that ticket, all the way to the completion. In a typical servlet deployment, there are many options to achieve this: the service can use the Java Heap to store the ticket information or use a database. To achieve the same result with Google App Engine, I needed to pick a persistence mechanism. For simplicity’s sake, I picked Google Cloud Storage. The state of each ticket is stored as a file in that storage. 

Background Task Executions

The fourth constraint was the lack of support for background executions. Google App Engine by itself doesn’t allow processes to execute in the background. Google however provides integration with another product called Google Cloud Tasks. Using the Google Cloud Tasks API, you can submit HTTP requests to a queue, and Google Cloud Tasks will make sure these requests get executed eventually. Essentially, when the gridding service receives an HTTP request, it creates a ticket number, submits this HTTP request immediately to Google Cloud Tasks, which in turn calls back Google App Engine. The IVAAP service recognizes that the call comes from Google Cloud Tasks and stores the result to a file in Google Cloud Storage instead of the servlet output stream. It then notifies the client that the process has completed.

Here’s a diagram that describes the complete workflow: 

INT_GCP_Workflow

Constraints and Considerations

While the SDK did provide the API to implement this workflow out of the box, getting this to work took a bit of time. I had to learn 3 Google products at once to get it working. Also, I encountered obstacles that I will share here so that other developers benefit:

  1. The first obstacle was that the Java SDK for Google App Engine requires the Eclipse IDE. There is no support for the NetBeans IDE. I am more proficient with NetBeans.
  2. The second obstacle was that I had to register my Eclipse IDE with Google so I can deploy code from that environment. It just happened that that day, the Google registration server was having issues, blocking me from making progress.
  3. The third obstacle was the use of Java 8. The Google Cloud SDK required Java 8, but Eclipse defaulted to Java 11. It took me a while to understand the arcane error messages thrown at me.
  4. The fourth obstacle was that I had to pick a flavor of Google App Engine, either “Standard” or “Flexible”. The “Standard” option is cheaper to run because it doesn’t require an instance running at all times. The “Flexible” option has less warmup time because there is always at least one instance running. There are many more differences, not all of them well documented. The two options are similar, but do not share the same API. You don’t write the same code for both environments. In the end, I picked the “Standard” option because it was the most constraining, better suited to a proof of concept.
  5. The fifth obstacle was the confusion due to the word “Promote” used by the Google SDK when deploying an instance. In this context, “Promote” doesn’t mean “advertising”, it means “production”. For a while, I couldn’t figure out why my application wouldn’t show any changes where I expected them. The answer was that I didn’t “promote” them.
  6. The last obstacle was the logging system. Google has a “Google Logging” product to access logs produced by your application. Logging is essential to debugging unruly code that you can’t run locally. Despite several weeks of use, I still haven’t figured out how this product really works. It is designed to be used to monitor an application in production, not so much for debugging. Debugging with logs is difficult. There might be several reasons why you can’t find a log. The first possibility is that the code doesn’t go where you think it’s going, and the log is not produced. The second possibility is that the log was produced, but I am too impatient, there is a significant delay and it hasn’t shown up yet. The third possibility is that it has shown up, but is nested inside some obscure hierarchy, and you won’t see it unless you expand the entire tree of logs. The log search doesn’t help much and has some strange UI quirks. I found that the most practical way to explore logs is to download them locally, then use the search capabilities of a text editor. Because the running servlet is not local to your development environment, debugging a Google App Engine application is a time-consuming activity.

In the end, the IVAAP Data Backend SDK passed this proof of concept with flying colors. Despite the constraints and obstacles of the environment, all the REST services that were written with the IVAAP Cluster in mind are compatible with Google App Engine, without any changes. Programming is hard, it’s an investment in time and resources. Developing with the IVAAP Data Backend SDK preserves your investment because it makes a minimum amount of assumptions on how and where you will run this code.

For more information or for a free demo of IVAAP, visit int.com/products/ivaap/.


INT to leverage IBM, Red Hat Open Technologies to Enhance Efficiencies in Oil & Gas Exploration and Drilling Workflows

The latest version of INT’s data visualization platform, IVAAP, integrates with IBM Open Data for Industries, IBM’s hybrid cloud implementation of the Open Group OSDU data.

HOUSTON, TX — April 15, 2021 — INT today announced that IVAAP — a leading enterprise subsurface data visualization platform — integrates with IBM Open Data for Industries for Cloud Pak for Data built on Red Hat OpenShift. These open technologies will enable the upstream industry to visualize exploration and drilling data across hybrid cloud environments to drive efficiencies and speed decision making.

As the oil and gas industry undergoes unparalleled transformation, members of the value chain are seeking solutions to maximize digital opportunities across their operations. Data is a critical asset and when leveraged with artificial intelligence, can yield mission-critical insights and competitive advantage.  However, quite often, domain-specific data exists in silos — limiting workflow efficiencies that leverage data insights for real-time decision-making.

Today’s announcement focuses on open technologies for the IVAAP platform that remove these barriers:

  • IVAAP leverages IBM Open Data for Industries for IBM Cloud Pak for Data, a data platform enabling seamless integration of data sources with the flexibility to deploy across any infrastructure that leverages The Open Group OSDU Forum data standard. The Open Group OSDU Forum is an industry organization that brings together technology leaders, producers and members of the oil and gas value chain to build a data foundation based on data standards making subsurface data easier to find, combine, and use across geoscience workflows.
  • IVAAP integrates with IBM Open Data for Industries for IBM Cloud Pak for Data powered by Red Hat OpenShift, the industry’s leading open source Kubernetes platform. This will enable INT customers to run IVAAP on-premise, unlocking speed and agility for local operations as well as remote locations through the cloud, yet all with one consistent architecture. This provides upstream operators greater value and flexibility for real-time operational decision-making, as well as consistency to the enterprise operations.

“We are extremely excited about the work we are engaging in with The Open Group OSDUTM Forum and our collaboration with IBM. For clients who manage hybrid cloud environments, our goal is to bring the technology to where they need it — in the cloud and in their locally hosted environment.”

— Olivier Lhemann, President of INT.

“Our collaboration with INT enables choice and flexibility in deploying subsurface data visualization across hybrid cloud environments. By leveraging IBM Open Data for Industries for IBM Cloud Pak for Data, INT and the oil and gas community can leverage an enterprise-grade, open-source platform based on the OSDU data foundation, improving access to subsurface data across any infrastructure, including public cloud, local country cloud and on-prem,” says Dariusz Piotrowski, Global Solutions Leader,
IBM Industrial Sector.

IVAAP is a sophisticated domain visualization and analytics platform that provides central access to data regardless of where it is — stored in the cloud and on-premise. Users can access, view, search, visualize, and analyze their subsurface data — wellbore, seismic, schematics, log, and more — in a web browser.

About INT:

INT software empowers the largest Oil & Gas (IOC/NOC) and services companies in the world to visualize their complex subsurface data (seismic, well log, reservoir, and schematics in 2D/3D). INT offers a visualization platform (IVAAP) and libraries (GeoToolkit) developers can use with their data ecosystem to deliver subsurface solutions (Exploration, Drilling, Production). INT’s powerful HTML5/JavaScript technology can be used for data aggregation, API services, high-performance visualization of G&G and petrophysical data in a browser. INT simplifies complex subsurface data visualization.

Read the press release on PRWeb.

For more information about IVAAP or INT’s other data visualization products, please visit https://www.int.com.

INT, the INT logo, and IVAAP are trademarks of Interactive Network Technologies, Inc., in the United States and/or other countries.

Red Hat, the Red Hat logo and OpenShift are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the U.S. and other countries.

 

INT Supports The Open Group OSDU™ Forum Mercury Release with Advanced Domain Data Visualization in the Cloud

As a long-standing OSDU Forum Member, INT has worked closely with the OSDU development teams to ensure seamless integration of IVAAP visualization of OSDU data on all major cloud providers.

Houston, TX – March 24, 2021 – INT is pleased to announce our partnership with The Open Group OSDU™ Forum as part of the new Mercury Release. INT’s flagship data visualization platform, IVAAP, offers a unique way for operators to search, explore, interact with, and automate their data on OSDU in a single platform in the cloud. 

Developed by The Open Group OSDU™ Forum, the OSDU Data Platform is an Open Source, standards-based and technology-agnostic data platform for the energy industry that stimulates innovation, industrializes data management, and reduces time to market for new solutions.

For companies adopting OSDU, IVAAP is a powerful, fast, and cost-effective alternative to custom building an application or assembling multiple components to visualize domain data. INT partners with all major cloud providers that support OSDU — AWS, Microsoft Azure, Google Cloud Platform, and RedHat OpenShift by IBM. And it includes multiple customization options, including an SDK to provide a complete, end-to-end visualization solution.

Olivier Lhemann, founder and president of INT, explains: “As more energy companies transition their data and workflows to the cloud, it’s more important than ever to have a common data standard. Our work with OSDU is critical to helping companies solve the challenge of interoperability, of viewing their data from a single application, eliminating silos and liberating workflows. IVAAP is a universal cloud viewer that significantly reduces time to market and accelerates the adoption of innovative technologies.”

To learn more about IVAAP and how it works with OSDU, visit INT.com/IVAAP.

Read the full press release on PRWeb.

About INT:

INT software empowers the energy companies to visualize their complex data (seismic, well log, reservoir, and schematics in 2D/3D). INT offers a visualization platform (IVAAP) and libraries (GeoToolkit) that developers can use with their data ecosystem to deliver subsurface solutions (Exploration, Drilling, Production). INT’s powerful HTML5/JavaScript technology can be used for data aggregation, API services, high-performance visualization of G&G and petrophysical data in a browser. INT simplifies complex subsurface data visualization.

About The Open Group

The Open Group is a global consortium that enables the achievement of business objectives through technology standards. Our diverse membership of more than 800 organizations includes customers, systems and solutions suppliers, tool vendors, integrators, academics, and consultants across multiple industries. For more information, visit www.opengroup.org.

INT, the INT logo, and IVAAP are trademarks of Interactive Network Technologies, Inc., in the United States and/or other countries.

 

Open Subsurface Data Universe™ and OSDU™ are trademarks of The Open Group.

How OSDU Can Help Data Management

Discussing data management challenges with major oil companies, national oil companies (NOCs), and oil services companies over the last few months, we found that it is still quite difficult to find published metrics about their KPIs. During a recent conversation with an operator, for example, he shared that finding the right data for analysis could take anywhere from one to SEVEN weeks.

data-viz-geo-challenge
Source: Lee C Lawyer Chevron Chief Geophysicist Oil & Gas Journal Nov. 4, 1991, pp 51-52.

Whether afraid releasing this data might expose inefficiencies or simply that the data is proprietary, what is interesting is that this problem persists—multiple people doing the same checks, companies can’t find their data, and what they can find, they don’t trust. For most geologists and geoscientists, data validation—validating, correcting, and verifying data—takes between 30 and 90 percent of their time before they can even begin to use the data. This challenge has become a major blocker as they transition from human analysis to Artificial Intelligence (AI) and Machine Learning (ML) to automate tasks and decisions.

The problems lie in how application systems have been built over time: in silos. To quote Teradata’s Jane McConnell in her recent blog post on OSDU, “Keeping data in separate systems with separate indexes, separate master data management issues, and often separate physical hardware, only means extra work, master data management problems, and unnecessary hassle when we try to bring the data together so we can analyze it as a whole.”

So, how do we fix it? OSDU data standardization with IVAAP data visualization is one approach.

What Is OSDU?

The Open Subsurface Data Universe™ (OSDU) Forum is an Industry Forum formed to establish an open subsurface Reference Architecture. OSDU is created around a simple idea: can you find, use, and trust your data? 

The objective is to move from a traditional model based on multiple types of data into a single integrated data model. The digital transformation approaches data as an asset that can be used throughout various stages of the workflow and applications, decoupled of the native workflow and application. 

data-viz-silos

The OSDU mission is to establish an open subsurface Reference Architecture as follows:

  • A cloud-native data platform reference architecture, with usable implementations for Microsoft Azure, Amazon AWS, and Google GCP
  • Application Standards (APIs) to ensure that all applications (microservices), developed by various parties, can run on any OSDU data platform
  • Leverage Industry Data Standards for frictionless integration and data access

OSDU Value Proposition: Access and Acceleration

To increase data accessibility, data can no longer be kept siloed. Companies must now accelerate their digital transformation by taking advantage of the growing OSDU marketplace and the rapid adoption of new solutions using OSDU APIs.

osdu-environment
OSDU standardizes and secures data currently spread across applications in different formats.
 
The OSDU data platform architecture helps separate data from its native application—from the workflow and from the storage infrastructure. It is indexed, discoverable, and consumable. This evolution is critical as we automate analysis with AI, ML, etc. Digitalization requires consistency and confidence in all data and standards for data and metadata to take out the guesswork.

Improve Findability, Collaboration, and Data Exchange

To improve not only data management workflows, but also to increase collaboration across teams, the smart approach is to combine a common data standard (OSDU) and leverage enterprise viewer technology (IVAAP).

Data Standardization: Findability and Discovery

A common data standard improves findability, using powerful search engines to reduce time to find the right data from weeks to days. When possible, it leverages data lakes to access all data from a single repository or a hybrid model to aggregate data in place. Removing the bottleneck of ”person dependent” models means immediate access and data discovery with security at the object level.

Data Visualization: Collaboration and Data Exchange

The ability to view multiple sources of data in a single dashboard is a critical piece of the new data management workflow. It makes data exchange easier, enables process optimization and better QC, and translates to better visibility for KPIs. Enabling data selection directly from the search eliminates the need to move data, and aggregating data in a single, shared dashboard means faster collaboration and better decision-making.

Empower Better Data Management

Data management can benefit from a single point of access of all subsurface data, simplifying data exchange, sharing, and consumption. For data managers, OSDU can enable new technology solutions that aggregate data, simplify search, and improve discoverability. Paired with the right subsurface data visualization platform technology, OSDU can pave the way to automate tedious tasks, workflows, and analysis, ultimately providing quicker information to stakeholders for faster decisions.

For more information or for a free demo of IVAAP, visit int.com/products/ivaap/