INT

Empowering Visualization

CONTACT US SUPPORT

ivaap

Microsoft and INT Deploy IVAAP for OSDU Data Platform on Microsoft Energy Data Services

This post was co-authored by Kadri Umay, Principal Program Manager, Microsoft.

Energy companies are currently going through a massive transformation by moving hundreds of applications to monitor, interpret, and administer their data into the cloud. In addition, they have embarked on adopting a common data standard, the OSDU™ Data Platform, to simplify interoperability between applications to facilitate data access, exchange, and collaboration.

With Microsoft Energy Data Services, energy companies can leverage new cloud-based advanced data visualization capabilities for geoscientists provided by INT and Microsoft Energy Data Services.

Microsoft Energy Data Services is a data platform fully supported by Microsoft, that enables efficient data management, standardization, liberation, and consumption in energy exploration. The solution is a hyperscale data ecosystem that leverages the capabilities of the OSDU Data Platform and Microsoft’s secure and trustworthy cloud services with our partners’ extensive domain expertise.

INT is proud to be among the early adopters who have been involved since the preview of Microsoft Energy Data Services. INT is a very active member of the OSDU Forum that offers IVAAP™, an advanced data visualization platform that allows geoscientists to easily access, interact with, and visualize data to create dashboards within Microsoft Azure, leveraging Microsoft Energy Data Services.

The IVAAP data visualization platform helps geoscientists and data scientists simplify their data work with the following features:

  • Access to the OSDU Data Platform is already supported (well, seismic, reservoir), and any other data sources from a single application in the cloud.
  • Full interoperability, which means data types aligned with the OSDU Data Platform can be extended to support custom formats and aggregate custom DDMS.
  • Intuitive, user-defined dashboards for engineers, geophysicists, and managers to visualize and interact with large datasets of well logs and seismic schematics, build data collections, and launch their machine learning—all from one place.
  • Many standard data connectors, powerful APIs, and SDKs that provide developers and architects ways to implement their own workflow easily.
  • Accelerated delivery of geoscience, drilling, and production cloud-enabled solutions with seamless support on Microsoft Azure.

“Providing a reliable, trusted platform as a service that accelerates the deployment of the OSDU Data Platform is key for any successful cloud transformation. Through the IVAAP platform’s integration with Microsoft Azure, customers will now have immediate access to these capabilities. This integration will simplify the access and provisioning of the massive amount of data generated by the energy industry, enabling impeccable and secure digital interactions. Our partnership with Microsoft in deploying Microsoft Energy Data Services is an important step toward our goal of providing reliable, cost-effective solutions for energy ISVs in the OSDU Data Platform.”

—Dr. Hughes Thevoux-Chabuel, VP Cloud Solutions, INT

Microsoft Energy Data Services is an enterprise-grade, fully managed OSDU Data Platform for the energy industry that is efficient, standardized, easy to deploy, and scalable for data management—ingesting, aggregating, storing, searching, and retrieving data. The offering will provide the scale, security, privacy, and compliance expected by our enterprise customers. The platform offers out-of-the-box compatibility with INT IVAAP, an advanced data visualization platform that allows geoscientists to easily access, interact with, and visualize the OSDU Data Platform to create dashboards with data contained in Microsoft Energy Data Services.

Learn more

Visit us online at int.com/ivaap for a preview of IVAAP or for a demo of INT’s other data visualization products. 

For more information, please visit www.int.com or contact us at intinfo@int.com.

 

How IVAAP Maximizes Use of HATEOAS Links

Ever since the concept of web services first gained popularity, developers attempting to use these web services have faced two challenges: The first challenge is finding the right service to use; the second challenge is writing the code to call these services. The goal of this article is to describe how IVAAP uses HATEOAS hypermedia links to address both problems. We’ll also try to highlight other use cases benefiting from the concept of hypermedia applied to microservices.

A Brief Description of HATEOAS

Most microservices developed today (including IVAAP’s) use a REST API. REST stands for “REpresentational State Transfer.” It’s a term coined by Roy Fielding, who is also the inventor of “Hypermedia as the Engine of Application State,” or HATEOAS. It describes REST services that use hypermedia links to describe how they relate to other services.

For example, in IVAAP, the metadata of a seismic dataset is typically accessible through a URL such as this one:

https://…/ivaap/api/ds/geofiles/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/seismic/cG9zZWlkb24gdmRzIG4gc2VneS9Qb3NlZGlvbiBkZXB0aC9wc2RuMTFfVGJzZG1GX2Z1bGxfd19BR0NfTm92MTFfdmVsNV9kZXB0aF8zMmJpdHMueGd5

Unlike the JSON content returned by a classic REST service, the JSON content returned by this IVAAP service contains more than just the requested metadata. It also contains a “links” JSON node that leads to additional information about this seismic dataset.

Picture1A sample JSON output for the IVAAP “seismic metadata” service, as shown in Google Chrome Developer Tools


In the example above, there are multiple HATEOAS links. One of them is the “Geometry” link. The purpose of the seismic geometry service is to expose the shape of seismic surveys. The URL of this service is:

https://…/ivaap/api/ds/geofiles/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/seismic/cG9zZWlkb24gdmRzIG4gc2VneS9Qb3NlZGlvbiBkZXB0aC9wc2RuMTFfVGJzZG1GX2Z1bGxfd19BR0NfTm92MTFfdmVsNV9kZXB0aF8zMmJpdHMueGd5/geometry

This Geometry service is meant to be used by applications showing seismic datasets on a map. An application leveraging HATEOAS links would typically examine the “links” returned by the “seismic metadata” service to retrieve the URL of the associated “geometry” service. An application not leveraging HATEOAS links would hardcode the logic that /geometry needs to be added to the URL of the “seismic metadata” service to get the same result.

Both approaches are valid, but the HATEOAS approach brings multiple benefits that we are going to detail.

The “Broken Link” Issue Applied to Data

If you surfed in the ’90s, you are certainly familiar with the concept of “broken links.” Back in those days, websites were made of pages maintained by hand, and text on these web pages was often peppered with underlined words (often colored in blue) leading to another page. If the target page was moved, the link would stop working and the web surfer would be greeted by an unhelpful 404 error instead.

The lesson from these early days is that web page URLs are anything but permanent. The initial idea behind HATEOAS links is that this lesson can be applied to web services, too. If an application uses a hard-coded service URL to read data, this application will immediately stop working if that web service is moved. If each microservice describes the URL of related microservices, then the application can just follow URLs instead of using hard-coded versions. The maintenance of URLs becomes a server concern, and no longer a service consumer concern.

The main issue with this concept is that its purported benefits are unproven in the real world. To prove the benefit of this “forward compatibility” approach, you’d have to observe the life-cycle of many microservices (and consumers of these microservices) over a long period of time to determine whether the use of HATEOAS links was worth it. Taking IVAAP as an example, even though the IVAAP microservices use HATEOAS links, changing the URL of an IVAAP REST API is a rare event. One of the reasons is that there is no way to enforce the use of these links on the service consumer side. HATEOAS doesn’t provide a guarantee that no consumer hard-coded any URL. It is even sometimes faster to use hard-coded URLs, for example, to restore dashboards.

The second issue is that the “broken link” issue is a narrow backward compatibility concern, focused on URL changes only. While services may move, their API might also change, and HATEOAS doesn’t provide a way to address the backward compatibility of service APIs.

Discoverability

While HATEOAS links may help address issues associated with changing URLs, HATEOAS links really shine when it comes to discoverability. A component like the IVAAP Data Backend has hundreds of microservices. It doesn’t matter if each one of these services is documented, just finding whether a service exists is a complex task. HATEOAS links clearly indicate all URLs related to the data being accessed, in a consistent manner.

IVAAP is a platform. It was designed so that INT customers can modify the user interface using the IVAAP Client SDK, and we strive to make it as easy as possible. HATEOAS links give contextual documentation of the services that are available for any server-side object that the UI is accessing. As a result, modifying the IVAAP UI doesn’t require client-side developers to discover the server-side REST API before getting started. Developers can be immediately productive.

Testing

Developing a web-based application like IVAAP has a bit of a chicken-and-egg problem. You need to start by developing the data services first, but you don’t really know how well they work until the UI consuming these services is complete.

To get ahead of the game, there are methods to unit test data services, but they are time-consuming to follow. Just building by hand the right URL to test takes time, especially with long URLs. And because data quality varies, bugs might be data-specific and you need to test a bunch of them to make sure your data services are rock solid.

 

Following HATEOAS Links with Postman

Following HATEOAS links with Postman.


A widely used tool to inspect and test web services is Postman. Postman “understands” HATEOAS links and testing your work just consists of following links within Postman, just like you would do with an HTML-based website.

The most common use case of the IVAAP Data Backend SDK is when INT customers write a connector that accesses a proprietary data source. The testing steps of such a connector are typically very fast because they don’t require the UI to be ready. Most bugs can be found immediately, just using Postman.

Going Beyond: Automatic UI Generation

Discoverability and testability are well-known benefits of including HATEOAS links in a REST API. IVAAP also uses HATEOAS links to generate part of its user interface. For example, the tree that is shown to users when they open a well is server-driven, not client-driven. The IVAAP UI parses the HATEOAS links and builds a tree of nodes based on them.

Not all wells have the same details of data. Some wells might only have a location, others might have a trajectory. The presence of relevant HATEOAS links is what gives the UI the information on which data is available for that well. The IVAAP UI doesn’t need to understand what a trajectory is to include a trajectory node under a well node. The tree is generated automatically from HATEOAS links.

The Nodes Under the “AKM-11” Well (left)

The nodes under the “AKM-11” well (left), as listed by the HATEOAS link for that well (right).


Not all HATEOAS links associated with an object are meant to be shown as nodes in the UI. By convention, only the HATEOAS links with the attribute “children” set to true should be shown. Customers who want to customize the nodes shown in the UI don’t need to write client-side code. They have complete control by just plugging their own code into the Data Backend.

The same technique is used to build the UI, allowing users to add datasets to their projects. The Data Backend advertises through HATEOAS links how data from a connector can be browsed, and the UI parses these HATEOAS links to build a matching user interface.

User Interface Generated when Listing Wells in a “mongo” Connector

User interface generated when listing wells in a “mongo” connector.


Each data source has different capabilities, and this is reflected in the user interface. Some data sources might support search by name, paging, or sorting. For example, when search by name is supported on the server side, the IVAAP UI may propose a search box. IVAAP advertises querying capabilities to the user interface by including a “supportedQueries” attribute along its HATEOAS links.

A Sample JSON Output for the IVAAP “connector” Service

A sample JSON output for the IVAAP “connector” service, as shown in Google Chrome Developer Tools.

Likewise, it is sometimes convenient to be able to edit the name of a dataset from the same user interface. Not all data sources support name editing, and it’s only when editing is supported by a connector that a relevant HATEOAS link should be included in the server responses.

A Sample JSON Output for the IVAAP “connector” Service, as Shown in Google Chrome Developer Tools

A sample JSON output for the IVAAP “connector” service, as shown in Google Chrome Developer Tools.

In the response above, not only the Data Backend advertises that data names can be edited, but it also indicates it supports data deletion.

This concept of automated UI generation using HATEOAS links is not a standard use. It requires the addition of attributes that are typically not seen in web services using HATEOAS links. They have powerful tools as they reduce the amount of work on the client side. Actually, the IVAAP REST API is designed to support more complex than the two use cases already mentioned.

A majority of the IVAAP REST services are either services returning a collection of meta-data, or services returning the meta-data of a single dataset. The JSON format of these two types of services is standardized across the IVAAP Data Backend. Because the services provide consistent JSON outputs, it is easy to write a generic UI that will browse through the entire tree of meta-data and even allow editing. In other words, you can write a basic IVAAP client from scratch without much effort on the UI side.

A completely automated UI would not be limited by the search and editing capabilities advertised in HATEOAS links. Each IVAAP REST service comes with its own documentation. This documentation is accessible in the OpenAPI 3.0 format using a standard mechanism.

In this mechanism, if the URL of the service listing wells in a MongoDB database is: https://…/ivaap/api/ds/mongo/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/wells/, the URL of its matching OpenAPI documentation would be: https://…/ivaap/api/ds/mongo/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/wells/openapispecs

A completely automated UI could expose the search parameters described in this documentation, a bit like SwaggerEditor does. This wouldn’t be limited to search, the same principle can be applied to updating and deleting data.

 

A Form Generated Automatically by SwaggerEditor

A form generated automatically by SwaggerEditor from the OpenAPI specification of the wells service.

 

Going Beyond: Batch Support

Another feature enabled by HATEOAS links is the ability to fetch multiple aspects of a dataset in one HTTP call.

Microservices work best when they do only one thing at a time, but this means the IVAAP client needs to make multiple calls to the Data Backend to restore a dashboard. Currently, Google Chrome only allows up to 6 concurrent HTTP connections per host, sometimes forcing the client to “wait” for the availability of connections. This has a direct impact on the user experience.

To help with this, the IVAAP Data Backend provides a so-called “Batch” REST API to retrieve the content behind multiple URLs in one go. Other servers also have this feature, but what’s different about IVAAP’s Batch API is that it allows developers to leverage HATEOAS links.

For example, if you are building a data map and need to retrieve the metadata of a seismic dataset along with its outline, you would specify to the Batch REST API that you need to retrieve the content behind  https://…/ivaap/api/ds/geofiles/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/seismic/cG9zZWlkb24gdmRzIG4gc2VneS9Qb3NlZGlvbiBkZXB0aC9wc2RuMTFfVGJzZG1GX2Z1bGxfd19BR0NfTm92MTFfdmVsNV9kZXB0aF8zMmJpdHMueGd5 as well as the content behind the associated “Geometry” HATEOAS link. This method of fetching multiple aspects of a dataset at once is much more expressive than passing multiple URLs to the Batch REST API.

Something that should be noted when it comes to performance is that we made the HATEOAS links an optional feature of IVAAP. Consumers of the IVAAP Data Backend API who don’t use these links can opt to reduce the size of the JSON payload between the client and the server. The default behavior of the Data Backend is to include HATEOAS links, but the collection services can be called in a way that excludes these links completely or only includes specific, named links.

Conclusion

HATEOAS links have been a part of the IVAAP Data Backend since day one. Over time, we found that they pack much more functionality than we initially thought. All these features have a common goal: facilitating the work of the UI developers and accelerating the delivery of software. While I used examples from IVAAP, the ideas in this article can easily be applied to your own data backend.

For more information or for a free demo of IVAAP, visit int.com/products/ivaap/.

INT and ANPG Improve Collaboration on Oil & Gas Concessions by Creating Virtual Data Room Leveraging IVAAP Technology

The ANPG VDR solution—implemented with a unique partnership between MIAPIA, SATEC and INT—enables companies to remotely assess the potential and requirements for better block concession management.

INT announced today the completion of a Virtual Data Room (VDR) for the Angola’s National Agency for Oil, Gas and Biofuels (Agência Nacional de Petróleo, Gás e Biocombustíveis) (“ANPG”), developed through collaboration with SATEC and MIAPIA.

The ANPG VDR is a cloud platform based on INT’s IVAAP subsurface data visualization platform and implemented with and by SATEC and MIAPIA. The platform allows secure access to E&P data managed by the Gabinete de Archivo de Datos (Data Management Department) of ANPG. The platform restricts user access to specific, predefined data sets.

Some of the VDR’s greatest functionalities are the support for the most important objects and formats in the sector (well logs, 2D and 3D seismic, trajectories, or lithology) through connectors to databases that store E&P information in the most common file formats: LAS, DLIS, SEG-Y. It also allows the visualization and research of these data on a GIS platform and access and preview of technical reports in PDF format.

“ANPG is very pleased with the implementation of the Virtual Data Room solution, a strategic tool for the promotion of Angola’s oil potential and the attraction of foreign investment, as well as the digital transformation of the organization itself,” said Lúmen Sebastião, GAD Director, ANPG.

To date, the VDR supplies E&P data related to offshore blocks, Baixo Congo Basin, and Kwanza Basin. More than 20 VDR sessions have been held, with the participation of 15 companies. Furthermore, ANPG is participating in benchmarking activities, sharing with colleagues from other countries their experience with the VDR solution.

To learn more about the ANPG VDR, visit int.com/anpg-vdr/ or for a preview of IVAAP, visit int.com/ivaap/.

For more information, please contact us at intinfo@int.com.

____________
ABOUT INT
INT software empowers energy companies to visualize their complex data (geoscience, well, surface reservoir, equipment in 2D/3D). INT offers a visualization platform (IVAAP) and libraries (GeoToolkit) that developers can use with their data ecosystem to deliver subsurface solutions (Exploration, Drilling, Production). INT’s powerful HTML5/JavaScript technology can be used for data aggregation, API services, and high-performance visualization of G&G and energy data in a browser. INT simplifies complex subsurface data visualization.

INT, the INT logo, and IVAAP are trademarks of Interactive Network Technologies, Inc., in the United States and/or other countries.

How Apache SIS Simplifies the Hidden Complexity of Coordinate Systems in IVAAP

See how Apache SIS, with IVAAP, helps support our client’s coordinate systems by using less code.

With the recent release of IVAAP 2.9, now is a good time to reflect on the help we got along the way. One of the components that made IVAAP possible is the Apache SIS library. The goal of this blog article is to bring more visibility to this awesome Java library.

What Is the Apache SIS Library?

Apache SIS is an open-source library written in Java that makes it easier to develop geospatial applications. Many datasets have metadata designating their location on Earth, and these locations are relative to a datum and a map projection method. There are many datums and many map projection methods. Apache SIS facilitates their identification and the accurate conversion of coordinates between them.

What’s a Datum and What’s a Map Projection Method?

Most people are familiar with latitude and longitude coordinates. This geographic coordinate system has been used for maritime and land-based navigation for centuries. Since the late 1800s, the line defining 0º of longitude has been the Prime meridian, crossing the location of the Royal Observatory in Greenwich, England. This meridian defined one axis, from South to North. The equator defined the other axis, from West to East. The origin point of this system on the Earth’s surface is in the Gulf of Guinea, 600 km off the coast of West Africa.

The traditional geographic coordinate system
The traditional geographic coordinate system (Source)

 

Similarly, a datum defines the origin point of the coordinate axes on the Earth’s surface and defines the direction of the axes. To account for the fact that the Earth is not a perfect sphere, a datum also describes the generalized shape of the Earth. For example, WGS 84 (World Geodetic System 1984) is a widely-used global datum based on latitudes and longitudes where the Earth is modeled as an oblate spheroid, centered around its center of mass.

The WGS 84 reference frame. The oblateness of the ellipsoid is exaggerated in this image. (Source)

 

WGS 84 is used by GPS receivers and the longitude 0º of this datum is actually 335 ft east of the Greenwich meridian.

While universal latitude and longitude coordinates are convenient, they are not universally practical because of land masses drift. Satellite measurements show that the location of Houston relative to the WGS 84 datum changes by 1 inch each year. A local datum is a more pragmatic choice than a global datum because distances from a local point of reference are smaller and don’t change over the years when all locations are on the same tectonic plate. A local datum may also align its spheroid to closely fit the Earth’s surface in this particular area.

A map projection method indicates how the Earth’s surface is flattened into a plane in order to make a 2D map. The most widely known projection method was presented by Gerardus Mercator in 1569. This is a global cylindrical projection method. It preserves local directions and shapes but distorts sizes away from the equator.

Cylindrical Projection
An example of global cylindrical projection (Source)

 

In the US, the Lambert Conformal Conic projection has become a standard projection for mapping large areas. This is a projection that requires multiple parameters, defining the longitude and latitude of its center, a distance offset to this center, and the latitude of its southern and northern parallels.

Conical Projection
An example of local conical projection (Source)

 

When a datum and a projection are used together, they define a projected coordinate reference system. While local systems limit distortions, they are only valid in a small area, an area known as the ”area of use” where a minimum level of accuracy is guaranteed.

Select Coordinate System
A screenshot from INTViewer showing the area of use of NAD27 / Wyoming East Central, a derived projected coordinate reference system

 

How Does Apache SIS Help IVAAP?

To show geoscience datasets on one of IVAAP’s 2D map widgets, you need to use a common visualization coordinate reference system.

IVAAP Screenshot
An IVAAP screenshot showing the location of wells on a map

 

This is where Apache SIS helps us: It understands the properties of both the data and visualization systems and is able to convert coordinates between them.

The math to perform these conversions is complex, this is not something you want to implement on your own. It requires specialized skills, both as a programmer and a domain expert. And just beyond the math, the number of datums and projection methods is mind-boggling. Many historical surveys are still in use today. For example, there are two datums used for making horizontal measurements in North America: the North American Datum of 1927 (NAD 27) and the North American Datum of 1983 (NAD 83). The two datums are based on two different ellipsoid models. As a result, the two datums have grid shifts of up to 100 meters, depending on location. IVAAP is able to visualize datasets that used NAD 27 as a reference, and it is Apache SIS that makes it possible to accurately reproject these coordinates into modern coordinate systems, accounting for their respective datum shift.

The datum shift between NAD 27 and NAD 83 (Source)

 

The oil and gas industry is at the origin of some of these local coordinate systems. Many of today’s new oil fields are in remote areas, initially lacking a geographical survey. There is an organization called the “OGP Surveying and Positioning Committee” which keeps track of these coordinate systems. It is colloquially known as “EPSG” for historical reasons. It regularly provides a database of these coordinate systems to all its members. This database is used by IVAAP and Apache SIS provides a simple API to take advantage of it. Each record in this database has a numerical WKID (Well Known ID). To instantiate a projection method or a coordinate system defined in this database, you just need to prefix this id with the “EPSG:” string.

OperationMethod method = getCoordinateOperationFactory().getOperationMethod("EPSG:9807"); // Transverse Mercator method

CoordinateReferenceSystem crs = CRS.forCode("EPSG:32056”);

 

The EPSG database itself is extensive, but it is common for INT customers to use unlisted coordinate reference systems, created for brand new oil fields. In these cases, a WKT (Well Known Text) string can be used instead. This text is a human-readable description of a projection method or coordinate system. The Apache SIS provides a clean API to parse WKTs. It also provides an API for formula-based projection methods that can’t be described by a WKT.

PROJCS["NAD27 / Wyoming East Central",
    GEOGCS["NAD27",
        DATUM["North_American_Datum_1927",
            SPHEROID["Clarke 1866",6378206.4,294.9786982139006,
                AUTHORITY["EPSG","7008"]],
            AUTHORITY["EPSG","6267"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4267"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",40.66666666666666],
    PARAMETER["central_meridian",-107.3333333333333],
    PARAMETER["scale_factor",0.999941177],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["US survey foot",0.3048006096012192,
        AUTHORITY["EPSG","9003"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    AUTHORITY["EPSG","32056"]]

The WKT of NAD27 / Wyoming East Central, with the WKID 32056

Why Did INT Choose Apache SIS Over Other Options?

INT had previous experience using GeoTools. Similarly to Apache SIS, GeoTools is a Java library dedicated to facilitating the implementation of geographical information systems. Being an older library, it goes much further than Apache SIS. For example, one of its components allows the parsing of shape files, something currently outside of the scope of Apache’s library. As a matter of fact, the first versions of IVAAP were using GeoTools for coordinate conversions.

One of the issues we encountered with GeoTools is that it is a library that provides only fine-grained Java conversion APIs. There are several paths to convert coordinates between two systems, and GeoTools allows the developer to choose the best method. Choosing the “best” method without human interaction is complex; it depends on the extent of the data being manipulated and the “area of use” of each coordinate reference system involved. It also depends on the availability of well-known transformation algorithms between datums. In North America, the standard for transformations between datums was formerly known as NADCON. The rest of the world uses a standard known as NTV2. Apache SIS works with both datum shift standards. It may elect to use WGS 84 as a hub when no datum shift is applicable. An algorithm to pick the best method would require a significant amount of code for INT to write and maintain. While Apache SIS allows fine-grained control over the different transformations used when converting from one coordinate reference system into another, it also provides a high-level API to perform this conversion. The picking of the best algorithm is part of the Apache SIS’ implementation. Its high-level Java API that picks a conversion algorithm matches IVAAP’s general use microservice for the same function. To pick the right algorithm, it only takes 3 parameters:

  • A definition of the “from” coordinate system
  • A definition of the “to” coordinate system
  • A description of the “extent” of the coordinates to convert
double x = …
double y = …
GeographicBoundingBox extentInLongLat = …
DirectPosition position = new DirectPosition2D(x, y);
CoordinateReferenceSystem fromCrs = CRS.forCode("EPSG:32056");
CoordinateReferenceSystem toCrs = CRS.forCode("EPSG:3737");
CoordinateReferenceSystem displayOrientedFromCrs = AbstractCRS.castOrCopy(fromCrs).forConvention(AxesConvention.DISPLAY_ORIENTED);
CoordinateReferenceSystem displayOrientedToCrs = AbstractCRS.castOrCopy(toCrs).forConvention(AxesConvention.DISPLAY_ORIENTED);
CoordinateOperation operation = CRS.findOperation(displayOrientedFromCrs, displayOrientedToCrs, extentInLongLat);
MathTransform mathTransform = operation.getMathTransform();
double[] coordinate = mathTransform.transform(position, position).getCoordinate();

Sample code to convert a single x, y position from “NAD27 / Wyoming East Central” to “NAD83 / Wyoming East Central”

We still use GeoTools for other parts, but as a general rule, the Apache SIS Java API tends to be simpler, more modern than GeoTools when it comes to manipulating coordinates and coordinate systems.

After 3 years of use, we are happy with our decision to move to Apache SIS. This library allows us to support more of our customers’ coordinate systems, with less code. We are also planning to use it to interpret the metadata of GeoTIFF files. The support has been excellent. When we needed help, the members of the Apache SIS development team were really keen to help us. This is one of the reasons why INT felt we needed to give back to the open-source community. Being a long-time member of OSDU, INT contributed to OSDU a coordinate conversion library built on top of Apache SIS. This coordinate conversion library converts GeoJSON and trajectory stations between different coordinate reference systems. Users can specify the specific transformation steps that will be used in the conversion process, either through EPSGs or WKTs. Behind the scenes, it’s the Apache SIS’ fine-grained API that is being used.

An Inside Look at the New Release of the IVAAP Data Backend SDK

INT recently announced the release of IVAAP 2.9. We sat down with Thierry Danard, INT’s VP of Core Platform Technologies, for a quick chat about what this release includes for the Data Backend SDK.

 

Hi Thierry, what can you tell us about IVAAP 2.9 for the Data Backend?

There are a few low-level changes that won’t make it to the release notes, but that will make a difference for users. For example, we optimized the data caching mechanism. Instead of being based on the number of datasets, the data caches are now based on the size of these datasets. This typically means lower latency as more datasets tend to be cached within the same memory space. This is a feature we initially used only with WITSML wells that we have extended to other connectors and data types.

At a high level, we added a powerful “tiled images” feature. Some INT customers have large rasters that they need to visualize, or that their customers need to visualize as part of a portal. These images are stored as good old TIFF image files on a local file system or in the cloud. To visualize these images in IVAAP, you just need to point at the location of these tiles, and IVAAP does the rest. While the idea of visualizing images online might seem mundane, there is quite a lot of technology to make this happen.

First, these image files are LARGE. They can be up to 4 GB in TIFF format, and that’s a compressed format! To make the visualization of these images seamless, you need to be really good at:

  1. Downloading these files from the cloud, fast
  2. Reading these images in their native format
  3. Rendering these images as small tiles and sending these tiles to the viewer efficiently
  4. Stitching these images back together as one raster on the viewer side
  5. Caching these images long enough, but not too long
  6. Doing all this concurrently, for a large number of simultaneous users

That’s the technology side of it. From a business perspective, what’s really interesting is that there is no preprocessing of the images required. The ingestion workflow requires no specialized tooling. There’s no need to “precut” each TIFF as multiple tiles at multiple resolution levels. The “cutting” is all done on the fly and it’s seamless to end users. This is particularly important to keep storage to a minimum when you migrate your image library to the cloud. When each TIFF file takes gigabytes, a library of only a few thousand files already has a significant footprint, so you don’t want to expand that footprint with file duplication.

 

Is this image tiling feature something I could use for my own image storage?

Yes. The technology we developed makes abstraction of the storage mechanism possible. The reading, cutting, rendering, and caching are part of the IVAAP Data Backend SDK’s API. You can reuse these parts in your code.

 

What’s the purpose of this SDK, and how does it apply to images?

One of the strengths of IVAAP is that it allows the visualization of data from multiple data sources. For example, just for visualization of wells, IVAAP supports Peloton WellView, PPDM, WITSML, LAS files in the cloud, etc. These data sources typically share a common data model, but what’s different between them is the medium or protocol to access this data. The typical use case of the IVAAP Data Backend SDK is to facilitate the implementation of that access layer.

Images are treated as data. Just like we have “well” web services, we have “image” web services. The image web services are generic in nature—it’s the same service code running regardless of their underlying storage mechanism. It’s only the access code that differs. As a matter of fact, we implemented access to four types of image stores on top of our SDK:

  • Images stored locally on disk
  • Images stored in Amazon S3
  • Images stored in Microsoft Azure Blob Storage
  • Images stored in Google Cloud Storage

In this particular example, we used our own SDK to develop both the data layer and the web service layer.

 

Is the IVAAP Data Backend SDK a web framework?

While this SDK does provide an API to create your own web services, it goes well beyond that. If you call the IVAAP Data Backend SDK a framework, then it’s a highly specialized framework. And it’s not just a web framework, it’s also a data framework. Also, web frameworks tend to force you into a container. To scale your application, you are limited by the capabilities of this container. The IVAAP Data Backend SDK abstracts the container away, giving customers multiple options for how to deploy it and scale it.

From a technical perspective, a good SDK should empower developers. Often, this means requiring the least amount of effort to get the job done. From a business perspective, a good SDK should reduce development costs. Calling the IVAAP Data Backend SDK “yet another web framework” is missing the point of its value. Its value is not about doing the same thing as your favorite “other framework,” it’s about fitting your use case in the most effective way.

 

Then … how is the IVAAP Data Backend SDK a good fit?

Obviously, the integration with the rest of IVAAP is a strong benefit. You could write your data access object (DAO) layer using the API of your choice, but you would still need to write more mundane aspects, such as:

  • Exposing this data in a REST API, following the exact protocol that the IVAAP client expects. There would be hundreds of REST services to implement.
  • Integrating with services from the Admin server. For example, how users are identified, how to find out who has access to what, what are the datasets within a project, etc.

The second fit is that the SDK is designed to work with geoscience data. It integrates the data models of multiple data types, such as wells, seismic, surfaces … and raster images. While the IVAAP Data Backend SDK can be used outside of a geoscience context, a good portion of its API is geoscience-specific.

During design, we were particularly careful not to make any assumptions about how our customers’ geoscience data is stored. No matter how exotic your data systems might be, the IVAAP Data Backend will be able to access them. We are not just talking about storage, we are also talking about real-time feeds and machine learning (ML) workflows. We verified this multiple times. One particularly visible example is OSDU. INT has been on the leading edge of OSDU development, following the multiple iterations or flavors of OSDU services without requiring a change to our SDK.

 

How is the IVAAP Data Backend SDK effective?

Its API is quite simple to learn. Multiple INT customers have used this SDK to develop their own IVAAP customizations, and its API has always been well received. One aspect that is particularly liked is the lookup architecture. Plugging a class requires no XML, just one Java annotation, and it’s the same annotation used across the entire API. It’s an elegant mechanism, easy to learn, and even easier to use.

The real proof of the IVAAP Data Backend SDK’s effectiveness is the time it takes external developers to learn it and develop their own connector. We tested the effectiveness of the SDK by asking new hires to develop a connector, only armed with their knowledge of Java. Without any prior geoscience knowledge, the average time to get that connector up and running has been 2 weeks.

Another way that the backend keeps developers effective is by not taking away their favorite development tool. While the IVAAP Data Backend uses a powerful cluster architecture in production deployments, the typical developer’s day is with their favorite IDE (NetBeans, Eclipse, IntelliJ, etc.) and a well-known application server (Tomcat, Glassfish). Development is much faster when you don’t need to launch an entire cluster for a simple debugging session.

Effectiveness and fit are key. The goal of a typical framework is to help you get started faster. It provides a shortcut to skip the implementation of the mundane concerns of an application. In IVAAP’s case, for many customers, the application itself is already written. For these customers, the IVAAP Data Backend SDK helps you get finished faster instead. It provides customers with the API to finish the last mile of an IVAAP deployment, the access layer to their proprietary data store.

For more information or for a free demo of IVAAP, visit int.com/products/ivaap/.