INT

Empowering Visualization

CONTACT US SUPPORT

Products

How IVAAP Maximizes Use of HATEOAS Links

Ever since the concept of web services first gained popularity, developers attempting to use these web services have faced two challenges: The first challenge is finding the right service to use; the second challenge is writing the code to call these services. The goal of this article is to describe how IVAAP uses HATEOAS hypermedia links to address both problems. We’ll also try to highlight other use cases benefiting from the concept of hypermedia applied to microservices.

A Brief Description of HATEOAS

Most microservices developed today (including IVAAP’s) use a REST API. REST stands for “REpresentational State Transfer.” It’s a term coined by Roy Fielding, who is also the inventor of “Hypermedia as the Engine of Application State,” or HATEOAS. It describes REST services that use hypermedia links to describe how they relate to other services.

For example, in IVAAP, the metadata of a seismic dataset is typically accessible through a URL such as this one:

https://…/ivaap/api/ds/geofiles/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/seismic/cG9zZWlkb24gdmRzIG4gc2VneS9Qb3NlZGlvbiBkZXB0aC9wc2RuMTFfVGJzZG1GX2Z1bGxfd19BR0NfTm92MTFfdmVsNV9kZXB0aF8zMmJpdHMueGd5

Unlike the JSON content returned by a classic REST service, the JSON content returned by this IVAAP service contains more than just the requested metadata. It also contains a “links” JSON node that leads to additional information about this seismic dataset.

Picture1A sample JSON output for the IVAAP “seismic metadata” service, as shown in Google Chrome Developer Tools


In the example above, there are multiple HATEOAS links. One of them is the “Geometry” link. The purpose of the seismic geometry service is to expose the shape of seismic surveys. The URL of this service is:

https://…/ivaap/api/ds/geofiles/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/seismic/cG9zZWlkb24gdmRzIG4gc2VneS9Qb3NlZGlvbiBkZXB0aC9wc2RuMTFfVGJzZG1GX2Z1bGxfd19BR0NfTm92MTFfdmVsNV9kZXB0aF8zMmJpdHMueGd5/geometry

This Geometry service is meant to be used by applications showing seismic datasets on a map. An application leveraging HATEOAS links would typically examine the “links” returned by the “seismic metadata” service to retrieve the URL of the associated “geometry” service. An application not leveraging HATEOAS links would hardcode the logic that /geometry needs to be added to the URL of the “seismic metadata” service to get the same result.

Both approaches are valid, but the HATEOAS approach brings multiple benefits that we are going to detail.

The “Broken Link” Issue Applied to Data

If you surfed in the ’90s, you are certainly familiar with the concept of “broken links.” Back in those days, websites were made of pages maintained by hand, and text on these web pages was often peppered with underlined words (often colored in blue) leading to another page. If the target page was moved, the link would stop working and the web surfer would be greeted by an unhelpful 404 error instead.

The lesson from these early days is that web page URLs are anything but permanent. The initial idea behind HATEOAS links is that this lesson can be applied to web services, too. If an application uses a hard-coded service URL to read data, this application will immediately stop working if that web service is moved. If each microservice describes the URL of related microservices, then the application can just follow URLs instead of using hard-coded versions. The maintenance of URLs becomes a server concern, and no longer a service consumer concern.

The main issue with this concept is that its purported benefits are unproven in the real world. To prove the benefit of this “forward compatibility” approach, you’d have to observe the life-cycle of many microservices (and consumers of these microservices) over a long period of time to determine whether the use of HATEOAS links was worth it. Taking IVAAP as an example, even though the IVAAP microservices use HATEOAS links, changing the URL of an IVAAP REST API is a rare event. One of the reasons is that there is no way to enforce the use of these links on the service consumer side. HATEOAS doesn’t provide a guarantee that no consumer hard-coded any URL. It is even sometimes faster to use hard-coded URLs, for example, to restore dashboards.

The second issue is that the “broken link” issue is a narrow backward compatibility concern, focused on URL changes only. While services may move, their API might also change, and HATEOAS doesn’t provide a way to address the backward compatibility of service APIs.

Discoverability

While HATEOAS links may help address issues associated with changing URLs, HATEOAS links really shine when it comes to discoverability. A component like the IVAAP Data Backend has hundreds of microservices. It doesn’t matter if each one of these services is documented, just finding whether a service exists is a complex task. HATEOAS links clearly indicate all URLs related to the data being accessed, in a consistent manner.

IVAAP is a platform. It was designed so that INT customers can modify the user interface using the IVAAP Client SDK, and we strive to make it as easy as possible. HATEOAS links give contextual documentation of the services that are available for any server-side object that the UI is accessing. As a result, modifying the IVAAP UI doesn’t require client-side developers to discover the server-side REST API before getting started. Developers can be immediately productive.

Testing

Developing a web-based application like IVAAP has a bit of a chicken-and-egg problem. You need to start by developing the data services first, but you don’t really know how well they work until the UI consuming these services is complete.

To get ahead of the game, there are methods to unit test data services, but they are time-consuming to follow. Just building by hand the right URL to test takes time, especially with long URLs. And because data quality varies, bugs might be data-specific and you need to test a bunch of them to make sure your data services are rock solid.

 

Following HATEOAS Links with Postman

Following HATEOAS links with Postman.


A widely used tool to inspect and test web services is Postman. Postman “understands” HATEOAS links and testing your work just consists of following links within Postman, just like you would do with an HTML-based website.

The most common use case of the IVAAP Data Backend SDK is when INT customers write a connector that accesses a proprietary data source. The testing steps of such a connector are typically very fast because they don’t require the UI to be ready. Most bugs can be found immediately, just using Postman.

Going Beyond: Automatic UI Generation

Discoverability and testability are well-known benefits of including HATEOAS links in a REST API. IVAAP also uses HATEOAS links to generate part of its user interface. For example, the tree that is shown to users when they open a well is server-driven, not client-driven. The IVAAP UI parses the HATEOAS links and builds a tree of nodes based on them.

Not all wells have the same details of data. Some wells might only have a location, others might have a trajectory. The presence of relevant HATEOAS links is what gives the UI the information on which data is available for that well. The IVAAP UI doesn’t need to understand what a trajectory is to include a trajectory node under a well node. The tree is generated automatically from HATEOAS links.

The Nodes Under the “AKM-11” Well (left)

The nodes under the “AKM-11” well (left), as listed by the HATEOAS link for that well (right).


Not all HATEOAS links associated with an object are meant to be shown as nodes in the UI. By convention, only the HATEOAS links with the attribute “children” set to true should be shown. Customers who want to customize the nodes shown in the UI don’t need to write client-side code. They have complete control by just plugging their own code into the Data Backend.

The same technique is used to build the UI, allowing users to add datasets to their projects. The Data Backend advertises through HATEOAS links how data from a connector can be browsed, and the UI parses these HATEOAS links to build a matching user interface.

User Interface Generated when Listing Wells in a “mongo” Connector

User interface generated when listing wells in a “mongo” connector.


Each data source has different capabilities, and this is reflected in the user interface. Some data sources might support search by name, paging, or sorting. For example, when search by name is supported on the server side, the IVAAP UI may propose a search box. IVAAP advertises querying capabilities to the user interface by including a “supportedQueries” attribute along its HATEOAS links.

A Sample JSON Output for the IVAAP “connector” Service

A sample JSON output for the IVAAP “connector” service, as shown in Google Chrome Developer Tools.

Likewise, it is sometimes convenient to be able to edit the name of a dataset from the same user interface. Not all data sources support name editing, and it’s only when editing is supported by a connector that a relevant HATEOAS link should be included in the server responses.

A Sample JSON Output for the IVAAP “connector” Service, as Shown in Google Chrome Developer Tools

A sample JSON output for the IVAAP “connector” service, as shown in Google Chrome Developer Tools.

In the response above, not only the Data Backend advertises that data names can be edited, but it also indicates it supports data deletion.

This concept of automated UI generation using HATEOAS links is not a standard use. It requires the addition of attributes that are typically not seen in web services using HATEOAS links. They have powerful tools as they reduce the amount of work on the client side. Actually, the IVAAP REST API is designed to support more complex than the two use cases already mentioned.

A majority of the IVAAP REST services are either services returning a collection of meta-data, or services returning the meta-data of a single dataset. The JSON format of these two types of services is standardized across the IVAAP Data Backend. Because the services provide consistent JSON outputs, it is easy to write a generic UI that will browse through the entire tree of meta-data and even allow editing. In other words, you can write a basic IVAAP client from scratch without much effort on the UI side.

A completely automated UI would not be limited by the search and editing capabilities advertised in HATEOAS links. Each IVAAP REST service comes with its own documentation. This documentation is accessible in the OpenAPI 3.0 format using a standard mechanism.

In this mechanism, if the URL of the service listing wells in a MongoDB database is: https://…/ivaap/api/ds/mongo/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/wells/, the URL of its matching OpenAPI documentation would be: https://…/ivaap/api/ds/mongo/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/wells/openapispecs

A completely automated UI could expose the search parameters described in this documentation, a bit like SwaggerEditor does. This wouldn’t be limited to search, the same principle can be applied to updating and deleting data.

 

A Form Generated Automatically by SwaggerEditor

A form generated automatically by SwaggerEditor from the OpenAPI specification of the wells service.

 

Going Beyond: Batch Support

Another feature enabled by HATEOAS links is the ability to fetch multiple aspects of a dataset in one HTTP call.

Microservices work best when they do only one thing at a time, but this means the IVAAP client needs to make multiple calls to the Data Backend to restore a dashboard. Currently, Google Chrome only allows up to 6 concurrent HTTP connections per host, sometimes forcing the client to “wait” for the availability of connections. This has a direct impact on the user experience.

To help with this, the IVAAP Data Backend provides a so-called “Batch” REST API to retrieve the content behind multiple URLs in one go. Other servers also have this feature, but what’s different about IVAAP’s Batch API is that it allows developers to leverage HATEOAS links.

For example, if you are building a data map and need to retrieve the metadata of a seismic dataset along with its outline, you would specify to the Batch REST API that you need to retrieve the content behind  https://…/ivaap/api/ds/geofiles/v1/sources/a8b05811-6409-43bb-8902-c9142ab48cff/seismic/cG9zZWlkb24gdmRzIG4gc2VneS9Qb3NlZGlvbiBkZXB0aC9wc2RuMTFfVGJzZG1GX2Z1bGxfd19BR0NfTm92MTFfdmVsNV9kZXB0aF8zMmJpdHMueGd5 as well as the content behind the associated “Geometry” HATEOAS link. This method of fetching multiple aspects of a dataset at once is much more expressive than passing multiple URLs to the Batch REST API.

Something that should be noted when it comes to performance is that we made the HATEOAS links an optional feature of IVAAP. Consumers of the IVAAP Data Backend API who don’t use these links can opt to reduce the size of the JSON payload between the client and the server. The default behavior of the Data Backend is to include HATEOAS links, but the collection services can be called in a way that excludes these links completely or only includes specific, named links.

Conclusion

HATEOAS links have been a part of the IVAAP Data Backend since day one. Over time, we found that they pack much more functionality than we initially thought. All these features have a common goal: facilitating the work of the UI developers and accelerating the delivery of software. While I used examples from IVAAP, the ideas in this article can easily be applied to your own data backend.

For more information or for a free demo of IVAAP, visit int.com/products/ivaap/.

INT and ANPG Improve Collaboration on Oil & Gas Concessions by Creating Virtual Data Room Leveraging IVAAP Technology

The ANPG VDR solution—implemented with a unique partnership between MIAPIA, SATEC and INT—enables companies to remotely assess the potential and requirements for better block concession management.

INT announced today the completion of a Virtual Data Room (VDR) for the Angola’s National Agency for Oil, Gas and Biofuels (Agência Nacional de Petróleo, Gás e Biocombustíveis) (“ANPG”), developed through collaboration with SATEC and MIAPIA.

The ANPG VDR is a cloud platform based on INT’s IVAAP subsurface data visualization platform and implemented with and by SATEC and MIAPIA. The platform allows secure access to E&P data managed by the Gabinete de Archivo de Datos (Data Management Department) of ANPG. The platform restricts user access to specific, predefined data sets.

Some of the VDR’s greatest functionalities are the support for the most important objects and formats in the sector (well logs, 2D and 3D seismic, trajectories, or lithology) through connectors to databases that store E&P information in the most common file formats: LAS, DLIS, SEG-Y. It also allows the visualization and research of these data on a GIS platform and access and preview of technical reports in PDF format.

“ANPG is very pleased with the implementation of the Virtual Data Room solution, a strategic tool for the promotion of Angola’s oil potential and the attraction of foreign investment, as well as the digital transformation of the organization itself,” said Lúmen Sebastião, GAD Director, ANPG.

To date, the VDR supplies E&P data related to offshore blocks, Baixo Congo Basin, and Kwanza Basin. More than 20 VDR sessions have been held, with the participation of 15 companies. Furthermore, ANPG is participating in benchmarking activities, sharing with colleagues from other countries their experience with the VDR solution.

To learn more about the ANPG VDR, visit int.com/anpg-vdr/ or for a preview of IVAAP, visit int.com/ivaap/.

For more information, please contact us at intinfo@int.com.

____________
ABOUT INT
INT software empowers energy companies to visualize their complex data (geoscience, well, surface reservoir, equipment in 2D/3D). INT offers a visualization platform (IVAAP) and libraries (GeoToolkit) that developers can use with their data ecosystem to deliver subsurface solutions (Exploration, Drilling, Production). INT’s powerful HTML5/JavaScript technology can be used for data aggregation, API services, and high-performance visualization of G&G and energy data in a browser. INT simplifies complex subsurface data visualization.

INT, the INT logo, and IVAAP are trademarks of Interactive Network Technologies, Inc., in the United States and/or other countries.

How to Generate Dynamic Forms Using JSON in IVAAP

In late 2020, INT decided to make it easier for IVAAP developers and SDK users to quickly and efficiently create forms and dialogs. After working with the teams to determine the best way, the solution became clear: JSON forms. 

What Are JSON Forms?

So, what is a JSON form? First, JSON is an open standard file format and data interchange format that uses text to store and transmit data objects. This format allows users to create JSON Forms using JSON Schema. JSON Schema is a vocabulary that allows you to annotate and validate JSON documents. The schema itself will annotate the data model. 

As human beings, we can read the schema ourselves and know what properties are included in a data model, what the possible values are, etc. Programmatically with a schema, we can validate our data to make sure that the data model we’re taking as input or providing as output is in the correct shape it needs to be for the purpose it needs to serve.

How Do You Format a JSON Form? 

Formatting or shaping user data is always going to depend on the requirements of what you’re developing. So for an actual example, let’s say for our use-case we have a user, and we want to represent some data about that user with JSON. What’s the correct way to format or shape the user data? 

For our example, let’s say we have two different ways to represent the same basic qualities about a user, with their name and age. The first format just has the name as a single string, with their first and last name and also age as an integer. Using the first example, but this time, the name property itself is an object with a first name property and the last name property, which are both strings. Now with this format, I can write a JSON Schema to precisely define what I’ll consider valid and anything that’s structured any other way is invalid. So you can see that my schema says I’m expecting an object, that object will have a name property and an age property that’s an integer. But with the name, I’m expecting that that property itself should be an object with separate properties for first and last names, that are both string types.

So now you have your concrete data model and the schema that defines which format is valid. The final piece that’s needed to be able to actually use this is a validator. Using your validator, you provide the schema and then pass your data, where it can process it as either valid or invalid and give you an output for you to handle accordingly. This pattern isn’t particular to just JSON either, there are schema specifications and validators, for example, for XML documents.

How Does It Work?

So, with all that covered, you can now put it all together to explain how JSON Forms works and what’s required to make it work.

The package is split into a core library and submodules for each of the three currently-most-popular front-end frameworks — React, Vue, and Angular. These submodules provide a component written natively for each framework. You’ll have a native React or an Angular component that you use just like any other third-party component you might add as a dependency through an npm install.

Since it’s just a typical front-end component, JSON Forms can take props just like any standard component you might write. Now, get the forms rendered by providing a validator class. INT currently uses AJV, which is the most popular open-source validator written in JavaScript. Then comes the actual schemas that we provide. 

There are three pieces of JSON data we can pass into JSON Forms, two that are required and one that is optional: 

  • First is the schema that describes our data model — see it referred to as just “schema.” This should look similar to the example schema previously explained with the name and age of a user.
  • Second is the UI schema — this is where you describe certain properties of the UI for the form.
  • Third, which is optional, you can pass a pre-existing data model as well and this will pre-populate the forms.  

You now have your data schema that also describes your data model. And you have the UI schema — where, like the name suggests, we can define various aspects of the UI of the form. This is where we essentially map the properties in our data schema to the actual form control that’s going to be rendered to the screen. Something that’s not shown here but that you can also define here is some extra options for grouping and layouts. So you can group certain controls together and also define if you want certain controls laid out horizontally or vertically. The data model is just the concrete data that follows the schema laid out by the data schema. With all of that together, JSON Forms can render a form with two controls, already pre-filled with those values.

Another helpful way to conceptualize what JSON Forms does is that it’s basically a way to turn a data model into a set of forms so you can edit or update that data model.  

How Does INT Use JSON Forms?

The last feature that brings it all together and allows you to cleanly integrate JSON Forms into IVAAP is the ability to create your own custom controls on top of JSON Forms.

JSON Forms comes packaged out of the box with its own library of controls. The UI utilizes Google’s Material UI designs. If you are familiar with Material UI, you’ve seen controls like this before. It’s very minimal, but still a very well-functional UI design. This is helpful because if a user just needs a form in their app, all that’s required of them to do is to write up data schemas for their particular model. Then the UI schema lays out the controls based on how the user wants them and they’re good to go.

If you’re developing a large web application that already has a well-defined UI and theming that also deals with a very particular domain that may require more specialized controls, JSON Forms gives you the ability to develop your own custom controls, giving you total control over how you want the look and feel of the controls to be. 

The great thing here is that you can develop these custom controls as components, for any of the three frameworks mentioned. So you can still use the tools and API of what you’re already familiar with to more efficiently add more controls as you need them. You just need to make sure to set it up to accept a few props that JSON Forms will pass down from its root-level component. 

Final Thoughts

While there are other ways to implement forms and dialogs, we found that using JSON Forms – with its ability to dynamically generate dialogs and to create custom controls to meet our feature requirements – to be an impressive tool that improves our developer and user experience.

GeoToolkit.JS: New Features and Improvements in Carnac 3D

The Carnac3D module is part of the GeoToolkit.JS library and utilizes WebGL 2.0 technology for 3D rendering on web browsers. For those who are not familiar with it, WebGL doesn’t require an extra plug-in and is built-in for almost all browsers, such as Chrome, Firefox, Edge, and Safari. 

GeoToolkit’s 3D library is extremely useful for developing professional applications. It provides rich oil & gas-oriented components, including seismic, grid surface, well trajectory, well log, heightmap, reservoir, schematics, volume rendering, and a bunch of visualization tools like picking, highlighting, and cursor visuals.

From the relational graph, you can see Carnac3D is integrated with the GeoToolkit base module, and also uses Three.js. Three.js is an open-source library that provides many low-level functionalities like renderer and scene graph. It provides many useful data types that can be used directly in our users’ applications. In GeoToolkit 3.3, we upgraded our internal Three.js to version 126, to maintain compatibility. For users that care about the core implementation of 3D, and maybe want to extend GeoToolkit on their own, WebGL 2.0 is a subset of OpenGL ES 3.0, and the shader version is GLSL 3.0.

carnac

 

In 3.3, we added seismic intersection with other 3D objects and a new grid surface loader called EarthVision loader. We also added a new widget called the projection widget. For minor features, an option was added to display flat shading surfaces, and also an auto-rotating mode to the 3D plot.

For rendering, we improved the accuracy of transparency rendering, reservoir grid performance, gridline, and isoline rendering, and anti-aliasing.

New Feature: Seismic Intersection with Grid Surface

We know that we can add multiple objects to a seismic volume such as grid surface, horizon, and reservoir. They are all intersected with each other. This causes an issue that makes it hard to observe the details with so many objects on the screen. Users must adjust the camera constantly to find the best angle. To resolve this issue, we introduced IntersectionHelper.

 

horizon1

horizon2

The intersection helps us preserve the detail of seismic and objects. We can clearly see its position in inline, xline, and time slice.

IntersectionHelper also supports reservoir intersection. When we visualize a reservoir with seismic, some cells are hidden inside the reservoir. With intersection, we can better observe the properties at intersected planes.

horizon3D

horizon4

horizongif


New Feature: EarthVision Surface Loader and Un-Triangulated Data

EarthVision is a commonly used format in oil and gas applications for loading grid surfaces. It converts the field data to a format that can be recognized by a 3D renderer.  

The format specification is, for each line, it has the following: 

Column index and row index are optional. If they are not provided, the data loader will mark it as un-triangulated data. So in the next step, the SurfaceData class will triangulate vertices using a gridding algorithm. If indices are provided, the SurfaceData has more information, so it can perform a much faster triangulation.

EarthVision loader can load data from either cloud server or local files, as long as it has a proper URL. Then it sends the result to SurfaceData, which is a predefined data class for preparing the data that the GPU can read. We include that and send it to the GridSurface instance. By adding surface instances to the 3D plot, we can finally visualize it.

The specification is: 

xPos, yPos, zPos [, colIndex],[rowIndex]

 

The data loader automatically identifies the type of data and triangulates for users if necessary.

 

carnac2

 

New Feature: Projection Widget

GeoToolkit’s new projection widget can project objects onto its grid planes so that users can better visualize an object’s position in the x/y/z axis. And most importantly, it supports almost all objects in 3D. This widget includes a built-in grid, and it can add projections using only one line of code.

projection2

projection1

 

New Feature: Flat Shading Surface

A new minor feature that has been added to 3.3 is the flat shading surface. 

The first example below shows a surface with smooth shading. While most of the area is super smooth, when it comes to faults or cliffs, it isn’t that good. This is because two adjacent polygons have a huge gap in terms of the surface normal, meaning the transition is not smooth anymore. 

unnamed (1)

 

To solve this uncomfortable artifact, we use flat shading. For flat shading, the lighting is evaluated only once for each polygon, and each polygon only has one color. You can see this in the second screenshot, where faults and cliffs are preserved very well. Here, the lighting is evaluated only once for each triangle so each triangle only has one light level.

 

unnamed (7)

 

Improvement: Accurate Order-Independent Transparency Rendering

One of the most distinct improvements is the switch to accurate transparency rendering. To produce accurate transparency in rasterization is a challenging task, so the usual “go-to “ solution is to render transparent objects from back to front.

But this solution is not perfect, and in many cases, it can result in artifacts. The most obvious case is when we have two or more intersecting objects — none of them is really in front or behind the other, much like in the first picture.

To solve this, GeoToolkit.js now uses Depth-peeling rendering. It provides what is called “Exact Order Independent Transparency,” meaning transparent objects are rendered correctly, without even needing to sort them. The result can be seen in the second picture.

The principle is quite simple to understand: To render each transparent object layer on top of each other, we have to render them one by one, separately and combine the results in the right order.

unnamed (3)

unnamed (4)

 

The final result of a single depth-peeled frame requires multiple rendering passes.

Each pass carefully keeps its color result, but also its depth buffer result, which is a map of the depth of each pixel of the 3D scene.

Now, with that in mind, here’s the process:

First, we render all the opaque objects, since they are opaque, everything behind them is not visible, so we only need to draw them once. This is a big time saver. Then, we render every transparent object, but only keep one fragment per pixel, which is a single object color per pixel. This means if in the same pixel, two or more transparent objects overlap, we only keep the front-most one. Once done, we keep the image result for later, and also update the depth buffer based on the fragment we kept. Then we repeat the last step, we draw every transparent object again, but this time we keep the pixel color only if they are deeper than the previous pass pixel. This depth test ensures that we render a different layer each time. We repeat this step as much as we need since each pass provides one additional transparency layer. And finally, all passes are composed, from back to front, in a final image.

In our case, depth peeling uses five transparent passes by default. It is of course customizable, but this configuration provides a good compromise between transparency depth and performance.

 

unnamed

 

Improvement: Improved Reservoir Grid Performance

In 3.3 we improved the performance so that reservoirs now render between 2x times and 4x times faster than before. Reservoirs are infamous for being slow to render, mainly because of their potentially large number of cells.

Starting with a couple of thousands of cells, most current GPUs hit what we call a vertex bottleneck, meaning there is simply way too much geometry to render, even when using advanced techniques like instanced rendering.

So the only solution to improve performance here is to draw fewer cells. In 3.3, we improve the performance by optimizing the reservoir geometry and the updates to the GPU memory.

First, we do not draw filtered cells that have been hidden by the user through value or position filters. Then, we do not draw cells that are hidden by their surrounding cells; we call them occluded cells.

So on an average reservoir, only 30 to 40% of the cells are actually visible. This means we have to render 60 to 70% less geometry, thus removing a huge load on the main bottleneck in the rendering pipeline.

 Of course, this improvement sounds simple enough, but in practice, there are important points that need to be carefully addressed. First, we must accurately identify the occluded cells, making sure they are hidden by all six of their neighbors. And we need to be able to update the geometry whenever a filter is updated, fast enough so that the user does not notice it.

Improvement: Lines Rendering

Before, lines were showing a lot of step artifacts, and it was difficult to have consistent and good looking lines. The issue is related to aliasing, but the principle is that it is hard to properly draw a line, which is a vector, onto a screen, which is a raster device. It is hard to draw oblique lines with square pixels, as you can see in the picture on the left. To solve this, we have implemented a powerful anti-aliasing technique into the line rendering.

As you can see in the pictures below, the step-looking artifacts are entirely gone in the new version on the right, and the linewidth is much more consistent across all view angles. So this improvement will be mainly visible on the plot grids.

unnamed (5)

unnamed (6)

Improvement: Anti-Aliasing Optimizations

Finally, we have the optimizations to the full scene Anti-Aliasing. GeoToolkit 3.3 introduces new anti-aliasing techniques and strategies, to improve image quality with a limited impact on performance.

Aliasing artifacts are those steps you can see in the left-most picture — instead of a straight line, we see jagged edges. This is because we are trying to represent a line with pixels, which are effectively squares, thus an oblique line will produce steps if nothing is done.

A more mathematical approach to the issue is that we are sampling and displaying a line, which is a function, with a raster monitor, which has pixels. A function can produce an infinite number of positions, but a pixel based monitor can only display a set number of pixels.

And the step issue gets worse as you reduce this number of samples and pixels. Different anti-aliasing techniques provide different results at different costs. 

The simplest ones try to detect and blur the edges of polygons, in a post-processing treatment, while the more advanced ones generally increase the sampling rate, by taking multiple samples per screen pixel, resulting in increased quality, but lower performance.

In 3.3, we introduce two new anti-aliasing techniques that try to cover both ends of the spectrum. 

For low-pixel-density monitors, high-quality anti-aliasing is used, called Super-Resolution Anti-Aliasing. This AA increases the number of samples per pixel, so each pixel is the product of several samples, increasing the image quality while also greatly reducing aliasing. This compensates for low-resolution monitors that naturally produce more visible artifacts because of their reduced pixel density.

For high-pixel-density monitors, however, more performance-friendly anti-aliasing will be used, called FX Anti Aliasing (FXAA). This AA reduces artifacts by detecting edges in the image and thus is much more performance-friendly. This compensates for large resolution monitors, which already have good image quality, but are also much more demanding in terms of performance.

Overall, this approach allows reduced aliasing in the scene, while keeping consistent performance and quality across different monitors. Or to put it more simply, higher quality for small monitors, and better performance for large monitors.

To learn more about GeoToolkit.JS, please visit int.com/products/geotoolkit/or contact us at support@int.com.

How Apache SIS Simplifies the Hidden Complexity of Coordinate Systems in IVAAP

See how Apache SIS, with IVAAP, helps support our client’s coordinate systems by using less code.

With the recent release of IVAAP 2.9, now is a good time to reflect on the help we got along the way. One of the components that made IVAAP possible is the Apache SIS library. The goal of this blog article is to bring more visibility to this awesome Java library.

What Is the Apache SIS Library?

Apache SIS is an open-source library written in Java that makes it easier to develop geospatial applications. Many datasets have metadata designating their location on Earth, and these locations are relative to a datum and a map projection method. There are many datums and many map projection methods. Apache SIS facilitates their identification and the accurate conversion of coordinates between them.

What’s a Datum and What’s a Map Projection Method?

Most people are familiar with latitude and longitude coordinates. This geographic coordinate system has been used for maritime and land-based navigation for centuries. Since the late 1800s, the line defining 0º of longitude has been the Prime meridian, crossing the location of the Royal Observatory in Greenwich, England. This meridian defined one axis, from South to North. The equator defined the other axis, from West to East. The origin point of this system on the Earth’s surface is in the Gulf of Guinea, 600 km off the coast of West Africa.

The traditional geographic coordinate system
The traditional geographic coordinate system (Source)

 

Similarly, a datum defines the origin point of the coordinate axes on the Earth’s surface and defines the direction of the axes. To account for the fact that the Earth is not a perfect sphere, a datum also describes the generalized shape of the Earth. For example, WGS 84 (World Geodetic System 1984) is a widely-used global datum based on latitudes and longitudes where the Earth is modeled as an oblate spheroid, centered around its center of mass.

The WGS 84 reference frame. The oblateness of the ellipsoid is exaggerated in this image. (Source)

 

WGS 84 is used by GPS receivers and the longitude 0º of this datum is actually 335 ft east of the Greenwich meridian.

While universal latitude and longitude coordinates are convenient, they are not universally practical because of land masses drift. Satellite measurements show that the location of Houston relative to the WGS 84 datum changes by 1 inch each year. A local datum is a more pragmatic choice than a global datum because distances from a local point of reference are smaller and don’t change over the years when all locations are on the same tectonic plate. A local datum may also align its spheroid to closely fit the Earth’s surface in this particular area.

A map projection method indicates how the Earth’s surface is flattened into a plane in order to make a 2D map. The most widely known projection method was presented by Gerardus Mercator in 1569. This is a global cylindrical projection method. It preserves local directions and shapes but distorts sizes away from the equator.

Cylindrical Projection
An example of global cylindrical projection (Source)

 

In the US, the Lambert Conformal Conic projection has become a standard projection for mapping large areas. This is a projection that requires multiple parameters, defining the longitude and latitude of its center, a distance offset to this center, and the latitude of its southern and northern parallels.

Conical Projection
An example of local conical projection (Source)

 

When a datum and a projection are used together, they define a projected coordinate reference system. While local systems limit distortions, they are only valid in a small area, an area known as the ”area of use” where a minimum level of accuracy is guaranteed.

Select Coordinate System
A screenshot from INTViewer showing the area of use of NAD27 / Wyoming East Central, a derived projected coordinate reference system

 

How Does Apache SIS Help IVAAP?

To show geoscience datasets on one of IVAAP’s 2D map widgets, you need to use a common visualization coordinate reference system.

IVAAP Screenshot
An IVAAP screenshot showing the location of wells on a map

 

This is where Apache SIS helps us: It understands the properties of both the data and visualization systems and is able to convert coordinates between them.

The math to perform these conversions is complex, this is not something you want to implement on your own. It requires specialized skills, both as a programmer and a domain expert. And just beyond the math, the number of datums and projection methods is mind-boggling. Many historical surveys are still in use today. For example, there are two datums used for making horizontal measurements in North America: the North American Datum of 1927 (NAD 27) and the North American Datum of 1983 (NAD 83). The two datums are based on two different ellipsoid models. As a result, the two datums have grid shifts of up to 100 meters, depending on location. IVAAP is able to visualize datasets that used NAD 27 as a reference, and it is Apache SIS that makes it possible to accurately reproject these coordinates into modern coordinate systems, accounting for their respective datum shift.

The datum shift between NAD 27 and NAD 83 (Source)

 

The oil and gas industry is at the origin of some of these local coordinate systems. Many of today’s new oil fields are in remote areas, initially lacking a geographical survey. There is an organization called the “OGP Surveying and Positioning Committee” which keeps track of these coordinate systems. It is colloquially known as “EPSG” for historical reasons. It regularly provides a database of these coordinate systems to all its members. This database is used by IVAAP and Apache SIS provides a simple API to take advantage of it. Each record in this database has a numerical WKID (Well Known ID). To instantiate a projection method or a coordinate system defined in this database, you just need to prefix this id with the “EPSG:” string.

OperationMethod method = getCoordinateOperationFactory().getOperationMethod("EPSG:9807"); // Transverse Mercator method

CoordinateReferenceSystem crs = CRS.forCode("EPSG:32056”);

 

The EPSG database itself is extensive, but it is common for INT customers to use unlisted coordinate reference systems, created for brand new oil fields. In these cases, a WKT (Well Known Text) string can be used instead. This text is a human-readable description of a projection method or coordinate system. The Apache SIS provides a clean API to parse WKTs. It also provides an API for formula-based projection methods that can’t be described by a WKT.

PROJCS["NAD27 / Wyoming East Central",
    GEOGCS["NAD27",
        DATUM["North_American_Datum_1927",
            SPHEROID["Clarke 1866",6378206.4,294.9786982139006,
                AUTHORITY["EPSG","7008"]],
            AUTHORITY["EPSG","6267"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4267"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",40.66666666666666],
    PARAMETER["central_meridian",-107.3333333333333],
    PARAMETER["scale_factor",0.999941177],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["US survey foot",0.3048006096012192,
        AUTHORITY["EPSG","9003"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    AUTHORITY["EPSG","32056"]]

The WKT of NAD27 / Wyoming East Central, with the WKID 32056

Why Did INT Choose Apache SIS Over Other Options?

INT had previous experience using GeoTools. Similarly to Apache SIS, GeoTools is a Java library dedicated to facilitating the implementation of geographical information systems. Being an older library, it goes much further than Apache SIS. For example, one of its components allows the parsing of shape files, something currently outside of the scope of Apache’s library. As a matter of fact, the first versions of IVAAP were using GeoTools for coordinate conversions.

One of the issues we encountered with GeoTools is that it is a library that provides only fine-grained Java conversion APIs. There are several paths to convert coordinates between two systems, and GeoTools allows the developer to choose the best method. Choosing the “best” method without human interaction is complex; it depends on the extent of the data being manipulated and the “area of use” of each coordinate reference system involved. It also depends on the availability of well-known transformation algorithms between datums. In North America, the standard for transformations between datums was formerly known as NADCON. The rest of the world uses a standard known as NTV2. Apache SIS works with both datum shift standards. It may elect to use WGS 84 as a hub when no datum shift is applicable. An algorithm to pick the best method would require a significant amount of code for INT to write and maintain. While Apache SIS allows fine-grained control over the different transformations used when converting from one coordinate reference system into another, it also provides a high-level API to perform this conversion. The picking of the best algorithm is part of the Apache SIS’ implementation. Its high-level Java API that picks a conversion algorithm matches IVAAP’s general use microservice for the same function. To pick the right algorithm, it only takes 3 parameters:

  • A definition of the “from” coordinate system
  • A definition of the “to” coordinate system
  • A description of the “extent” of the coordinates to convert
double x = …
double y = …
GeographicBoundingBox extentInLongLat = …
DirectPosition position = new DirectPosition2D(x, y);
CoordinateReferenceSystem fromCrs = CRS.forCode("EPSG:32056");
CoordinateReferenceSystem toCrs = CRS.forCode("EPSG:3737");
CoordinateReferenceSystem displayOrientedFromCrs = AbstractCRS.castOrCopy(fromCrs).forConvention(AxesConvention.DISPLAY_ORIENTED);
CoordinateReferenceSystem displayOrientedToCrs = AbstractCRS.castOrCopy(toCrs).forConvention(AxesConvention.DISPLAY_ORIENTED);
CoordinateOperation operation = CRS.findOperation(displayOrientedFromCrs, displayOrientedToCrs, extentInLongLat);
MathTransform mathTransform = operation.getMathTransform();
double[] coordinate = mathTransform.transform(position, position).getCoordinate();

Sample code to convert a single x, y position from “NAD27 / Wyoming East Central” to “NAD83 / Wyoming East Central”

We still use GeoTools for other parts, but as a general rule, the Apache SIS Java API tends to be simpler, more modern than GeoTools when it comes to manipulating coordinates and coordinate systems.

After 3 years of use, we are happy with our decision to move to Apache SIS. This library allows us to support more of our customers’ coordinate systems, with less code. We are also planning to use it to interpret the metadata of GeoTIFF files. The support has been excellent. When we needed help, the members of the Apache SIS development team were really keen to help us. This is one of the reasons why INT felt we needed to give back to the open-source community. Being a long-time member of OSDU, INT contributed to OSDU a coordinate conversion library built on top of Apache SIS. This coordinate conversion library converts GeoJSON and trajectory stations between different coordinate reference systems. Users can specify the specific transformation steps that will be used in the conversion process, either through EPSGs or WKTs. Behind the scenes, it’s the Apache SIS’ fine-grained API that is being used.