Support a Large Amount of Frequently Changing Entities Simultaneously With ArcGIS API for JavaScript Using WebGL

One day my boss came to me and my colleague Ron Avraham and said “We need to step up our game”. We were working on this GIS project using Esri ArcGIS API 4.0 for JavaScript, and our client woke up in the morning with an experimental mood. They wanted a new feature that will visualize a large number of entities on the map, and update their location-property frequently. They were talking about 10,000 entities moving on the map with their location updated every 0.5 seconds.

I was wondering whether ArcGIS API can handle this situation using the good old FeatureLayer. I tested it with mock data and, sadly, it didn’t keep up with the load. It took more than 500 milliseconds to update 10,000 entities and I felt defeated. If it works so slow with mock data, how will it cope with a WebSocket stream that was planned to feed the entities’ location?

Why did the ArcGIS API fail to do this task? Well, It’s written using WebGL, and god knows what incredible and complicated graphics can be made with WebGL. So I was pretty sure the problem was not with WebGL. But what about the way ArcGIS API renders to WebGL? Maybe the bottleneck hides there?

How to Plug Your Own WebGL Rendering Algorithm

In this part, I will show you how I solved my problem using my own WebGL rendering algorithm and creating a custom layer in ArcGIS API for JavaScript. I based my solution on this sample code, which I modified for my purpose. I highly recommend understanding the process of creating a custom layer before you continue reading.

For simplicity reasons, I used mock data of 10,000 entities and planned to keep their representation simple and cohesive (small green boats ⛵). The mock data was changed constantly by updating the location of the entities, so our little boats would be able to move around. In the “real-world”, the entities can be updated using some feeding source (for example WebSocket as I suggested earlier) but let's not focus on that.

10,000 green boat symbols on a map of the world.
10,000 green boat symbols on a map of the world.
The mock data. It‘s a bit crowded over there.
Moving green boats on the map of the world, each boat moves in a different direction.
Moving green boats on the map of the world, each boat moves in a different direction.
The final result: this is what it looks like while their locations update. Pretty smooth.

There are several ways to draw those graphics for the entities. I’ll show you one bad practice and one good practice. In both solutions we use vertex data and shaders. In the first solution, we contain the shape of each entity in the vertex data, and in the second solution, we move the drawing logic to the shaders and see better performances.

For me, the intuitive way to draw a boat is with lines, so the first adjustment is to change the drawing mode of the WebGL from TRIANGLES to LINES.

change this
to this

When using LINES mode, the vertex data is our canvas. Before starting to draw boats with code, let's draw one with Paint:

Plan how you want your symbol to look on a coordinate system.

In the sample code we modify, there is a conversion algorithm from location (XY) to screen pixel that WebGL understands, so every geographic location of an entity is translated into a point on the screen where its symbol should be drawn. This code is flipping the symbol that the vertex data is providing. To neutralize it, you can flip your design and used its points on the coordinate system as you can see momentarily in the vertex data.

Flip the design so it will appear good eventually on the map.

Later in the article, I will refer to the symbol’s lines by their color in the image above.

So now we can convert this shape from the coordinate system to our vertex data:

Every vertex data chunk is two points that represent a line of the boat symbol.

Now the vertexData holds the shape of each graphic/entity, and we need to adjust the fragment shader so it will draw that:

And it should look like this:

This solution works pretty well, it took 60 milliseconds to update 10,000 entities, but I do not recommend this solution because of three main reasons:

  1. It’s hard to draw. Even for a simple symbol such as our little boat, it’s barely tolerable to create this giant vertexData array.
  2. Passing the vertex data is an expensive action performance-wise. In this scenario, vertexData is not a light array, which makes this solution less efficient than WebGL can be.
  3. It’s quite a bad practice to use vertex to hold the graphic. Vertex attributes can include position, color, vectors (to create shading in a 3D graphic for example), etc. Who needs to be in charge of drawing the graphic are the shaders. And this leads me to the next section.

Although LINES mode might be the intuitive way, in my opinion, I find it clumsy, cumbersome and less efficient than WebGL can be. Now let's see how we can modify the code again and draw the symbols using fragment and vertex shaders only. Because the shaders will be responsible for the shape of the boat symbol, we can remove this logic from the verdexData array and keep it clean with the locations of our entities only:

In this solution, we are no longer using LINES mode and change to POINTS. This drawing mode allows the fragment shader to go through all the pixels and decide which ones to fill with color. We also can set width and height so it will paint a square of pixels every time:

The fragment shader goes through every square and decides which ones to fill with color.

Let's add the drawing logic to the shaders. In the fragment shader, we have 7 conditions as the number of the lines assemble the boat symbol (each call to step function is a condition). We check every pixel/point whether it’s part of a line or not and set its intensity to 1 or 0 respectively. In addition, the abs functions give the thickness of the lines.

Just as we flipped the symbol in the last solution, we must do it here as well. The first line of the fragment shader is doing just that. It flips the x dimension and y dimension to make sure our symbol is in the right direction:vec2 coord = vec2(gl_PointCoord.x — 0.5, gl_PointCoord.y * (-1.0) + 0.5);

And this is it:

Winning the Game — Performance Gains

With this solution, every update of the 10,000 entities took only 47 milliseconds. More efficient than the solution using vertex data, that came to the performance of 60 milliseconds and ten times more efficient than the regular FeatureLayer with more than 500 milliseconds for the same amount of entities.

Software developer Tel-Aviv, Israel

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store