SharpGIS

#GIS from a .NET developer's perspective

Profanity usage

Inspired by xkcd, here's my take on it as a javascript developer:

My Profanity Usage by Browser

To be fair to the numbers above, I first of all base my JavaScript on the Microsoft AJAX client library, so usually this works pretty well with IE7. I start out with making stuff work in IE7 and then adjust it to work in FireFox, IE6, Safari etc. This is probably why I spend more time fixing quirks in FireFox than in IE7. I'm currently doing some testing against Safari 3.0beta, and some of the issues I'm having were often resolved by making some subtle (but illogical) changes in the JavaScript, but these changes were often something I've stumbled upon by sheer luck (hence the profanities). It might just as well be because this is a beta browser. Surprisingly I do find myself spending more time working around issues and bugs in FireFox than in IE, but in the end I'm usually able to come up with a script that works in all major browsers and only have very few tweaks that tests against which browser I'm using (thank God for JS base API's that does most of that work).

IE6 wins as the browser that required the most extra code needed, just because of one thing: PNG Transparency. I need objects that need to know whether its using images that are PNG or not, so I can apply the transparency workaround (but at least this is a well-known workaround). It just complicates the JS API, requiring developers to deal with more properties to ensure it works in IE6. I can't wait till the day where we can sign of that browser version.

So why do I primarily develop in IE7, and not FireFox which seems to be the case for a lot of developers? Many reasons:

  • The debug integration with Visual Studio.NET is lightyears ahead of FireBug and all the other JS debug tools in FireFox (and is the JS console really all I get in Safari 3.0 on Windows?).
  • There are some great tools to browse and modify the DOM, see requests etc. My favorites are "Web Development Helper" which even have JSON and Microsoft AJAX Partial Postback deserializers to help you easier analyze the server responses, and IE Developer Toolbar for viewing the DOM and modifying it on the fly (priceless when you are tweaking your CSS styles).
  • It's the most used browser out there, so this is in my mind also the most important one. I'm not saying the others aren't important, but this should ensure that the most used browser also gets the most preliminary testing.
  • I know all the non-ECMA-Script properties and methods that IE adds to the DOM, so I also know which properties and methods to avoid. I'm not that familiar with Mozilla's extensions and don't want to risk using methods that doesn't exist in other browsers (that just adds to the number of profanities :-).
  • This one is probably going to give me some flaming, but I do think IE7 is the best overall browser.

 

.NET 2.0 Raytracer

I've had a lot of requests for the .NET 2.0 based raytracer that I used in my Master Thesis to generate true orthophotos based on complex 3D surface models (not necessarily 2½D). To save me from having to dig it up all the time, I've now posted it here on my blog so you can download it from here.

Basically it will read in a text file with triangles (each triangle defined per line as comma-separated double values X1,Y1,Z1,X2,Y2,Z2,X3,Y3,Z3), create a spatial index using axis-aligned boundingboxes, and perform ray/triangle intersections, returning any triangle intersecting the ray, including distance to triangle and angle of attack.

To save memory on larger DSM, all internal computation is done using floating-point insted of using double precision. This poses a problem for large values like UTM based coordinate systems, where rounding errors will occur when using floating point. The raytracer is internally applying an offset to the center of the data extents to accommodate for this, but it's worth remembering if your DSM covers a vast area. In most cases it's not a big issue though.

The raytracer scales very well (O(log n)), and can handle hundreds of thousands of faces and still raytrace in a few milliseconds (see chapter 7.3.3 of the thesis for details). Performing the required indexing first might be a little slow, so if you want to optimize this, be my guest ;-).

Some uses this engine has been involved in is generating rough worldfiles for aerial images based on absolute camera orientation and a surface model or Simulating Navstar and Galileo GPS availability in dense urban areas. The latter was actually some very cool stuff developed by some students at DTU. You could click anywhere on a map, and it would instantly simulate the visibility of Navstar and Galileo GPS satellites and give you the PDOP values throughout the day, based on a detailed 3D city model. This was used to determine where the zones of GPS based roadpricing should be (you didn't want the zone edges to be close to a place where you had poor accuracy causing people to be charged incorrectly). The raytracer was used for doing the line-of-sight analysis between a virtual car and all the satellites.

Download: Raytracer.zip (86.29 KB)

Silverlight managed code vs. JavaScript

I finally had a chance to play with Silverlight today. The managed code capabilities is very interesting, so I set out to create a simple performance tester.

I created two basic managed methods in Silverlight. One that multiplies a number and returns a result, and another that does the same thing a specified number of times:

[Scriptable]
public double Multiply(double a, double b)
{
   return a * b;
}
[Scriptable]
public double MultiplyLoop(double a, double b, int count)
{
   double result = 0;
   for (int i = 0; i < count; i++)
      result = a * b;
   return result;
}

This gives me 3 test-cases to compare:

  • Multiply two numbers n times in pure javascript.
  • Multiply two numbers n times, by calling Multiply(a,b) from JavaScript into managed code n times
  • Multiply two numbers n times, by calling MultiplyLoop(a,b,n) from Javascript into managed code one time.

What I found was:

  • JavaScript is fairly slow to do this (345 ms for 10 millions calculations).
  • Calling a managed method millions of times is VERY slow (IE asked me several times whether I wanted to abort it before it finished)
  • Calling a managed method once and do the looping in managed code is VERY fast (0-2 ms).

The conclusion here would be that you shouldn't have methods in managed code that you call a lot from javascript unless they are process-intensive. There seems to be a big overhead in calling into managed code, but as soon as you are in the managed process, its blazing fast.

You can download my test sample here: SilverlightProject1.zip (18.35 KB)  (requires Orcas beta 1 and Silverlight add-on)

Now on to move all my existing .NET libraries into a Silverlight app... :-)

Spatial references, coordinate systems, projections, datums, ellipsoids – confusing?

People are often mixing the above as if they were one and the same, so here’s a recap of them. One of the things you often find people saying is that “my data is in the WGS84 coordinate system”. This doesn’t really make sense, but I will get back to this later.

This is a very confusing subject, and I might have gotten a few things wrong myself, so please add a comment and I’ll update it ASAP.

Coordinate systems

A coordinate system is simply put a way of describing a spatial property relative to a center. There is more than one way of doing this:

  • The Geocentric coordinate system is based on a normal (X,Y,Z) coordinate system with the origin at the center of Earth. This is the system that GPS uses internally for doing it calculations, but since this is very unpractical to work with as a human being ( due to the lack of well-known concepts of east, north, up, down) it is rarely displayed to the user but converted to another coordinate system.
  • The Spherical or Geographic coordinate system is probably the most well-known. It is based on angles relative to a prime meridian and Equator usually as Longitude and Latitude. Heights are usually given relative to either the mean sea level or the datum (I’ll get back to the datum later).
  • The Cartesian coordinate system is defined as a “flat” coordinate system placed on the surface of Earth.  In some projections it’s not flat in the sense that it follows the earth’s curvature in one direction and has a known scale-error in the other direction relative to the distance of the origin. The most well-known coordinate system is the Universal Transverse Mercator (UTM), but surveyors define their own little local flat coordinate systems all the time. It is very easy to work with, fairly accurate over small distances making measurements such as length, angle and area very straightforward. Cartesian coordinate systems are strongly connected to projections that I will cover later.

Sidenote: The geocentric coordinate system is strictly speaking a cartesian coordinate system too, but this is the general terms I've seen used the most when talking about world coordinate systems.

Geocentric.png Spherical.png Cartesian.png

Datums and ellipsoids

Some of the common properties of the above coordinate systems are that they are all relative to the center of Earth and except the Geocentric coordinate system, uses a height system relative to the surface of the earth.

This poses two immediate problems:

  • Where is the center of the earth
  • What is the shape of the earth?

By now most people should know that that the earth isn’t flat (although there are still some who doubts it). If we define the surface of Earth as being at the mean sea level (often referred to as the Geoid), we don’t get a spheroid or even an ellipsoid. Because of gravitational changes often caused by large masses such as mountain ranges etc, Earth is actually very irregular with variations of +/- 100 meters. Since this is not very practical to work with as a model of earth, we usually use an ellipsoid for approximation. The ellipsoid is defined by its semi-major axis, and either the flattening of the ellipoid or the semi-minor axis.

The center and orientation of the ellipsoid is what we call the datum. So the datum defines an ellipsoid and through the use of a set of points on the ground that we relate to points on the ellipsoid, we define the center of the Earth.  This poses another problem, because continental drift moves the points used to define the points around all the time. This is why the name of a datum usually have a year in it, often referring to the position of those points January 1st of that year (although that may vary).

There are a vast amount of datums, some used for measurements all over the world, and other local datums defined so they fit very well with a local area. Some common ones are: World Geodetic Datum 1984 (WGS84), European Datum 1950 (ED50) and North American Datum 1983 (NAD83).

The most well-known is WGS84 used by the GPS systems today. It is a good approximation of the entire world and with fix-points defined almost all over the world. When it was defined they forgot to include points in Europe though, so the Europeans now have their own ETRS89, which is usually referred to as the “realization of WGS84 in Europe”. The problem here was solely because of continental drift, so they defined some points relative to WGS84 in 1989, and keeps track of the changes. In most use-cases it is of no real importance and you can use one or the other.

I mentioned earlier that people often refer to having their data in WGS84, and you see now why this doesn’t make sense. All you know from that is that the data is defined using the WGS84 datum, but you don’t know which coordinate system it uses.

Read more on Datums and Spheroids.

Projections

The earth isn’t flat, and there is no simple way of putting it down on a flat paper map (or these days onto a computer screen), so people have come up with all sorts of ingenious solutions each with their pros and cons. Some preserves area, so all objects have a relative size to each other, others preserve angles (conformal) like the Mercator projection, some try to find a good intermediate mix with only little distortion on several parameters etc. Common to them all is that they transform the world onto a flat Cartesian coordinate system, and which one to choose depends on what you are trying to show.

A common statement that I hear in GIS is the following “My map doesn’t have a projection”, but this is simply not possible (unless you have a good old rotating globe). Often people are referring to data that is in longitude/latitude and displayed on a map without having specified any projection. What happens is that the system applies the simplest projection it can: Mapping Longitude directly to X and Latitude to Y. This results in an equirectangular projection, also called the “Plate Carree” projection. It results in very heavy distortion making areas look squashed close to the poles. You can almost say that the “opposite” of the Plate Carree is the Mercator projection which stretches areas close to the poles in the opposite direction, making them look very big. Mercator is the type of projection you see used on Live maps and Google maps, but as many often mistakenly thinks, they do NOT use WGS84 for the projected map, although WGS84 is used when you directly input longitude/latitude values using their API (read more on this here).

More on projected coordinate systems

Spatial reference

The spatial reference is a combination of all the above. It defines an ellipsoid, a datum using that ellipsoid, and either a geocentric, geographic or projection coordinate system. The projection also always has a geographic coordinate system associated with it. The European Petroleum Survey Group (EPSG) has a huge set of predefined spatial references, each given a unique ID. These ID’s are used throughout the industry and you can download an Access database with all them from their website, as well as some very good documents on projection (or see the Spatial References website).

So when you hear someone saying they have their data in WGS84, you can often assume that they have longitude/latitude data in WGS84 projected using Plate Carree.  The spatial reference ID of this is EPSG:4326.

Spatial References are often defined in a Well-known format defining all these parameters. The Spatial Reference EPSG:4326 can therefore also be written as:

GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.0174532925199433]]

As mentioned Live/Google maps use a Mercator projection, but although their datum is based on WGS84, they use a sphere instead of an ellipsoid. This means that they use the same center and orientation as WGS84, but without applying any flattening. The spatial reference string for their projection therefore becomes:

PROJCS["Mercator Spheric", GEOGCS["WGS84based_GCS", DATUM["WGS84based_Datum", SPHEROID["WGS84based_Sphere", 6378137, 0], TOWGS84[0, 0, 0, 0, 0, 0, 0]], PRIMEM["Greenwich", 0, AUTHORITY["EPSG", "8901"]], UNIT["degree", 0.0174532925199433, AUTHORITY["EPSG", "9102"]], AXIS["E", EAST], AXIS["N", NORTH]], PROJECTION["Mercator"], PARAMETER["False_Easting", 0], PARAMETER["False_Northing", 0], PARAMETER["Central_Meridian", 0], PARAMETER["Latitude_of_origin", 0], UNIT["metre", 1, AUTHORITY["EPSG", "9001"]], AXIS["East", EAST], AXIS["North", NORTH]]

Notes on javascript performance

Earlier today (some would say way too early after a long night at Pure) I attended the session "How to make AJAX Applications scream on the client". This turned out to be more about general patterns in Javascript and not so much about AJAX.

Anyway here are my notes from that session:

Avoid Eval(). Instead use parameterized code.

'Switch' is costly for large sets and grows linearly. Instead use a hashtable and wrap in try/catch.

Getters/setters are inefficient. Use direct access to the variables in an instance.

DOM Instantiating and traversing is slow because it has to go from the JS layer to the DOM layer through COM. Use it as little as possible. Example:

   Bad: for(var i=0;i<100;i++) { document.getElementById('myDiv').innerHTML = i; }
   Better: var div = document.getElementById('myDiv'); for(var i=0;i<100;i++) { div.innerHTML = i; }

Use speculative download. Try and anticipate what the user needs to download next. For instance, on the login page start loading icons that the users will need after logging in. The browser might as well spend the time he is spending on entering his password on downloading stuff. When he then logs in the browser will retrieve the images from the cache instead of having to download them from the server, making the page load much quicker. Try going to the login page of an Outlook Webaccess website and see what happens behind the scenes through the Fiddler tool.

Enable GZIP encoding on the webserver. Most browsers supports it and will result in smaller downloads. Again Fiddler is great for experimenting with this.

You can see the full session online here.

Getting (re)started with Silverlight and some notes

On my way back from the Mix conference in Vegas I was trying to get one of my WPF/E applications to run on the new Silverlight 1.1alpha. It wasn't as easy as I thought it would be because a lot seems to have changed. I finally figured it out after reading Bryant Likes's blog. He already figured it out and shared the solution (Thanks Bryant!):

If you want to go the "real" route of working with Silverlight 1.1alpha, here's what you need:

I saw two really good sessions on Silverlight and .NET this morning. They are online now, and I really recommend that you watch them:

One of the really neat things are that Silverlight also supports LINQ in the client browser. Part 2 shows a cool example on that and how easy it makes it to work with data. Furthermore you can easily communicate between Browser DOM, Javascript and .NET assembly code which makes for some interesting scenarios. Since .NET excution is 400-1000 times faster than JavaScript, you could have your heavy algorithms inside silverlight, and call them from Javascript. If you don't need the presentation-layer, you could just hide it and have it as a library you utilize from JS.

So what is the main differences between v1.0 and v1.1? v1.1 contains:

  • Everything from v1.0
  • A subset of the .NET 3.5 framework and some extra browser and AJAX specific classes
  • XAML extensibility
  • Control class (for user controls)
  • Sample controls

Currently Silverlight works with Safari, Firefox and IE on both Mac and Windows and soon to be Opera as well. I asked them whether they would also support other platforms like Linux. Basically they didn't have any 'issues' with making that, but were currently not considering it until there is a larger demand for it.

Update: Mono has already announced that they will make sure we get Silverlight support on Linux. COOL!

Ajax View: Remotely Monitoring Web 2.0 Applications

Today Microsoft demoed their AjaxView application in one of the Mix07 sessions. It acts as a proxy and monitors requests, as well as execution time for each javascript function that gets executed. This is really neat for tracking what methods are putting the most load on the client browser. Now all we need is some built-in tracking of memory leaks ;-)

A tech preview will be available within the next 1-2 months.

Update: You can now see the full session where this was demoed online here.

Mix07 Keynote

The big topic of today’s keynote by Ray Ozzie & Scott Guthrie at the Mix07 was Silverlight. Today several new things were unveiled:

  • Silverlight 1.0 beta was released
  • Silverlight 1.1 alpha was released / unveiled
  • Silverlight Streaming was unveiled
  • Silverlights new community website was unveiled at www.silverlight.net
  • Blend Expression was released to day (we all got a free copy too)

And to me the biggest news of them all:

  • Silverlight 1.1 alpha has support for .NET code inside the Silverlight application!

Basically this means that you now can use C#, VB, IronPython, the all-new IronRuby (also unveiled today) and any other .NET language in an app that runs inside the browser crossbrowser AND crossplatform. Through .NET you can also work with and manipulate the HTML DOM, use REST services etc. No more JavaScripting! (and I was just about getting to a point where I really started to like JavaScript)

Scott Guthrie demoed a cool Silverlight-based chess application that used both JavaScript and C# for the AI, and had the two languages “compete” against each other. According to Scott, .NET is 400-1000 times faster than JavaScript, so it could calculate a lot more moves in the same time. Of course .NET won the simulated game :-) There were a lot of other cool Silverlight demos by non-Microsoft companies, including a kick-ass video-editing app running in your browser.

“Silverlight Streaming” (http://silverlight.live.com/) is Microsofts new website for sharing video (A Youtube killer perhaps?). You can freely sign up for 4gb space, and supports up to DVD-quality videos of up to 10mins. You can completely integrate videos streaming from Microsofts servers onto your webpage through Silverlight. So the next time you upload a really cool video to your website with the risk of overloading your server, Silverlight Streaming might be your answer.

So, Silverlight 1.0beta is out now, and will be finally released in the Summer 2007. 1.1alpha is out now too, but the final release is still to be determined. This includes a subset of the .NET framework, similar to what they did to the compact framework.

They also showed that through Visual Studio Orcas you can now remotely debug Silverlight applications that run on a Mac, set breakpoints, change properties on the fly etc. REALLY cool stuff.

Hi I’m a Mac. Hi I’m a PC and I can control you remotely! ;-)

You can download Silverlight 1.0b, 1.1a and SDKs at http://msdn2.microsoft.com/en-us/silverlight/default.aspx

So basically this means you now can use .NET for a larger array of purposes:

  • ASP.NET (Webserver)
  • XNA (XBOX)
  • CF.NET (Pocket PC/Phone)
  • Silverlight (Webclient – Crossplatform!)
  • WPF (Desktop)

I can’t wait to get started with this!

The other sessions I attended today was pretty good too (although missing most of the one I wanted the most because the hotel got my reservation screwed up), none of it was really new stuff we haven’t seen before. Hopefully there will be some more WOW tomorrow…

Going to Mix '07

The weather outside is awesome today. A shame I have to spend it getting ready for the Mix conference in Las Vegas. (but then again, I'm going to Vegas :-)

There's a lot of focus on Microsoft AJAX and SilverLight (formerly known as WPF/E) this year. Both of which I find really interesting technologies for use in Web GIS.

Some of the sessions that I plan to see are:

  • Building Rich Web Experiences using Silverlight and JavaScript for Developers
  • Using Visual Studio Codename "Orcas" to Design and Develop Rich AJAX Enabled Web Sites'
  • Using LINQ to Dramatically Improve Data Driven Development in Web Applications (By my Danish fellow Anders Hejlsberg)
  • Developing ASP.NET AJAX Controls with Silverlight
  • Go Deep with AJAX
  • How to Make AJAX Applications Scream on the Client
  • Navigating the Programmable Web
  • Accessing Data Services in the Cloud

Are you going too?