Gary has an interesting post about conference feedback, in particular around the usefulness of a session. As both a speaker and organiser, I was going to reply in the responses, but maybe it’s worth a broader audience. Is the usefulness of a session actually relevant to anyone at all? I go to plenty of sessions that aren’t useful in my day to day job, but that doesn’t mean it wasn’t a good session or I didn’t enjoy it. Here’s what I make the choice based on:

• Do I have a pressing need to learn the topic and will the session give me the knowledge I need? It’s mostly about pointers to the technology, the speaker’s insight, that one nugget of information that makes it worthwhile.
• Do I enjoy the speaker? Sometimes a session is worthwhile just because the speaker is entertaining, even if I know I’ll get nothing out of it.
• Is it something I know nothing about? I sometimes attend sessions I know will have little or no impact on my daily work, just because I feel I ought to have a passing knowledge of the topic; sometimes that’s even as slight as what the heck it is.
• Is it something I know I’ll never master? I’ll attend lots of design and user experience talks, because those are particular interests of mine. I know I have limited (none, some would say) design skills and am intensely jealous of people who do. But I go in the faint hope that some of the sparkle will rub off.

Assuming I stay through the session, or the majority of it at least, I’ll always leave feedback, because as a speaker I know I value that feedback. If I did badly I want to know why, which is why it’s so frustrating to receive a low score with no indication of why. I once had “terrible session” written alongside a low score; if I’m so bad, how am I going to know why you thought it was terrible? My response is “terrible feedback”. Still, it beats “didn’t like his accent”; sorry, but it’s the only one I’ve got.

Now technical knowledge, speaker ability, etc, are easier to score numerically, and these are valuable to the organiser. But usefulness? Maybe for a big conference, trying to gauge the sessions to include, but for smaller events such as DDD, where the topics are picked by the public? Don’t blame the conference or the speaker if it wasn’t useful, you picked it. Now it’s a different matter it the session didn’t match the abstract, as that’s what you made your decision on, and that is useful information; maybe the speaker didn’t write a concise enough abstract, or got off topic, or any number of other reasons, but don’t just say it wasn’t useful, say it didn’t match the abstract.

I’m not suggesting we get rid of usefulness altogether as a question, it can help balance the other scores (eg a new speaker, with nerves, who doesn’t perform well, but still provides useful information). Maybe we just need to force a textual response if the usefulness is scored low. Or perhaps if any session is scored low.

Once of the biggest problems conference organisers have is getting feedback, and therein lies an immediate dilemma. The paper feedback form, handed in at the end of a session (or the day) gives you the immediacy; you’re less likely to forget what you thought of the session, but sometimes you need time to think about it. Plus, the organisers need to type all this information in if they want any form of analysis. On the other hand, the online form is convenient for later analysis and probably easier for attendees, but lacks the the ability for immediate response, unless it’s available during the session and there’s a network connection available.

So ultimately this comes down to a plea. If you attend an event, no matter what size, and are asked for feedback, please give it; make it useful feedback though, tell us what did and didn’t work, what you liked and what you didn’t. Certainly for UK events, where we have an active community, all of that information gets shared, so all events benefit.

I’m not really sure I need to blog this, because if you read this blog and not Scott Guthrie’s I’d be very surprised, but Scott just posted about technical debates, including WebForms versus MVC. Worth a read, including the comments.

I’m a big fan of RedGate’s tools, and their next little one has hit beta. SQL Search provides a quick and simple way to search for, and through, objects on SQL Server and is invaluable if you have a large number of objects. It’s fast, unobtrusive and works really well. Go give it a try.

If, like me, you’ve just upgraded to the V1 release of the Silverlight Bing Maps Control, and are wondering why the mouse wheel no longer zooms the map on browsers other than IE, it’s because you’ve got Windowless set to true. Eg:

<asp:Silverlight ID="Xaml1" runat="server"
    Source="~/ClientBin/SilverlightApplication1.xap" 
    MinimumVersion="2.0.31005.0" Width="100%" Height="100%" 
    Windowless="true" />

The solution is to set Windowless to false, or remove the attribute since false is the default. Of course, if you need windowless, then you might have to wait for a fix, or use IE.

There’s a post on the forums where you can document and discuss bugs.

I’ve been investigating the issues with dealing with invalid geography data, from live GPS points, and how that can be stored in SQL Server spatial types. Firstly I must retract the statement about using TRY/CATCH, since although you can use it, the resultant type won’t contain invalid data, it will be null. I have so much data I screwed the tests. So now my options are to create MULTILINESTRINGs or use the Geometry type, both of which I’ve considered and are both solutions that were suggested as a comment on the previous entry. I also spent some time yesterday discussing this with Johannes Kebeck (who incidentally is a great speaker), and he pretty much came to the same conclusions.

Using MULTILINESTRINGs

Let’s look at the MULTILINESTRING option in more detail, using some realistic data. Consider the following set of points:

This represents the following line:

It’s difficult to see the flow properly here, but it’s along, down, back, up, along again. If I shift a couple of points down a bit, you can see the flow a little better, starting from the left, doing a loop down before heading back to the right.

In real life we could end up with either up these situations; the difference between points 2 and 6 (the top left of the box area) in the second picture is only 4 metres, so well within the bounds of GPS drift. But, the above diagram is just to illustrate the flow of points better, so is really an aside. Let’s get back to the original point data, which we’d normally load into a spatial type using the following:

DECLARE @coords nvarchar(max) = 'LINESTRING(-0.17372131347655934 51.50372732305939, -0.17196178436280585 51.50386088908848, -0.168142318725583 51.50382081932085, -0.16829252243042417 51.50233821314898, -0.171983242034926 51.5023248561174, -0.17196178436280585 51.50386088908848, -0.168142318725583 51.50382081932085, -0.1652884483337343 51.50372732305939)'
DECLARE @line Geography = Geography::STGeomFromText(@coords, 4326)

But of course we can’t use this code, because the data creates an invalid geography type; the lines between points 2 – 3 and 6 – 7 are the same.With the MULTILINESTRING approach, we’d use the following:

DECLARE @coords nvarchar(max) = 'MULTILINESTRING((-0.17372131347655934 51.50372732305939, -0.17196178436280585 51.50386088908848),(-0.17196178436280585 51.50386088908848, -0.168142318725583 51.50382081932085),(-0.168142318725583 51.50382081932085, -0.16829252243042417 51.50233821314898),(-0.16829252243042417 51.50233821314898, -0.171983242034926 51.5023248561174),(-0.171983242034926 51.5023248561174, -0.17196178436280585 51.50386088908848),(-0.17196178436280585 51.50386088908848, -0.168142318725583 51.50382081932085),(-0.168142318725583 51.50382081932085, -0.1652884483337343 51.50372732305939))'
SET @multiline = Geometry::STMLineFromText(@coords, 4326)

This just repeats each pair of coordinates, apart from the first and last. It’s still an invalid geography type though, so I’m not sure of the value of this. Everything I’ve read, and the blog comment plus Johannes’s comments led me to the view that this would be valid, but I still get an exception when trying to create this.

Making valid data with Geometry types

The second option is to load the original data into a geometry type instead of a geography type:

This loads without an exception because the geometry type can contain invalid data. You can use IsValid() to check, and MakeValid() to make it into a valid type:

DECLARE @fixed Geography = Geography::Parse(@line2.MakeValid().ToString())

We now have a geography type, with corrected data. What does it look like? Well, on an image, pretty much the same:

The points however, show that the data has changed:

It’s not so easy to see, but points 6 and 7 have changed. What’s the difference? Well, we can use the spatial features to find out:

DECLARE @p1 geography = Geography::Point(51.503834175913900, -0.172004699707046, 4326)
DECLARE @p2 geography = Geography::Point(51.503860889088400, -0.171961784362805, 4326)

DECLARE @p3 geography = Geography::Point(51.503794106122800, -0.168099403381342, 4326)
DECLARE @p4 geography = Geography::Point(51.503820819320800, -0.168142318725583, 4326)

PRINT @p1.STDistance(@p2)
PRINT @p3.STDistance(@p4)

The answer for both is 4.20861 metres apart. Is this accurate enough? Possibly, depending upon your application, but I’m not sure it is for me, especially if you consider the data could be used for legal reasons. We’d still have the original point data, but if the corrected line string data is used to make decisions, there could be an issue. It would be good to work out what the formula used is when SQL Server makes this line string valid (one day I’ll delve into the assembly and look); the real question is, could making the line valid radically change the ordering of points, as it does for some geometry types. Consider the following:

Made up from the following points:

MULTILINESTRING ((0 10, 5 10), (5 10, 10 10), (10 10, 10 5), (10 5, 5 5), (5 5, 5 10), (5 10, 10 10), (10 10, 15 10))

Making this valid, results in the same view, the same set of lines, but the points are completely different.

MULTILINESTRING ((15 10, 10 10), (0 10, 5 10, 5 5, 10 5, 10 10, 5 10))

If all you’re describing is a shape, or a boundary, or a set of lines, then this is perfectly acceptable, but when the ordering of the points within your line is critical, then MakeValid won’t work for you. Will this affect my real life data? Who knows, I’ve yet to perform an analysis; if MakeValid only moves points (and even then that 4m move may be too much), then it might be OK, but if it strips points and re-orders, then I’ll have to abandon all plans to use line strings.

In defence of SQL Server, I understand why the data is deemed invalid, because that’s what the standard says, and Microsoft have followed the standard. But this is one area where the standard really sucks for real life tracking data; maybe these scenarios weren’t considered, or they were considered and the view was that line strings wouldn’t be suitable.

In my previous post I was talking about the new SQL Server 2008 spatial features and how they help with most types of location data, but I mentioned the big issue is the trace back one, where a line isn’t valid if it goes back on itself. I thought I’d explain this more clearly, so consider the following points that define a line:

-0.17187595367432515, 51.50221132119076
-0.17191886901856548, 51.503840854209066
-0.1680564880371023, 51.50378742782091
-0.17191886909856548, 51.503840854209066

This defines 3 segments: vertically up, across to the right, and then back to the left, back to the second point. It’s not immediately obvious that there are two horizontal lines, because they are so close to each other, but they are there.

Points 2 and 4 are almost the same, only 1 digit difference and that at the 11th decimal place, and that’s enough to make this a valid line string. Change that single digit so points 2 and 4 are the same and it’s invalid; attempt to create a LINESTRING based on the invalid points gives:

A .NET Framework error occurred during execution of user-defined routine or aggregate "geography":
System.ArgumentException: 24200: The specified input does not represent a valid geography instance.

The figures above are to a very high resolution, but consider a general purpose GPS unit, with perhaps only 5 significant places for it’s resolution, the chances of points overlapping increases. You can easily take two GPS readings without moving and get two different points, or three readings and three points, but giving only two locations (I have data where this has happened). The accuracy of modern GPS receivers is around 5-10m, sometimes better, sometimes worse; more expensive differential or gyro-based GPS units give more accuracy, down to centimetres, but these are more expensive and generally not available in consumer units.

Most applications that track objects, whether they be people or vehicles, could easily come up against this sort of data. It’s not hard to think of scenarios; consider tracking a car, which is stopped in traffic; a car coming the opposite way wishes to turn in front of it, so the first car reverses back a little, gives the second car room, then edges forward again as traffic moves. As we start to rely upon location based services more, our reliance upon accurate data increases. My view is that the rules on validity of line strings needs to be relaxed, although some browsing indicates that the rules are a well-established standard, so I’m on a knife edge: extend the standard or break it to make my life easier? I’m a huge fan of standards, but this one doesn’t seem to be that practical in real life. Many of the examples you see show a few individual points, or county and country boundaries, but there’s very little real life tracking data.

Multiple Line Strings

Even if Microsoft do relax the limitations on what constitutes a valid line string, it’s not going to appear until the next version, so workarounds need to be used. First is the support for multiple geometries, which is easier to explain if we go back to simple points:

0, 0
0, 10
0, 5

This is an invalid LINESTRING, so you couldn’t do this:

DECLARE @line Geography = 
        Geography::STGeomFromText(‘LINESTRING(0 0, 0 10, 0 5)’, 4326)

But you could represented this as a MULTILINESTRING:

This consists of two line strings, so it perfectly valid. The problem now becomes, how do you know where to split the list of points? You could, perhaps, build up a collection; if a point already exists in the collection, start a new set of points. It would work, but isn’t very efficient (although if you don’t create these very often that might be irrelevant), plus it’s harder to do in SQL.

Exception Handling in SQL

Another option, which I’m currently investigating, could be to just wrap the creation of the line string in a try catch. Yes, for those that didn’t know, SQL has exception handling. For example, I could do:

DECLARE @line Geography
BEGIN TRY
  SET @line = Geography::STGeomFromText(‘LINESTRING(0 0, 0 10, 0 5)’, 4326)
END TRY
BEGIN CATCH
END CATCH

This solves the problem of the invalid geometry by simply ignoring the exception. The geography object exists and contains the correct data, although there’s no guarantee that any of the spatial methods will produce the correct results when used on this invalid type.

Investigations continue …

One of the applications I work on deals with a lot of geographical data; 32Gb database, one table having 22 million rows. Certainly not the biggest I’ve heard of, but the biggest I’ve worked on.

Most of this data is point data, which is currently stored as separate Longitude and Latitude fields, which are floats, but we’re now on SQL 2008 and using the new spatial features has some advantages, noticeably speed, and to take advantage of that we need to add a column of type Geography. Next, this column needs to be updated with the existing data:

UPDATE Waypoint
SET Location = GEOGRAPHY::Point(Latitude, Longitude, 4326)
FROM Waypoint

On it’s own this doesn’t bring us much; sure we’re now using the new type and can use the built-in functions, such as working out if one point is within 100 metres of another:

DECLARE @MyLocation Geography = GEOGRAPHY::Point(StartLat, StartLong, 4326)
DECLARE @MyBufffer Geography = MyLocation.Buffer(100)

SELECT Location
FROM Waypoint
WHERE Location.Filter(@MyBuffer) = 1

On it’s own this is an improvement from our existing code, which has to calculate distances manually, but any calculations we now use would be hampered without a spatial index:

CREATE SPATIAL INDEX Location_Idx ON Waypoint(Location) USING GEOGRAPHY_GRID

Now the filtering can use the geospatial index, which brings massive performance increases. One procedure in our old code is a complex cursor based one, matching waypoints to agreed waypoints (think mountain biking competitions, where you have to follow a set route; matching points along the way allows us to ensure the route was followed, give or take those 100 metres). Given that this table has 22 million rows, the old cursor approach barely worked at all; very high CPU usage and an eternity to run meant we didn’t really use it. The new procedure, using the spatial functions and appropriate indexes, finishes in 9 seconds – an improvement of over 30,000% – and I’m pretty sure we can improve it even further; the first attempt was simply to use the spatial features. This is a monumental improvement and will revolutionise what we can do with our application, doubled by the fact that we’ll also be using the Silverlight Bing Map control.

And now to a downside of the spatial features, which is that if your define a route as a set of waypoints and use the LINESTRING feature to define the polyline of the route, the points cannot trace back over themselves. Cross yes, but not trace back. Consider the following simplistic set of points:

0, 0
0, 10
0, 5

This describes a trace back situation, which may be perfectly valid; cycle to a marker and cycle back. Depending upon how close the forward and back routes are, the GPS points might give the above results. In fact, we have the above data issue for three points of a stationary cycle, where the GPS drift has caused the data, albeit at the 6th decimal point, to give two different readings, but ones that case a trace back. For our application, and possibly any application using GPS units, we cannot therefore use a LINESTRING; you can’t event create the Geography type with that data, because the data is “an invalid geography type”.  Why would we need this? Creating polylines allows the entire set of points to be treated as a single type for calculations, such as seeing if a point is within X metres of the route, or if one route crosses another. Without the polyline we have to iterate the points individually.

The Geometry type allows the type to be created, and I’ve yet to work out whether whether we could use this type as a fake Geography type. The Geography type has built in knowledge of the Earth and takes into account the spherical (oblate spheriod in fact) nature of the Earth in these calculations. The question I need to answer is whether the difference between using the two types would be large enough to render our calculations invalid. It may be a route (pun intended) I’m forced to use. In the meantime, I’ve logged a suggestion on Microsoft Connect asking for the rules on LINESTRING usage to be relaxed, but even if they are relaxed, it won’t show up until the next version of the product. In the meantime, I can continue to improve our product by replacing the manual calculations with the spatial ones.

By and large, server products don't excite me; they are utility things, there because they have to be. I may be changing my mind though.

Over the last few weeks I've reorganised my servers, virtualising many of them and migrating my data to a new server; it was meant to be an old server, re-housed in a case big enough to take the 2 4-disk hot-swap drive bays, but the old motherboard didn’t fit so a new one was slotted in place. An afternoon of plumbing and an installation of Widows Home Server, and the new storage server is ready, and boy, is it a good product. Since the great disc disaster of ‘05, I’ve been paranoid about my data, keeping it in multiple places, and always hated the fact that you can have oodles of discs, but each is standalone. Unless you have a RAID controller, which I did, but failures were multiple. I was away. It was traumatic.

The great thing about WHS is that you don’t bother about RAID, it does it for you; it aggregates all of your discs and presents a single big volume with shares (a bit like drobo). You can specify whether files under those shares are duplicated, and WHS takes care of keeping multiple copies in case of a disc failure. Or if you want to increase your storage, you can replace a disc with a larger one. Access control is simple, nothing like the complexities of the domain model, but that’s a good thing for a home product, or even for a small business and I think Microsoft should push this to the small businesses as it’s ideal - easy to use, and fairly cheap; you can buy a ready installed server for a few hundred pounds and be up and running in an hour or so.

Easily expandable and safe storage isn’t the only benefit though, backups figure heavily too. On client machines you run the connector software and your client machine is added to the server; you then specify which drives to backup and that’s it; everything happens automatically. It even keeps backups back to 3 months (by default), and allows instant restore of entire client machines in case of disc failures. I still keep data locally (robocopying it to the WHS share nightly) and use Mozy for offsite backup, but I’m much happier knowing my machines are backed up and easily restorable.

Client integration is good too, with the server shares (Music, Photos, Videos) being automatically added to Win7’s libraries. Media Centre integrates too, so the same folders are made available

Anyone who’s got several Windows machines and wants centralised storage and backup should take a look into it.

Next step, installing DVDShrink and ripping my DVDs. There’s a MyMovies plugin for WHS, so I won’t even have to get up to play my favourite movies. Once the new PSU for my MCE box arrives of course. That was traumatic too.

I'm trying not to complain too much, but some times you have to. Today I decided to upgrade my main machine to Windows 7 Release Candidate, while working on the laptop. The first attempt failed due to lack of disc space - it's needs 10Gb free - but I soon solved that. Then I left the upgrade to trundle away until the final stage of transferring files, settings and programs, whien 43% of the way through (ooh, so close), I received the annoying statement that there wasn't enough disc space free and it would revert to the previous version. So, you checked how much space I needed, installed, and then changed your mind. Way to go to give me a good impression of the product.

Now I know I need to rejig my partition sizes, but it strikes me as odd that the installer reports it has enough space at the start, then doesn't at the end. It did say it would prefer to have 16Gb free, but didn't say it would fail the install. Although whether it's failed is another matter; apparantly it's "sending diagnostic information to Microsoft", but there's no progress and no indication that anything is actually happening. ooh, ooh, now wait, it's rebooting....

Nope, it's rolling back the install. So, do I now try and free up more space or just bite the bullet and install from scratch, which means installing everything.