Thursday, October 30, 2008
#
Finally got to see Anders Hejlsberg reveal the new features in C# 4.0 today. As expected, nothing hugely radical. But the new ‘dynamic’ keyword did inspire one thought that I shared with a couple of guys in the ask-the-experts session and Dave Remy suggested I should blog it, so I shall…
Three years ago as part of the LINQ wave of announcements for C# 3.0, Anders introduced anonymous types and this necessitated the ‘var’ keyword. Now we all know this is just a static inferred type declaration, we’re all cool with it now, but at the time there was a lot of panicking and wailing and misunderstanding about what this new ‘var’ word really meant, much of it around the fear that C# had introduced a variant type. This turned out not to be true, and the panic subsided.
Now, with the new ‘dynamic’ keyword… guess what? C# has variants. And nobody’s noticed, and nobody’s panicking.
Yet…
Wednesday, October 29, 2008
#
With a combination of jetlag and a persistent cough keeping me from sleeping, it seems like a pretty good moment to jot down some thoughts on the first couple of days of the 2008 Microsoft Professional Developers Conference, “the first PDC in the history of PDCs”, as Don Box said, “where Microsoft has not launched a new data access stack”.
My overall feeling about this PDC is that MS has fluffed the marketing angle really badly. There doesn’t feel like there’s a coherent story behind this PDC launch wave. MS will tell you this is the “Software + Services” PDC, but they’ve failed to convince that that’s a central driving philosophy, rather than a generic term they’ve applied to try to categorise what they’re about.
The pitch of the keynotes has felt slightly wrong, and I think the main reason for that is that so much of what could have been the big splashy announcements to get the crowd on their feet had already been announced prior to PDC. The last month has seen a whole raft of things trickle out of microsoft not with a bang but with a whimper: the Windows 7 name; the fact that MS was launching what Ballmer called an ‘OS for the Cloud’; Silverlight 2.0. Even a couple of developer crowdpleasers, like ASP.NET MVC hitting beta and the deal where MS will start shipping JQuery sneaked out via the blogosphere rather than being saved a couple of weeks for the big event.
When the ‘Azure’ name leaked out (via a Microsoft RSS feed) before the Monday morning keynote, all the wind was taken out of the sails of that keynote. It wasn’t helped by the fact that Ray Ozzie feels too corporate as a presenter to engage a dev crowd. The crux of the Azure announcement really comes down to Microsoft going into web hosting, and while the cloud provisioning model is qualitatively different to your classic rackspace offering, it doesn’t demo any differently. The point of Azure is to make scaling transparent; transparency doesn’t demo well.
And my overall impression of Azure is… well, ‘meh’. Google and Amazon offer cloud hosting platforms; now so does MS. Each has proprietary management APIs, proprietary storage, proprietary hosting APIs. Developing a system for any one of these cloud services locks you in to that vendor’s offering. I’d expected Microsoft to, well, act like Microsoft, and make a platform play. Release Azure as a platform for cloud hosting services, let third party ‘hosting partners’ take up the hosting part of the problem, and change the cloud service hosting market. I could commit to building Windows-Azure-based apps if I knew that I had competition among hosting providers, just as I am happy now to build ASP.NET web apps knowing I can go to a range of hosts to get my app out on the net. With MS as the sole provider, I can’t commit to their platform, when it could turn out to just be another .NET MyServices and disappear quietly into the sunset in 12 months. No game-changer, so… ‘meh’.
Of course, MS wanted to use the day 1 keynote to set the message that will go out in the non-specialist press about their plans; the audience for the cloud announcement was the business world looking at Microsoft’s positioning relative to Google, not the crowd in the hall. But come on, the leak of the name and the foreshadowing done by Ballmer a few weeks back, this barely deserved to be called an announcement.
Meanwhile, while I was stuck in that keynote fighting to get a slice of WiFi bandwidth and failing, out on the net, Microsoft shipped the .NET 4.0 and Visual Studio 2010 CTP, and the Oslo SDK CTP. I didn’t know about this until I got back out of the PDC keynote hall. Frankly, if Microsoft is releasing a new version of Visual Studio but not making a big deal of it in the PDC conference hall, something has gone seriously wrong. It almost feels like there was originally a different plan, but that the Azure keynote took centre stage at the last minute, too late to change the scheduled VS and Oslo releases.
Something has definitely gone wrong with the Oslo launch. Oslo incorporates a new language, new tools, and represents a new development paradigm; it’s exactly the kind of stuff you can hang a PDC on. The Oslo team have come with a ton of marketing collateral, they’ve got branding, they’ve got t-shirts, they’ve got balloons… in the Microsoft pavilion, this is a big coming out party for a big technology, but if you’re looking to learn about what MS is releasing from sitting in the keynote hall, well you wouldn’t even know about it. Even after Don Box and Chris Anderson did their keynote – they slipped in a tiny demo of IntelliPad, an Oslo tool, but it felt almost like guerrilla marketing, not part of the message they were there to sell.
Day two keynote, we got some real meat, and felt like this was more like what we expect from PDC. But still the foreshadowing announcements had stolen half the thunder. YES we got to see the first ever public demo of the Windows 7 desktop (the first EVER DEMO! Steve Jobs would’ve had us eating out of his hands…), but imagine how much bigger that presentation would’ve been if it had also been the first time MS had confirmed that the product name was Windows 7 (or even if they’d chosen a better name).
As it was, the biggest cheers Microsoft squeezed out of the PDC crowd were for demoing Windows 7 multi-monitor remote desktop, and the announcement that VS2010 was being rebuilt in WPF (and would support multiple monitors too… spotting a theme here?) – other than that, polite ripples of applause greeted a few nice features, but there was no ovation. ScottGu’s demo of customising the WPF-based VS editor was incredibly well-received (it’s always great to watch Scott throw out code on a keynote stage), though the momentum was short-lived.
With the right staging, they could have whipped up the crowd with VS2010 multitargeting support; .NET 4.0/2.0 side-by-side in process; foreshadowing Anders’ C# 4.0 announcements… And what was that tantalising ‘SilverLight running outside the browser’ hint? Was that iPlayer app they demoed really SilverLight acting like Adobe Air? did they just forget to announce that one too?
Failing to keep Azure secret enough; failing to announce Oslo; failing to announce .NET 4.0, VS 2010, C# 4.0 (a language Eric Lippert has been carefully describing as ‘a potential hypothetical future language’ for the past three years) and VB10; failing to use this as a platform to properly launch Windows 7; there’s a lot of fail in the PDC marketing machinery.
But away from the keynotes, that marketing failure doesn’t take away from the fact that MS is launching all that technology here. Real stuff is being announced during every session slot, and I’m having to catch up on the news via twitter and the blogs from the sessions I’m not in. In many ways, it feels like I’d be better informed if I wasn’t actually here on site, but that would mean missing out on the chance to meet up and talk to the teams and get some real insight. I had a great catch up with Jeff King, who’s the guy behind the JQuery intellisense file you can now get hold of; he showed me a few cool new capabilities coming up in VS JavaScript intellisense capability. I also had some great opportunities to chat with guys from the Oslo group, to see how their tools fit with my apps. I’m looking forward to more of the same today, especially tonight at the Ask the Experts session. I’ll also be trying to take in the repeat of Anders’ C# 4.0 talk.
So I’m disappointed in Microsoft’s lack of control over the proceedings, its seeming lack of self-awareness and co-ordination. But I’m not disappointed in the tech, the content I’m seeing, or the people I’m meeting. PDC’s still worthwhile in a blog-driven age because of the connections you can make on site, and no leakage of codenames changes that.
Tuesday, August 05, 2008
#
Just a quick post to (belatedly) announce that I've updated the JQuery VS2008 IntelliSense 'Header file' and generator script. The generator supports the 'newer' style documentation XML, produces a better (more compatible) header file, and the version available on the site is now up to date for JQuery 1.2.6. You can use the script to generate headers for older versions yourself - download the zip file and replace the jquery.js and jquery-docs-xml.xml files, and then run the index.htm generator page.
You can read about the purpose and origins of this file on my original blog post.
There's been a lot of buzz since my original posting about the header file.
- My post was linked from Scott Guthrie's blog
- Brad Vincent posted his updated version (he tweaks some of the return types for methods that sometimes return jQuery objects to get more intellisense - I'll have to see about making that an option on the generator; at the moment it prioritises being correct, possibly at the expense of being helpful). Brad's post includes a great se tof clear instructions as to how to get up and running with the header file.
- Brad's file featured in Scott Guthrie's later post about Visual Studio 2008 SP1 features (it was great to see 'jquery.intellisense.js' turn up in a scottgu screen capture - he commented about where he got it here).
The basic adaptations to support the newer XML format were posted on the jQuery forum by Jeffrey Kretz. I've tried to get in touch with Jeffrey about including his adaptations in my download; I've had no response, so I'm hoping that the included credit for his work is acceptable.
If you just want to download the latest JQuery 1.2.6 header file, it's here.
Monday, July 21, 2008
#
Jon Skeet (author of the truly excellent C# In Depth - I mean truly excellent, the best book I've read in the .NET sphere, bar none) blogged recently about the mess of version numbers in book titles about current versions of C# and .NET technology. He's right, Microsoft's fast-and-loose interpretation of the major/minor version number convention and its separation of language versioning from framework versioning has made a mess that's especially visible in the pages of Amazon's programming section. But I believe Jon's wrong to complain quite so much about the publishers who've elected to title books with the '2008' moniker; it may not be pedantically correct, but it's a darn sight better than most alternatives. The titling of these books is way more complicated than it looks; there are more subtle dynamics than mere precision at play here.
I've been here before, you see. Back in the days before it became a Wiley imprint, I was a kind of series-commissioning-technical-editor-cum-technical-reviewer-cum-author at Wrox Press, so I've got some experience of titling programming books in the face of pathological version numbering disasters. In particular, I cut my editorial teeth in Java books just as Sun, in their wisdom, decided to call the version of the Java platform that they shipped with version 1.2 of the SDK "Java 2". At that point, it wasn't too bad. We were able to bring out books with Java 2 in the title, and everybody could see that they were about a newer technology than those old books about Java 1.1. So far, so rosy.
Then Sun decided to release a new SDK version, 1.3. Suddenly, not so rosy. We needed to sell books that were:
- clearly about a newer technology than older books - the ones with Java 2 in the title - you want a punter in a bookshop with your new book, next to one of your competitor's older books, to pick your new one. "Hmm - Learn Java 2 in 123 microseconds, or Beginning JDK 1.3? I'll take the version 2 book, please!"
- clearly about version 1.3 of the SDK - you want punters who know they want a 1.3 book to find your book. It has to come up when they search for... whatever it is they're going to type when they search for it.
In the editorial team, we had to pick titles and convince the sales team that they could convince the book buyers in Amazon and Barnes & Noble and Borders that the title covered a new technology, would appeal to customers, and justified its place on the shelf instead of all those old books (Chainstore book buyers don't like clearing books that have a good sales history off their shelves, even if you tell them that the technology they cover is obsolete. They know that book sells; they don't trust this new book yet.)
The sales channel is a crucial part of the titling dynamic, because even more than the customer they buy based on title alone. And they shelve by title. A badly titled book will end up in the wrong bookcase entirely. A lot of early C# books wound up under 'C++', while general .NET titles ended up under 'networking' because bookstores didn't have a .NET category yet, and it looked like a networky word.
In the case of Java's 1.3 naming crisis, we tried to title books as new 'JDK 1.3 Editions' of existing Java 2 titles, which worked reasonably. The problem didn't get much better when Sun - in its wisdom - decided to title the next release 'Java 2 SDK 1.4'; My name (and part of my face) appears on a couple of differently-titled books that reflect the uncertainty that name switch caused:
Seriously. We called the technology an abbreviation of "Java Java Two Standard Edition One-Point-Four". And I stand by that - it was the right thing to do from a search optimisation and version clarity perspective, even though it was, pedantically, a mess. But it was Sun's mess. Sun's term - J2SDK 1.4 - was never going to fly in a book title for the sales channel. They knew J2SE (because J2EE titles had been such a success), they new Java, they wanted something bigger than 1.3.
Microsoft have put publishers in a similar position with .NET. In fact, they've put publishers in a worse position, by releasing .NET 3.0 with so few actual changes. Publishers updated general C# and VB.NET books to match the version number - and of course, nobody bought them. People bought WCF, WF and WPF books if they had an interest in those fields, I'm sure, but who would buy a .NET 3.0 update of a generic .NET book? Only newcomers picking up a first title, and only because the 3.0 book had obsoleted the old 2.0 one.
Then 3.5 comes along, and there are genuine changes, and for once people do want to buy books on C# and VB.NET again that cover all these new features, and... the book buyers say "A point release? You saw the sales figures for version 3.0, and that was a major update. The .NET book market is dead. You can't seriously expect us to buy new books for a point release."
Or worse, you come to them with a book claiming to be on C# 3. "Three-dot-oh is DEAD, I told you. What are you doing bringing us new books on last year's tech, when we couldn't even sell books on last year's tech last year? Come back when it's four-dot-oh and we'll talk."
Yeah, you try telling the sales guys it's C# 3 not .NET 3.0... see how well that meeting goes.
Bear in mind that these conversations with book buyers are all going on early in the pre-release cycle; books are announced and ordered six months ahead of release. And all the publishers are trying to pick the same naming convention as each other, because it's critical that if you're going to position your book as a competitor for one of the big players' books, you need to pick the same name for the technology. No book buyer will believe your Crazy C# 3 Tricks book competes with the market-leading Dull Business Applications with C# .NET 3.5 - the titles are totally different. And besides, C# 3 - that must be older than this competing .NET 3.5 book, right? Who cares if they got the title technically wrong, and you're right - what matters on the shelf - or in the search results - is that you're the same.
So what do sensible publishers do? They tie the new release books to the new Visual Studio version. Put 2008 on a book and it looks a lot sharper and more interesting to a book buyer than that tired old 2005 edition (albeit with the 'Covers .NET 3.0!' flash on the cover). You can make up for the version number chaos in the subtitle (because even though nobody reads it, it'll help with search results). So you wind up with Serious C# 2008 Edn: .NET 3.5 Development With C# 3.
Now, not all of the publishers have played this game all that well, I'll admit, and I'm far enough out of the publishing loop that I'm sure my intuitions about how the market influenced the book titles in this release cycle is a mile off the mark, but even if I'm wrong about how it works nowadays, I think it's particularly unfair to criticise them for not using C# 3 in the book titles. That was never going to fly after the .NET 3.0 release.
Still, I guess with the way computer book publishing's been going in the years since I left the industry (that's a correlation, not a causation), optimising book titles is probably a case of rearranging the deckchairs on the beachfront at Atlantis, but I just wanted to point out that in the list of people I considered when I was choosing book titles, version number pedants came pretty low down.
Probably just below the ill-informed marketing people who tried to persuade us that using non-trademarked names instead of the Official Name™ when referring to their product was somehow a trademark violation...
Wednesday, May 28, 2008
#
C# 2.0 brought us the new 'yield' keyword. In a method or property that has an IEnumerable return type, yield return allows us to procedurally generate the values that will be provided by the resulting enumerator - such a method or property being known as an iterator. The assumption behind most uses of yield return is generally that someone will take the IEnumerable C# gins up for us and stick it in a foreach loop (potentially, in a LINQ world, after applying some query operators that filter or transform our yielded values first).
But ultimately that foreach loop will just be calling GetEnumerator on our iterator IEnumerable, then repeatedly calling MoveNext and extracting the Current value from the resulting IEnumerator. There's no rule that says only foreach loops are allowed to do that.
What matters here is that the ultimate upshot of our writing a yielding IEnumerable is that each call to MoveNext executes a bit of the method we write; whenever we yield return (or yield break - or reach the end of the method, which is equivalent to a yield break), that call to MoveNext returns. It returns true if we issued a yield return, and false if we issued a yield break. (The magic state machine that the C# compiler generates for us being an implementation detail we can ignore while working at the abstraction level of the language. Under the covers, all kinds of crazy stuff may go on at runtime, but syntactically, it's useful to imagine that we're executing our method a bit at a time.)
Some important consequences of this. After you yield return, and a call to MoveNext returns true, there's no guarantee that the calling code will subsequently call MoveNext and execute the subsequent lines of code...
- ... immediately
- ... from the same calling method at the same point on the stack
- ... on the same thread
- ... or even at all
although all of these things will normally be the case if the calling code uses a foreach loop. If the calling foreach loop contains a break or return statement, though, for instance, your code won't be called back.
(It's worth noting that the enumerator C# produces for you will be IDisposable, and so there SHOULD be a subsequent call to Dispose() that will, if your iterator code has any finally or using blocks, ensure they are tidied up appropriately - yet more state machine magic sorts this out for you. But of course, if the calling code doesn't explicitly call Dispose(), you'll be waiting for the garbage collector to come along before that disposal takes place. Foreach automatically disposes the enumerator whenever - and however - the loop exits.)
One thing you ought to be able to be reasonably certain of is that if MoveNext is called again, the next part of your code will at least be executing in the same appdomain that the first part was, although - and this is a REALLY dangerous idea - it's possible to hack together a serialization mechanism that allows some iterators to be serialized mid-iteration, then 'thawed' in a completely different appdomain and continue running - so you can't even completely rely upon that.
Now, this is supposed to be a post about dangerous ideas in C#, so obviously the question arises, if you can abuse an iterator by calling MoveNext on, say, different threads, does that let us do anything cool? I think the answer is yes, but just because we can doesn't mean we should. In production code, at least. Here, we can play...
For the purposes of my example, I'm going to introduce the TinyURL API. TinyURL takes long URLs, sticks them in a database for you, and hands you back a shorter URL that you can use in places where lengthy URLs might be awkward, such as Twitter. This short URL redirects people who access it on to the URL you originally supplied. Simple.
If you are writing an application (perhaps a twitter client) that needs to be able to generate tiny URLs on demand, you might need to access TinyURL's API. It's not terribly complicated, as APIs go. To get a tiny URL for a URL, X, you simply send an HTTP GET request to http://tinyurl.com/api-create.php?url=X. No URL encoding is normally needed (since you're passing a URL). The response body will be a plain text string containing the tiny URL.
You can get a tiny URL, then, from .NET code with a one-liner:
string tinyUrl = new WebClient()
.DownloadString("http://tinyurl.com/api-create.php?url=" + url);
That seems far too simple to me. This is .NET, not PHP. Let's see how much we can complicate matters, ostensibly with the aim of making the above call asynchronous. For starters, let's use System.Net.WebRequest instead of System.Net.WebClient. First of all, still synchronously:
public string GetTinyUrl(string url)
{
WebRequest req = WebRequest.Create(GetTinyUrlApiUrl(url));
WebResponse response = req.GetResponse();
return StringFromResponse(response);
}
private string StringFromResponse(WebResponse response)
{
return new StreamReader(response.GetResponseStream()).ReadToEnd();
}
private string GetTinyUrlApiUrl(string url)
{
return "http://tinyurl.com/api-create.php?url=" + url;
}
That's a start - we've used a factory method, so our need to use at least one GOF pattern in our solution is met, and we can feel slightly superior to PHP programmers as a result. But what we've ended up with looks like something a Java programmer might be comfortable with. We can't be having that. What happens when we go asynchronous?
public void GetTinyUrlAsync(string url, Action<string> handleTinyUrl)
{
WebRequest req = WebRequest.Create(GetTinyUrlApiUrl(url));
var handler = new Handler(req, handleTinyUrl);
req.BeginGetResponse(handler.HandleResponse, handleTinyUrl);
}
private class Handler
{
public Handler(WebRequest request, Action<string> handleTinyUrlCallback)
{
Request = request;
HandleTinyUrlCallback = handleTinyUrlCallback;
}
public WebRequest Request { get; private set; }
public Action<string> HandleTinyUrlCallback { get; private set; }
public void HandleResponse(IAsyncResult result)
{
WebResponse response = Request.EndGetResponse(result);
string resultString = StringFromResponse(response);
HandleTinyUrlCallback(resultString);
}
}
Goodness - it looks even more like Java now with that inner class bloating things. And wow - there's 16 lines of code between the call to BeginGetResponse and EndGetResponse, all of which is concerned with the messy business of preserving state between them. Thank goodness we can do better; a C# 2 closure will manage all that messy state for us:
public void GetTinyUrlAsync(string url, Action<string> handleTinyUrl)
{
WebRequest req = WebRequest.Create(GetTinyUrlApiUrl(url));
req.BeginGetResponse(delegate(IAsyncResult result)
{
WebResponse response = req.EndGetResponse(result);
string resultString = StringFromResponse(response);
handleTinyUrl(resultString);
}
, null);
}
Well, we wanted to get the Begin and End calls closer together - now the call to EndGetResponse is actually inside the call to BeginGetResponse. Not sure about you, but that feels a little bit too close to me. It's hard enough to follow asynchronous logic without it all being piled up in a single statement.
Using a slightly different approach, we can preserve a slightly better sequential ordering, but the code isn't getting any prettier overall:
public void GetTinyUrlAsync(string url, Action<string> handleTinyUrl)
{
WebRequest req = WebRequest.Create(GetTinyUrlApiUrl(url));
IAsyncResult result = req.BeginGetResponse(null, null);
ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle,
(state, timedOut) => {
WebResponse response = req.EndGetResponse(result);
string resultString = StringFromResponse(response);
handleTinyUrl(resultString);
}
, null, -1, true);
}
It would be nice to get rid of that anonymous delegate altogether, wouldn't it? And weren't we promised yield statements? Where are the yield statements? Okay, I'm going to cut straight to the chase now and show you this week's dangerous idea:
public IEnumerable<IAsyncResult> GetTinyUrlAsync(string url, Action<string> handleTinyUrl)
{
WebRequest req = WebRequest.Create(GetTinyUrlApiUrl(url));
IAsyncResult result = req.BeginGetResponse(null, null);
while (!result.IsCompleted)
{
yield return result;
}
WebResponse response = req.EndGetResponse(result);
string resultString = StringFromResponse(response);
handleTinyUrl(resultString);
}
Now, that does look a lot tidier - it concentrates the logic on the web request, not on the threading (See why I made you read all the ugly code? It makes this look almost readable by comparison). It actually leaves the threading decision up to the caller. Want to run it synchronously? Here's how:
string tinyUrl;
IEnumerable<IAsyncResult> iterator =
GetTinyUrlAsync("http://blogs.ipona.com/james", str => tinyUrl = str);
foreach (IAsyncResult res in yieldingRetriever) { };
Okay, how did that work? Well, essentially, it sets things up, then the foreach loop at the end runs the whole method, filling the while loop in the middle with a no-op. The upshot is that the thread spins until the async result returns. (you might want to put a Thread.Sleep in the foreach body to stop the loop hogging the CPU).
Now let's see how to run it asynchronously:
IEnumerator<IAsyncResult> enumerator = iterator.GetEnumerator();
WaitOrTimerCallback throwAsyncResultToTheThreadPool = null;
throwAsyncResultToTheThreadPool = delegate
{
// Looking to see if we've got another async result to handle
if (enumerator.MoveNext())
{
// Scheduling op to complete on threadpool thread
ThreadPool.RegisterWaitForSingleObject(enumerator.Current.AsyncWaitHandle,
throwAsyncResultToTheThreadPool, null, -1, true);
}
};
throwAsyncResultToTheThreadPool(null, false);
See why it was a dangerous idea? It leads to messes like this semi-recursive anonymous method from hell (and think how much worse it would look with the error handling left in - notice no call here to enumerator.Dispose(), for instance). But it's actually quite neat - whenever the iterator yields an IAsyncResult, we let the threadpool deal with waiting for it to return. For I/O operations, that won't block a threadpool thread until there's actually data to handle, which is pretty tidy. When the IAsyncResult's WaitHandle releases, the next part of the iterator is executed on an available threadpool thread. If that results in another IAsyncResult, we let the threadpool wait for that too; and so on until there are no more IAsyncResults - in other words, until the method completes.
Edited 29th May 2008 to add:
Now, I want to emphasize the really cool effect this has. The first half of the method runs on the original calling thread. The second half of the method, after the yield return, runs on a threadpool thread - and not until the web request has returned. Let me repeat that. The second half of the method runs on a different thread. And yet it uses the same local variables, the same parameters, the same everything as the first half of the method.
This is an example of what a computer sciency person would call a 'continuation', by the way. C# yield return statements make it dead easy to stop execution of a method, take the enumerator that represents the state of the method at that point (that's the 'continuation'), pass it around in our code until we want to carry on running it, and then execute the next bit, with all the same local variables and data as we had before.
Now, you're not going to want to write code like that to call the iterator every time. But the beauty of it is that these strategies can be encapsulated in completely generic methods:
public TReturn RunIteratorSynchronously<TArg, TReturn>(Func<TArg, Action<TReturn>,
IEnumerable<IAsyncResult>> iteratorGenerator, TArg arg)
{
TReturn returnValue = default(TReturn);
IEnumerable<IAsyncResult> iterator = iteratorGenerator(arg, val => returnValue = val);
foreach (IAsyncResult res in iterator) { Thread.Sleep(1); };
return returnValue;
}
public void RunIteratorAsynchronously<TArg, TReturn>(Func<TArg, Action<TReturn>,
IEnumerable<IAsyncResult>> iteratorGenerator, TArg arg, Action<TReturn> callback)
{
IEnumerable<IAsyncResult> iterator = iteratorGenerator(arg, callback);
IEnumerator<IAsyncResult> enumerator = iterator.GetEnumerator();
WaitOrTimerCallback throwAsyncResultToTheThreadPool = null;
throwAsyncResultToTheThreadPool = delegate
{
// Looking to see if we've got another async result to handle
if (enumerator.MoveNext())
{
// Scheduling op to complete on threadpool thread
ThreadPool.RegisterWaitForSingleObject(enumerator.Current.AsyncWaitHandle,
throwAsyncResultToTheThreadPool, null, -1, true);
}
};
throwAsyncResultToTheThreadPool(null, false);
}
So I can now call the IAsyncResult-yielding TinyUrl iterator with either of these approaches as follows:
Console.WriteLine(RunIteratorSynchronously<string, string>(GetTinyUrlAsync,
"http://blogs.ipona.com/james"));
RunIteratorAsynchronously<string, string>(GetTinyUrlAsync,
"http://blogs.ipona.com/james", Console.WriteLine);
Edited 29th May 2008 to add:
Also, worth thinking about what other execution strategies you could try. How about a scheduler that handles a number of these iterators all on one thread, by looping through them calling MoveNext() on each one in turn? That would basically be an implementation of a co-operative 'fiber' multitasking system...
And it's when you end up with something as tidy as that that it becomes clear just how dangerous an idea it was in the first place, because you end up being tempted to justify it...
Remember, James Hart is a professional stunt programmer. Do not try this on a production codebase.
Friday, May 23, 2008
#
Switch statements are rubbish. Only work on strings and numbers. Require an exact match value.
Let’s abuse, oh, to pick a C# feature at random, collection initialisers, and use it to make them better!
string switchValue = "house";
switch (switchValue)
{
case "hello":
Console.WriteLine("Value was hello");
break;
// But how can we handle, say, all other strings that start with h?
default:
Console.WriteLine("No match - Fallthrough");
break;
}
// here’s one way:
new Switch<string>(switchValue)
{
{ "hello",
()=> { Console.WriteLine("Value was hello"); }
},
{ x => x.StartsWith("h"),
()=> { Console.WriteLine("value started with h, but wasn't hello"); }
},
{ // default
() => { Console.WriteLine("No match - Fallthrough"); }
}
};
// how about a case insensitive comparison? switch (switchValue.ToLower()) ?
// or...
new Switch<string>(switchValue, StringComparer.InvariantCultureIgnoreCase)
{
{ "HOUSE", ()=> { Console.WriteLine("Value was house"); } },
{ x => x.StartsWith("H", StringComparison.InvariantCultureIgnoreCase),
()=> { Console.WriteLine("value started with h, but wasn't house"); }
},
{ () => { Console.WriteLine("No match - Fallthrough"); } }
};
// what about an expression equivalent?
string val = new Switch<string, string>(switchValue)
{
{ "hello", ()=> "Value was hello" },
{ x => x.StartsWith("h"), ()=> "value started with h, but wasn't hello" },
{ () => "No match - Fallthrough" }
};
Console.WriteLine(val);
// what about things other than strings?
DateTime today = DateTime.Today;
// can't switch (today)... but we can Switch<DateTime> on it:
new Switch<DateTime>(today)
{
{ new DateTime(2009, 01, 01), () => { Console.WriteLine("Happy 2009!"); } },
{ day => day.Day == 1,
() => { Console.WriteLine("It's the first day of a new month"); }
},
{ () => { Console.WriteLine("Nothing happening today"); } }
};
I should point out that I do have an implementation of the Switch class used here (well, technically Switch classes, and a bonus point to anyone paying enough attention to spot the second one), and it works exactly as shown, but I've less of an urge to share the grubby details of the implementation than to show the twisted client code it enables.
Next time on Dangerous Ideas in C#, we'll see what havoc can be wrought with 'yield return' statements.
Monday, March 17, 2008
#
What I cannot create, I do not understand.
Richard Feynman's reported last words
I reinvented the wheel last week. I sat down and deliberately coded something that I knew already existed, and had probably also been done by many many other people. In conventional programming terms, I wasted my time. But it was worthwhile, and what's more I would recommend almost any serious .NET programmer do precisely the same thing.
I recreated Linq.
Well, slight exaggeration there. More precisely, I recreated the basic Linq-to-objects functionality provided by the System.Linq.Enumerable class in the .NET 3.5 System.Core assembly.
Why on earth would I want to do that?
Well, a couple of reasons. The first is practical. Our product at work runs on .NET 2.0. Since we don't make use of WPF, WCF or WF, we never moved our platform requirements on to .NET 3.0, and since .NET 3.5 drags that heavyweight stack of useless (for our purposes) bumph along with it, we haven't moved on to 3.5 either. Nevertheless, we're using the .NET 3.5 generation tools, which include the C# 3.0 compiler. That gives us the ability to compile .NET 2.0 applications from code that makes use of all those exciting C# 3.0 features:
- Auto-implemented Properties
- Object initialisers
- Collection initialisers
- Anonymous types
- Partial classes
- Extension methods
- Linq query syntax
Except that... you don't get the last two, after all. Write C# code that uses extension methods or Linq queries and compile for .NET 2.0, and you won't get very far. Because Linq requires that the objects you're querying support certain methods, which almost certainly won't be there (although there's actually nothing to stop you adding such methods directly to your classes and then writing Linq queries). What you really want to do (or what I really want to do, at any rate) is write Linq queries against IEnumerable<T>, and that means you need to have extension methods. And extension methods depend upon being able to mark methods with the System.Runtime.CompilerServices.ExtensionAttribute, which is in the .NET 3.5-only System.Core assembly.
So we get everything else, but not extension methods, and thus not Linq.
But.. it turns out you can fool the C# compiler. It isn't, it turns out, looking for System.Runtime.CompilerServices.ExtensionAttribute in System.Core, it's just looking for System.Runtime.CompilerServices.ExtensionAttribute anywhere it can find it. And if you include the following class definition in your compilation, it will happily build code that declares extension methods, and then you can also compile code that calls them:
namespace System.Runtime.CompilerServices
{
public class ExtensionAttribute : Attribute
{
}
}
So, that gives us one more feature off the list. But we still can't use Linq against IEnumerables, because we don't have System.Linq.Enumerable and all its extension methods.
But there's nothing to stop us writing our own version...
Nothing, that is, except good programming sense. Normally, my well developed programmer spidey senses start tingling whenever I find myself thinking 'well, I could just build my own version....', and with good reason. It goes against all the instincts of "once and once only", "don't repeat yourself", and the aversion to reinventing the wheel, that a sensible programmer has ingrained in the soul.
And yet, I built my own version of the Linq to objects library. What was I thinking? Why not just step back, and say "Well, obviously we can make do without Linq?" We can, true. But a couple of things drove me on.
- It would be fun, and interesting. Most importantly, it would be a good way to get the hang of what Linq really does and how it works.
- I had access to a good test suite. For reimplementing an existing library of code, I needed a test suite, and one was already available in the form of the 101 Linq samples Microsoft produced as part of the VS2008 documentation. All I needed was a way to compare output of the samples running against my Linq library against the samples running against the System.Core implementation to get a relatively high degree of confidence that the implementation was sound. I really wouldn't have proceeded - nor got so far, frankly - without this.
Now, I'm probably not the first person to do this. And for a moment I considered googling for an existing implementation (I still haven't, actually. Anyone know if there's a good solid Linq-to-objects for .NET 2.0 out there already?). But I figured there might be licensing restrictions on any existing code, and besides, remembering point 1, where would be the fun? Equally, checking out the source code for System.Core's implementation, which is only a few clicks away in the VS2008 debugger these days, was a no-no, because that way lies EULA infringement hell.
But I'll admit, the main thinking was "Well, it'd be a good way to figure out how Linq ticks...", so that was reason number two.
So I set out by first of all modifying the Linq 101 samples to dump their output into a series of files, one per sample. I set those aside in a reference output directory.
Next, I recompiled against my (currently empty) Linq library. Obviously, I hit a lot of compile errors. These I fixed one batch at a time by stubbing out empty Linq operator extension methods. I took advantage of the great work done on Hooked On Linq to snag the method signatures and pseudocode explanations for each method, and got the app to a compilable state. Then, with a BeyondCompare window open showing the difference between my output files and the reference files, I set about making my output match the reference output (I could have had an NUnit harness compare the results directly, but the BeyondCompare output was a lot more revealing than a string mismatch assertion failure was going to be).
It didn't take that long, actually - about six hours' coding, I'd guess, including the test harness work. Linq against IEnumerable isn't that hard to write; although you do end up building a lot of methods to support every scenario in the 101 samples, none of them is that complicated - especially if you make the simpler overloads call into their more complex siblings and leverage your own growing codebase to build the more complex features, instead of building each operation in isolation. I did make use of some lightweight code generation to implement the 40 variant numerical aggregates (sum of a list of integers, sum of a list of doubles, average of a list of integers, etc.) - Visual Studio text templates are becoming my favourite toy of the moment; even so, so far I've not implemented the equivalent operations on lists of nullable numeric types. I had to draw the line somewhere.
How good are the results? Well, I'm outputting the same answers as the MS Linq library in general, with a few caveats. Occasionally I produce the results in a different order, and my Average implementation produces slightly different answers because I generate the average cumulatively, rather than accumulating the sum and dividing by the number of values (easily changed, but not just now). I'm sure that the 101 samples don't give that thorough a coverage of the code, either; they work as a good first pass test that the core functionality of filtering, grouping, joining and aggregating works, but they don't exercise corner cases (empty or infinite lists, for instance) or error scenarios, so a bit more robust unit testing around the framework might be worthwhile. (How does one unit test an infinite list, by the way?)
Still, all things considered it was a very worthwhile exercise for me personally. I now know what some of the funkier bits of the Linq query comprehension translation are asking for when they generate calls to GroupJoin and SelectMany. I discovered that writing the ThenBy method that is used after a call to OrderBy was probably the most complicated bit of the entire API (unless I'm missing something obvious). I found out about a few functions that exist in Linq that I didn't know about before, like Range, ToDictionary, and some of the variants of Aggregate.
So all in all, I think it makes for a pretty worthwhile exercise for anyone who has any intention of becoming familiar with Linq. Having built it up myself, I've now got a much greater understanding of what it's doing, and that's no bad thing.
Thursday, February 21, 2008
#
According to the MSDN documentation for Finding the Metabase Path when Given the URL, such a facility might be useful because "When IIS servers host a lot of Web sites, a Web site adminstrator might not remember the metabase path of each site or virtual directory."
Clearly, since I can't remember the metabase path of even the site root of my default website, let alone of every image file on my web server, I am not a real web administrator.
Wednesday, February 20, 2008
#
I've been working on the installation tooling for our web application product recently, finally getting round to upgrading it with IIS7 support. The setup tool is used to deploy different versions of our app onto sales demo and training machines, as well as onto real servers, so it has to handle a wide range of different deployment scenarios. We have to support deploying the .NET 1.1 or 2.0 versions of our code, onto anything from IIS 5.0 on a Windows Server 2000 machine to IIS 7.0 on Windows Server 2008, or onto desktop environments from IIS 5.1 on Windows XP to IIS 7.0 on Vista. And we can't assume .NET or IIS will be installed before our tool runs.
This week, I've been concentrating on dealing with determining if prerequisites are met for installation. Obviously, one prerequisite for the application is that IIS must be present. With IIS 5 or 6, we could look for IIS as a monolithic component, and assume that if we could find evidence of its presence, then the whole thing was installed. IIS7 changes the game, with a slew of individual optional components that can be installed. Just because IIS is there, doesn't mean all the features we require are.
So the first question facing our installer when it finds itself on a system where IIS7 is the supported server - I'm diagnosing that condition with a check for a CurrentVersion of 6.0 or greater at the moment - is to see if the components we rely upon are already installed.
On Windows Server 2008, it looks like we'd be able to ask a program called oclist.exe (it lists optional components, hence, Optional Component List) and determine which IIS features are present. But on Vista, sadly, no such toy is present. We have the analagous installation tool, ocsetup.exe, which installs optional components for us; we also have a related - more granular - tool called pkgmgr.exe, which manages the installed optional feature packages. And we have the GUI tool, optionalfeatures.exe, which allows us to manage installed packages through a nice tree view.
But there's no way to determine using these tools what's already installed. To find that out, after a lot of googling, I finally found that I need to be looking in the registry under HKEY_LOCAL_MACHINE\Software\Microsoft\InetStp\Components.
So, let's say I want to see if ASP.NET support is installed on the system. We need to look in that registry location for a key called 'ASPNET'. If we don't find it, then we know ASP.NET isn't installed, and we can then use pkgmgr.exe to install ASP.NET. pkgmgr.exe calls the ASP.NET component IIS-ASPNET, so we would use the command:
pkgmgr.exe /iu:IIS-ASPNET
So far so good. Except that the IIS-ASPNET module is a child of IIS-ApplicationDevelopment, which belongs to IIS-WebServer, and that is part of the overall IIS-WebServerRole feature. If these are not already installed, we have to specify them on pkgmgr.exe's command line also:
pkgmgr.exe /iu:IIS-ASPNET;IIIS-ApplicationDevelopment;IS-WebServer;IIS-WebServer
Role
And on top of that hierararchical dependency chain, there are also cross-dependencies. IIS-ASPNET directly requires modules called IIS-DefaultDocument, IIS-NetFxExtensibility, IIS-ISAPIExtensions, IIS-ISAPIFilter, IIS-RequestFiltering and WAS-NetFxEnvironment (this is all documented here). And all of these modules have dependencies of their own. In fact, unless you know any of them are already installed, the command to install IIS-ASPNET would be:
pkgmgr.exe /iu:IIS-ASPNET;IIS-ApplicationDevelopment;IIS-WebServer;IIS-WebServer
Role;WAS-ProcessModel;WAS-WindowsActivationService;WAS-NetFxEnvironment;IIS-Requ
estFiltering;IIS-Security;IIS-ISAPIFilter;IIS-ISAPIExtensions;IIS-NetFxExtensibi
lity;IIS-DefaultDocument;IIS-CommonHttpFeatures
Worse, even if all of these modules were already present, the command will take a considerable amount of time to run. And require an administrative elevation even if it has nothing to do. So running the command 'just to make sure' that the prerequisites are installed isn't really the best option.
So I've taken matters into my own hands and put together a command line tool to help. The tool does three things:
- calculates the package dependency list for a given list of required features
- determines which IIS features are currently installed
- runs
pkgmgr.exe for you with the necessary arguments to ensure a particular list of features are supported.
The tool's called IIS7.cmd. Here's some examples of its use:
c:\tools>iis7
IIS-WebServerRole [INSTALLED]
IIS-WebServer [INSTALLED]
IIS-CommonHttpFeatures [INSTALLED]
IIS-StaticContent [INSTALLED]
IIS-DefaultDocument [INSTALLED]
IIS-DirectoryBrowsing
IIS-HttpErrors [INSTALLED]
IIS-HttpRedirect
IIS-ApplicationDevelopment
IIS-ASPNET
IIS-NetFxExtensibility
IIS-ASP
IIS-CGI
IIS-ISAPIExtensions
IIS-ISAPIFilter
IIS-ServerSideIncludes
IIS-HealthAndDiagnostics [INSTALLED]
IIS-HttpLogging [INSTALLED]
IIS-LoggingLibraries
IIS-RequestMonitor [INSTALLED]
IIS-HttpTracing
IIS-CustomLogging
IIS-ODBCLogging
IIS-Security
IIS-BasicAuthentication
IIS-WindowsAuthentication
IIS-DigestAuthentication
IIS-ClientCertificateMappingAuthentication
IIS-IISCertificateMappingAuthentication
IIS-URLAuthorization
IIS-RequestFiltering
IIS-IPSecurity
IIS-Performance [INSTALLED]
IIS-HttpCompressionStatic [INSTALLED]
IIS-HttpCompressionDynamic
IIS-WebServerManagementTools [INSTALLED]
IIS-ManagementConsole [INSTALLED]
IIS-ManagementScriptingTools
IIS-ManagementService
IIS-IIS6ManagementCompatibility
IIS-Metabase
IIS-WMICompatibility
IIS-LegacyScripts
IIS-LegacySnapIn
IIS-FTPPublishingService
IIS-FTPServer
IIS-FTPManagement
WAS-WindowsActivationService [INSTALLED]
WAS-ProcessModel [INSTALLED]
WAS-NetFxEnvironment [INSTALLED]
WAS-ConfigurationAPI [INSTALLED]
Here we're calling the tool with no arguments, and it's showing us that we have a basic IIS system installed. The command output shows the dependency hierarchy (it's similar to the tree structure shown by optionalfeatures.exe).
So let's ask the command what we'd need to install to get ASP.NET working on this machine:
c:\tools>iis7 /q IIS-ASPNET
IIS-ASPNET;IIS-ApplicationDevelopment;IIS-WebServer;IIS-WebServerRole;WAS-Proces
sModel;WAS-WindowsActivationService;WAS-NetFxEnvironment;IIS-RequestFiltering;II
S-Security;IIS-ISAPIFilter;IIS-ISAPIExtensions;IIS-NetFxExtensibility;IIS-Defaul
tDocument;IIS-CommonHttpFeatures
We're specifying the '/q' switch to query dependencies. That's the full list of dependencies, not taking into account what we already have installed. To get the list of just what's missing from the current system, we can use the capitalised switch, /Q:
c:\tools>iis7 /Q IIS-ASPNET
IIS-ASPNET;IIS-ApplicationDevelopment;IIS-RequestFiltering;IIS-Security;IIS-ISAP
IFilter;IIS-ISAPIExtensions;IIS-NetFxExtensibility
Note that if we'd requested the outstanding dependencies for a feature that's already installed, we'd have got a blank output:
c:\tools>iis7 /Q IIS-DefaultDocument
c:\tools>
That provides us with that quick prerequisite check we were after. Is this feature present? If it is, empty response. If not, we'll get a list of everything we need to install to make it present.
If we want to go ahead an run the install, we can do so directly with the /I switch:
c:\tools>iis7 /I IIS-ASPNET
Installing modules.
Installation succeeded.
Behind the scenes, that command runs
pkgmgr.exe /iu:IIS-ASPNET;IIS-ApplicationDevelopment;IIS-RequestFiltering;IIS-Sec
urity;IIS-ISAPIFilter;IIS-ISAPIExtensions;IIS-NetFxExtensibility
i.e., it passes the minimal set of packages to install.
As well as individual packages, we can supply a list of packages we want to ensure. If we wanted ASP.NET and Server-side includes, we could specify both:
c:\tools>iis7 /I IIS-ASPNET IIS-ServerSideIncludes
The configuration file (from which the tool reads all the information it needs to know about package interdependencies) also contains several predefined 'installations', which are lists of standard module sets. So, for an application that needs a basic web server with ASP.NET and Metabase support, you could use the predefined installation sets:
c:\tools>iis7 /I Basic ASP.NET Metabase
And you can mix and match installations and module names. To add dynamic compression to the above configuration:
c:\tools>iis7 /I Basic ASP.NET Metabase IIS-HttpCompressionDynamic
The iis7.cmd file itself is just a wrapper around a Windows Script Host iis7.js script to simplify the command line syntax (you don't want to be typing cscript /nologo every time you run it). The dependency information (and the mapping from pkgmgr module IIS-* module name to HKLM Components registry key names, which aren't always the same) is all contained in iis7.xml.
If you want to take a look, I've posted a zip for download.
Monday, February 18, 2008
#
Front of BBC News technology page this morning:
"Game creators look to the future"... accompanied with a picture of a costume designed in 1977. From a film set 'a long time ago' (in a galaxy far far away). Ah well.