Roslyn changes are starting to kick in - 2014/15 should be pretty exciting!

"This means that folks like the Mono team could look at this Reference Source and not have to gouge their eyes out afterwards. "

Very reassuring as well.

The reference source license doesn't allow any use but reference. Does that help Mono? Wouldn't the possible pollution still mean someone that's seen the source can't contribute to Mono? (I had the private shared source access in 2004; AFAIK Mono would still ban me from contributing.)

It looks like there are still big censored parts: http://referencesource-beta.microsoft.com/#PresentationFrame...

One of the first things I had to do for my last job was subclass Window to support custom borders (Zune Player/VS2012/Office 2013 style.) I took a peek at the reference source, and I couldn't help but think the missing bits were the juicy bits, workarounds for window manager bugs or insights into how it does things.

No, we're NOT intentionally censoring anything. This looks like a bug in the source processing. We'll take a look.

If you see a munged file, report it with the feedback link. Sometimes regex's go bad and we can fix them.

That's good to hear! I only guessed the missing parts were censored because a lot of them came after comments that implied the following code was security critical, like here: http://referencesource-beta.microsoft.com/#PresentationFrame...

I apologise if this isn't the case, and I'll definitely use the feedback link if this issue comes up again.

Can anyone explain the benefits of a "compiler as a service"?

Mainly for extension authors and people writing tools that interact with source code. Having a "compiler as a service" allows you to generate data structures and query the structure of the source code you are working with without the need for writing your own parser.

For example, you could use the "compiler as a service" to create a semantic model from source code and then query it for "all classes that contain a public method that accept an int and return a bool". This might sound trivial, but using a tool like Roslyn to do this is much easier than writing a parser that handles a language and then doing the same thing.

To build on this:

1. It's nice to have the same view of the code that your compiler has. This matters less for C# than it would for C++, but it's nice to know you handle the edge cases in a consistent fashion.

2. Invalid code. Since Rosalyn is designed for being used in VS while you're in the middle of typing, it has advanced functionality for handling incomplete or erroneous code. If building your own parser for valid code is hard, building a parser that also gracefully handles half-finished lines is an order of magnitude harder. And having it handle half-lines in a fashion consistent with how VS handles it is another order of magnitude.

As far as visibility to the end user: For those of us using something like Resharper, there won't be any digression between errors reported by Resharper and errors being reported by the compiler.

This also means that the general bar for building code analysis tools has been lowered by quite a bit

The two parent's answers are completely correct.

Does this mean the C# compiler has now been refactored, so doing type inference on, say, fields, can be done?

Probably not: http://blogs.msdn.com/b/ericlippert/archive/2009/01/26/why-n...

EDIT: On second thought, Mads said primary contructors are being considered, and I think that would make implicitly-typed fields possible.

That post says "Doing so would actually require a deep re-architecture of the compiler." I thought Roslyn would satisfy that part.

I've talked to members of the Roslyn team and asked about some features I would have found interesting to get added to the C# team (immutable, non-nullable data-types, type-inference on return-types, etc).

They seem very much on top of their game, and their response has been that lots of these features would probably introduce problems with existing base-class libraries and classes. If they were to introduce those features, they would have to do it at release 1, not 10 years later. They agreed it would have made the platform better, but now it's too late.

As for the "deep architectural" changes required, Roslyn is indeed that change. But the team was very clear on wanting to deliver in stages. 1: Deliver the new compiler and then 2: later when it all has proven to work well, then start using this new architecture to deliver new features.

This is generally accurate, but the last paragraph is a little suspect -- we're iterating on C# 6 right now and using the Roslyn compiler to build the features necessary for it.

I don't think compiler architecture is seriously stopping features in their tracks -- there's only a cost/benefit analysis. 'MichaelGG's suggestion on inferred members, for example, has implications on the contracts of types and usability/user interaction with the language, so it's not just a question of if we can do it, but if we want to do it.

I haven't directly worked on Roslyn (although I work in another area in Cloud and Enterprise) but my understanding is that by providing its client an abstract representation on semantic C# source, Roslyn allows applications like this to be developed.

If a traditional compiler has an API of "source files and configuration flags in, object files out", the Roslyn API is completely different. Roslyn is about designing and providing a semantic API over C# language analysis and compiler services.

http://go.microsoft.com/fwlink/?LinkID=230702

Some of the features of the reference source website enabled by Roslyn's semantic understanding of symbols (assemblies, namespaces, types, members) are color-coding, search, browse, jump-to-definition, and deep-linking.

For example, I can deep-link to part of WPF like so:

http://referencesource-beta.microsoft.com/#PresentationFrame...

I can see which assemblies depend on it, which assemblies it depends on, and browse the source of the assembly in a tree view either by source file or by namespace -> type -> member name.

You can perform semantic analysis of a codebase, which is extremely useful for Visual Studio extensions. Beyond simple refactoring, it should be possible to make tooling a lot more intelligent.

today my source code is compiled three times because my plugins cannot hook into the analysis of the primary compiler.

compilation of code fragments, plugging into compiler processing pipeline, implementing new language constructs like macros, easier creation of domain specific languages, maybe some reasonable form of scripting

Here's very cool example of what compiler as service can do: let's say you want extend SQL so that you can write C# expressions in SELECT like this:

  SELECT MyClass1.Method1(ColumnA, Math.Round(ColumnB))
  FROM Table1

Another example is you can extend your scripting language so that user can use full power of Linq.

So in otherwords, you can extend any of your little languages so that you can mix C# in it. Considering C# supports tons of functional features, Linq, anonymous types, dynamic types etc this can really beef up host language.

For what it's worth if you set up your SQLCLR assemblies right, that line of Transact-SQL can run just fine now, on SQL Server 2005 and later.

I thought that Roslyn expressly did not support extending the language.

My understanding is that you use Roslyn to use C# anywhere. It can parse expressions and compile and execute them.

E.g. If you're writing a BASIC interpreter:

    10 PRINT "Hello World"
    20 CSHARP "Console.WriteLine(42);"
    30 GOTO 10

You call Roslyn on line 2 with the expression to parse and execute.

Roslyn expressly does not support accepting the entire string that he provided. So if you want SQL wrapping C#, Roslyn won't support it until (and if) it supports SQLCLR.

However what Roslyn does support is "transform this snippet for me", which is useful if you know the boundary between your code and code it cares about.

Roslyn sounds pretty cool. For the record, did Microsoft originate the 'compiler as service' idea or did it originate somewhere else?

Mono has had compiler as a service for quite some time now

They had the implementation, yes. But the idea predates Mono's implementation. I can't find any references offhand but I do believe Roslyn inspired the Mono team. Roslyn has been in development for a very long time.

Roslyn was announced several months after Mono has created their compiler as a service implementation. Having said that, clang/llvm was out there for some time already. So yes, the idea predates Mono for quite some time:-)

I know this is getting a lot of fanfare but let us apply four major costly problems I'm having with the CLR/Framework at the moment:

1. Microsoft.Build.Tasks (the bit that has been screwing with me all day) contains static sealed classes with nothing in them. I thought this was a reference source rather than a source of empty interfaces and empty implementations? Still having to resort to reflector to find out what is going on. Problem not solved.

2. So, ReaderWriterLockSlim has a bug in it with respect to low CPU allocation in virtual machines where 20 concurrent threads accessing a locked object at the point of write cause the spin lock in the implementation to thrash and up blows a race condition taking out the process. I can actually see where the bug is now. Where do I submit this bug, bear in mind I can actually recreate it? I phoned up our MSDN rep - they said raise it on Connect. There isn't a place for it on Connect. Phone up generic paid support, three levels of technical staff bounce me around for a week. as a PAID UP GOLD PARTNER, this sucks. If you open your source, track bugs openly too. Problem not solved.

3. This is only the core framework. We really need the entire lot including all dependencies and external tools. There aren't any symbols on symbol server or this for VSTO or Sharepoint for example? What we do get is another black box somewhere in the system that we can't inspect or poke inside even though it's a wrapper for a COM API or a product API. Problem not solved.

4. Does Rosyln/next MSBuild support parallel compilation? Being tied to a single compilation core on a VS2013 build means we have 8 core Xeons sitting there doing nothing. Our only hope of speeding this up is cranking up to high end i7 to get one core with crazy integer performance. 20 minute sit down compiles are too much these days. http://xkcd.com/303/

I understand this is progress and Roslyn is cool and all that, but what does it deliver for every day users of the CLR/Framework? It doesn't really make our day much easier. In fact stuff that really improves developers' lives has stopped appearing completely and instead we get tedious reengineering projects and new features designed only to lock us further into the system (VS online+sign in anyone?)

Also how does this compare to say LLVM/Clang?

I imagine this semantic compilation stuff will make some static analysis stuff easier and stamp on the toes of Coverity etc but that's about it.

Edit: sorry if this is purely a content-free rant and unpopular but after 12 years of this crap from Microsoft, I want to see new wheels, not the same ones reinvented and remarketed. Developers are lapping up this crap constantly as if it's as good as it gets. We need some REAL CHANGE towards openness and consistency and to ask the community and users what they want rather than telling us what we need.

So you have found some esoteric issues that are likely encountered by less than .01% of developers and this is somehow 12 years of crap from Microsoft?

I find it funny (not really) that somehow this floats up as the top comment when most developers have no clue what he is talking about but since this is HN and someone is saying how bad Microsoft is then lets upvote. Microsoft killed my pappy and all (ht to shanselman) so I must do everything I can to make sure no one else dares like them.

So, instead of addressing his points, you instead chose to make fun of it, criticize HN and invoke a marketing piece to somehow prove your stance on the matter. Nice going.

Note that I find Roslyn an interesting project and it's always a WTF for me whenever I touch .NET that I'm not able to easily view/step through the source code of everything I use, including the standard library, so for those suffering, this article is actually good news - and I'm fed up with negative and uninformed opinions on anything interesting, especially on my current platform of choice that's very trendy to bash lately, which I won't name because it would defeat the purpose - because it's much easier to provide uninformed, non-constructive criticism then it is to build stuff, plus we tend to treat our favorites as sports teams with us behaving like cheerleaders, am I right or what?

But for the uninformed reader, such as myself, comments such as yours just add to the noise, I mean WTF?

I get your point but I do empathize with the OP in the msoft bigotry here gets really old. Those of us that use it feel ostracized and bucketted to second class citizen status. For those lost in the .net world, I'll say the feeling is mutual on the other side of the fence.

I suspect part of the grind is that code changes for microsoft requires a litany of testing and release planning and such. Being beholden to a paying userbase ties ones hands in numerous ways (god forbid their patch breaks something else). That said, there is no excuse for sloppy communication.

Oh, so can I open the project in the IDE, click on any identifier and have the source-code of the library in question automatically downloaded, after which the IDE would navigate to the right file and line? I also expect it to work while debugging, no matter the library I'm using, including the standard library, so does it work?

Because that's what I expect from a "modern" developer's environment in year 2014.

This perhaps 0.01% of the problems I've encountered for reference.

I work in other ecosystems as well and there's just not as much crap to deal with by several orders of magnitude.

Which ecosystems? And if you work in those as well then why are you bothering with an ecosystem that has "several orders of magnitude more crap"?

.01% is a small number - so you have 3,000-4,000 problems with the CLR and .NET framework.

Sorry to translate literally, but this kind of gross exaggeration and lack of data is what creates the "Microsoft sucks!!" attitude that is based on a 10-year old reality (at best).

To quote Bill Nye - I find this claim to be extraordinary.

I'm not sure if you're meaning something else, but is /M or /MP not the direction for you? Though C# compiles can be pretty long, my experience is it's usually File I/O and ancillary tasks that take up the majority of the time spent building. Even our primary product which is an absolutely enormous webapp with tons of projects takes only 11 minutes to fully compile and about 5 of that is just shuffling files around.

To take from another conversation, msbuild is very powerful but it's not very usable at all. So much tribal knowledge and secret handshakes you can't learn unless you have a contact in devdiv.

Beyond that, I've opened 3 incidents with Microsoft in the past and they've been nothing short of amazing. One guy even busted out an assembler and stepped through my code to identify that our issue was actually a deliberately closed accidental exploitation path. Enormously frustrating at the time, but he even provided a workaround for us but, like you say, it took several months.

To be specifically productive though: I feel like Roselyn does provide lots of value to developers in terms of what VS plugins can do. There are infinity re-implementations of the MS Compiler out there in terms of plugins and analysis tools. Would be nice to have everyone at least speaking the same language. I think the other thing this initiative does is put the license out there with permissiveness and clarity. Mono can feel comfortable lifting the concepts and integrating it in to their own without worrying about being sued now. This is all great things that they are making available their compiler as a tool and non-platform limited licensing. But - it's only a step it's not the end-game.

What are things you think really improve developers' lives that microsoft used to build they don't build now?

Your support experience sounds really bad.

I'd file the bug here: http://connect.microsoft.com/VisualStudio (which despite the name is also the right place to report library issues)

It's always been bad. I'm the chief technical contact for a system with 50 developers and about 11,500 users. We've raised 20 cases which we couldn't work around and had one officially released solution (bug in IE11). We either have to work around or push a registry fix at great cost to thousands of users worldwide as an out of band patch that we package ourself.

For example there was a regression in ClickOnce with download prompting between IE8 and IE9 due ot the download bar throwing away some compatibility settings. Overnight, when IE9 became mandatory in Windows Update, we had 2000 users down. We got all the users to roll back to IE8. It took 6 months for a registry fix to be sent out to us from a 3rd line support executive after being on the phone daily about it. This is still not fixed in IE11. We've rewritten all our ClickOnce launch points to work around the regression. This took one developer 2 months of work. The regression appeared in the RTM IE9, not the previews.

As for that being the correct place, it's obviously not as the issue was raised there, marked private and closed as not reproducible despite test cases being issued. Not a single contact back from Microsoft bar "closing as can't reproduce". Our rep suggested contacting another support team. Even they aren't helpful.

I need a new job, writing C on Unix...

Looks like Microsoft has some deep structural problems with its developer support.

Let's hope the right persons at Microsoft see this thread.

I have a 100% reproducible (hang/freeze) bug in Visual Studio, with crash dump and everything. It's in 2012, so it'll probably never be fixed. The one developer who responded ignored the information I provided, anyway.

I can no longer find the bug on the dashboard.

If you browse through connect there's some really unhelpful responses from Microsoft and many people trying very hard to report problems in the most helpful possible way without getting a response. The idea is good but execution is weak.

Regarding #4, Mads Torgersen, the C# PM, intimated that the Roslyn compiler is slower than the C++ one, but will take advantage of more cores.

Source: http://vimeo.com/84677184 (07:55)

The c# build system (assemblies) is completely mis-designed for performance. I wish .net assembly modules were supported for separate compilation and link step.

2. This is on .NET 4? Send me the repro.

3. Yes, we're adding more libraries.

4. Have you tried using /maxcpucount on MSBuild to enable parallel compilation?

It's hard to disagree with what you've said, especially after Phoenix fiasco. Also, it's quite a shame that opening bugs for open source software requires following through on Connect.

What happened in Phoenix?

Not in Phoenix, with Phoenix - Phoenix Compiler and Shared Source Common Language Infrastructure:

http://research.microsoft.com/en-us/collaboration/focus/cs/p...

This was supposed to be what clang/llvm has become.

If you've got a problem with ReaderWriterLockSlim, email me (check my profile). I'll get it to the right people.

"Yo .NET Reference Source Release, I’m really happy for you, Imma let you finish but I have the best self-proclaimed content-free rant of all time. Of all time!"

This was a dumb comment, I'd delete it if I could. Sorry.

I wonder why they don't just publish the source repository on the web.

As what, a zip file or something? Good idea. http://referencesource-beta.microsoft.com/download.html

git repository would be better.

Baby steps.

My hunch is that they use some internal server and don't want to expose it to the world. They could copy it somewhere, but if you're not going to take contributions then why bother?

Also, their code explorer is far and away better than trying to spelunk through a Git repository looking for that one file you want to read. So I think it's actually an improvement for when you just want to look at a few files rather than download everything.

>My hunch is that they use some internal server and don't want to expose it to the world. They could copy it somewhere, but if you're not going to take contributions then why bother?

Ah. They should want to expose it to the world and take contributions.

We also have a ZIP download.