Another thing that I'm really surprised about is the salary range for the C++ developers with RTOS experience. In case we compare this with frontend React developers, then difference will be more than 50% (React guys will be paid more).
According to my understanding C++ development for projects that require RTOS is much more demanding in terms of skill and corresponding experience. I would also assume that there are much less developers with VxWorks experience: within my region I may find around 40 people with VxWorks in their CV on the job board against 1000 developers with React experience.
I really hardly get the economics of the modern IT job market.
1. There's a lot more demand for simple web apps/interfaces in areas where there are potentially absolutely insane gains to be had from very simple automation.^1
2. Most devs I know at least consider writing simple web apps/interfaces as the most boring work you could possibly do.
^1 (I've seen 1000+ employee companies with workflows that include multiple people simply sorting and searching through data in hundreds or thousands of files manually, something that any PHP or Javascript script would be able to do in seconds. One time they simply didn't have an interface between two apps and there was someone who's entire job was essentially to type the data from one app into the other. A motivated high schooler would have been able to write something that would have saved 4000-6000 hours of work per year. These kinds of insanely low effort automation opportunities simply don't tend to exist in embedded areas, and it will probably take at least 10 years before most of them are gone from small to medium sized businesses.)
The market for web developers is heavily distorted by companies that effectively own winner-takes-all markets and by a wide range of investor money bonfires chasing similar dreams (not just the unicorns of Softbank and Vision Fund but also a very long tail of much smaller investors and investments).
This isn't quite as prevalent in hardware applications where even well funded startups have to be concerned about actually achieving positive margins and cannot just hope for growth to eventually outpace whatever cost structure they have burdened themselves with.
I was an architect for a custom multi-platform RTOS built for many core NUMA systems (C/C++, Ada, Java). Most challenging work I ever did because you can't just look up the answer on Stack Overflow. Took a job in IT for 40% more. Now I'm bored to tears and hate my dumb coworkers who think web development is so hard.
I live and work in Cambridge UK, and it seems to be the other way around here. Cambridge's history is more of hardware based startups (particularly wireless related ones), so the demand mix is probably quite different. I have always done very well with my electronics/software mixed skill set, and it was a surprise to hear (in a previous HN discussion) that that wasn't as valuable elsewhere.
It’s not a simple embedded vs. front end divide. I think Cambridge has generally funded and valued companies that are see as solving, ‘hard problems,’ and, rightly or wrongly that viewpoint favours backend stuff and hardware.
I’m not sure this is a problem though. Many large companies are not hiring in the valley because the high level of compensation is making it uneconomical.
In my experience, front-end projects always seem to get more attention and visibility from product teams, and the business in general. This seems to happen even when it’s a nascent product that doesn’t make any money yet, and older, more boring systems responsible for large percentages of company revenue often get ignored. I’m not sure if this is related to pay in any way, but it definitely seems like the visibility of front-end apps, and being able to showcase them to wider audiences, doesn’t hurt.
I see it like this: In large engineering companies doing embedded systems, software developers are seen as fabricators who produce an item according to a specification, like welders or painters. A web developer in a tech company has a much higher status in the company.
Yeah I've been thinking about moving to lower level software development or even embedded and a ton of the more interesting sounding jobs seem to be outside of the US
I'm a former embedded software guy and I can explain a bit more. Hardware is extremely capital intensive, takes a long time for any return, and the margins tend to be pretty mediocre. Around the mid-2000s many embedded developers wound up getting siphoned off into mobile development (back to around Symbian times) which was far higher up the stack than doing anything with micro controllers, RTOS, etc. The embedded software community is mostly a bunch of electrical engineers that learned software second and this can be a small handicap against pure software people that learned hardware far later. Absolutely there are still great engineers doing embedded work in the US, but it's a shrinking market in a lot of ways and very regionally concentrated (not just _any_ metro area will do for stable employment - places in Texas or outside Boston were dominant).
I left embedded dev because I saw this coming and I'm just a slow developer that wouldn't have cut it with the demands of mobile software emphasizing cranking out stuff faster when that's just not how I work.
And not only that. Even if there is money on the table or lives at stake, many companies choose to outsource embedded development instead of increasing salaries to attract top talent. Just recently Boeing software practices were heavily criticized concerning the 737 Max fiasco [0]. Sadly it has become a tendency not only in the avionics industry.
I was young and am a mere hardware and server nerd at heart so I wound up down the cloud / DevOps / SRE or whatnot path unfortunately a little too early and it's been quite fun.
Embedded has been consolidated into a bunch of huge multinationals like Texas Instruments, Qualcomm, etc. and they're doing alright financially. It's just a very innovation-lacking industry compared to software, so you have a chicken-egg problem with investors. It'd be like trying to start a CPU company now when we've got Intel, AMD, IBM, and maybe one or two more boutique makers teetering on the edge of bankruptcy constantly.
I agree with you, as ultimately an employer will have to raise wages to get/keep employees.
But it's not really enough to explain it because the engineers who do Vx could always just switch to a better paying platform.
I wonder if the low salary indicates that the Vx Devs at that low price aren't able to switch out. Either due to location, lack of ability, or some other capture.
It's not always trivial to change technologies job-wise. You may be perfectly confident in a technology you haven't used before professionally, but people doing hiring may think different. I have been screened out for "only" having hobby/extracurricular experience with tech I'm fully qualified to use.
As anecdotal evidence I know many developers (including myself) who refuse to touch frontend. It still stands, supply and demand rules apply. Does not matter why the supply is low or the demand is high.
I am an embedded developer using primarily C++, but I've done a ton with Javascript. I don't know why an embedded developer would refuse to touch frontend, other than a suspicion it would fit the stereotypes of database applications (CRUD, tables). The embedded UI's I've worked with the most were not in HTML but we have done demos using WebKit and Angular or GWT. Given the asynchronous constructs it takes to keep our product running properly, the asynch in Angular was not an issue. We also had an in-house test tool that we ran for over a decade that used Javascript for the tests themselves. This was not asynchronous but we ended up with several engineers who knew Javascript's scope peculiarities and unit test requirements because of it.
Yeah, frontend dev is it's own beast. I've seen competent C++ devs at a complete loss when facing JavaScript. Especially if they've never done any asynchronous programming before.
I am not entirely sure what qualifies for being "competent C++ devs". My thinking goes that it is kinda hard to imagine competent developer in general (never mind C++) not having knowledge about async patterns.
I have done async in Clojure, Erlang (Elixir) and Rust but never touched Javascript (wat?). Not sure you can attribute async to frontend development. Erlang predates Javascript by almost 10 years.
Unlike Erlang/Go/certain Clojure libs, the JavaScript flavor of async is primarily around red-blue functions. A lot of stuff is explicit and not handled by the runtime. A single async keyword can easily taint the entire codebase.
More importantly, frontend is extremely demanding in terms of UI. While for e.g. QT, WxWidgets, native GUI toolkits, it is perfectly acceptable for things to look the same. On the modern web, every significant app is expected to have a "design language" and not look like each other. On the other hand, HTML by default isn't very stateful, sure it has some stateful components like checkboxes but in general you have to store the state in code. And every code base needs a custom UI. You end up with a state space explosion. That is also why frontend frameworks these days are trying to force things into finite state machines i.e. Redux, Elm architecture. Whatever term you prefer. Half of your UI codebase often have absolutely nothing to do with business logic and instead animations and user experience code to make things feel "slick". A ASP.NET WebForms just ain't going to cut it if you wanna build a SaaS. Sad because tons of money have been wasted on essentially marginal improvements on user experience, but good if you are the consumer of course.
> Not sure you can attribute async to frontend development.
Not trying to do that. But (unlike C++), you have to understand async to use JavaScript sensibly. If for whatever reason you haven't come across it another language, then you might struggle (esp. if you're coming from C or C++ where the skills required are very different).
Aside from the already mentioned supply/demand aspect, people working on things like that tend to do so for an entire career, so might be willing to accept a lower salary in exchange for stability.
The React developer will have to retrain and rebrand themselves as Bojombo developers in 2 weeks and as Klazoum Framework ninjas in 2 years.
It's a nice idea, but it doesn't match my experience.
I find that almost every new embedded project comes with yet another vendor architecture, vendor-specific toolchain, documentation that's wrong, totally different peripherals and drivers, weird limitations, undocumented pipeline glitches, and so on. Even the CPU architectures and instruction sets are often different, if it's a DSP, or an SoC with DSP coprocessors glued on.
Basically, new things have to be learned often in that domain, at least with the jobs that come my way. For a while I got a bit anxious on new projects because I couldn't understand how to make seemingly broken tools do things the client wanted. Things like getting signal data into a DSP simulator when all the (GUI only!) buttons didn't do anything at all for example; but eventually I learned that it takes about 2 weeks on every new project of restraining the urge to swear, and than all the undocumented stuff, tools working differently than documentation says, secret header files and options, libraries that no client of the vendor has ever used and are full of empty function bodies or just broken, but sold to my client as if they are actually in mainstream use, weird memory models etc. starts to make sense and I'll be productive. So I just expect that on any new architecture project now, and it works out ok. Just budget 2 weeks of swearing-restraint at the beginning.
On front end webdev, there is infamously a huge amount of churn too. E.g. every time a new Webpack comes out I have to spend quite a while figuring out how to rewrite the config files for the new version for best results. WASM differs from asm.js; grid and flexbox are a new way to achieve what we used to do with floats, and there's an interesting new browser API each week.
But React hasn't change much in half a decade, and continues to be one of the main front-end frameworks. You could have learned React 4 years ago, and keep on using the same skills today and continue to be well paid if you're good at it. That pay is, indeed, at least 50% more than the pay offerred for low-level embedded, even though React work is way, way easier.
But remember, you can negotiate... Just taking a software dev position as an individual is not the route to the best pay in embedded hw/sw projects. If you can instead sell product development, where the client wants something built for them, and you build it, complete with project management, architecture, technology selection, supply chain, and subcontracting or setting up a project-specific company, then the scarcity of relevant skills works in your favour and the pricing is quite different - better. Those projects are great if you can get them, and sometimes a lot of fun.
This surprised me too. But I think it really boils down to there being both more demand and less available supply of React developers.
Embedded C/C++ development has been around much longer and is far more conservative in terms of adopting new technology. So they have a much larger labor pool of experienced developers who are going to be familiar with the core technologies of the job. Web shops migrated quickly to technologies like Angular and React, whereas the vast majority of C++ shops likely won't be adopting C++17 any time soon unless it's a smaller team's greenfield endeavor. I'd also argue there's less training necessary to go from, say, C++03 to C++17 than going from ES5 jQuery to ES6+ (or Typescript) React.
I agree it's bizarre. I think it's as simple as there being a lot more demand for React developers. There are a lot of companies that want a web or mobile app. Comparatively few have embedded projects.
I'd guess that the economic value on the table in industrial controls is just not that high. It's a mature, stable, slow-moving market. The growth parts have to do with "IoT" and "Cloud," not so much the embedded side.
Gaming has similar-ish needs but is famously oversupplied.
Yes, embedded is harder than web. But embedded can only scale to as many hardware units the company can sell, while web can in theory scale to everyone on the planet. Web pays more because of this.
Details are a bit sparse, they likely support Rust only in RTP[0], not in Kernel. If someone has a more technical resource, I am interested.
This is an important milestone, because this is a significant actor in the embedded world, where suppliers are often rather conservative regarding language/toolchains they provide, that is putting some of its weight behind Rust.
I am one of the OS developers for a military vehicle that utilizes vxWorks and Linux. It's honestly no different than other OS work out there. Documentation and help outside of WindRiver's official docs are more scarce, and open source support is smaller, but grasping how an RTOS works vs a GPOS isn't any major mental leap. If you can program in C and understand communication protocols like i2c, UART, SPI, CAN, etc. then it's not terrible.
Right now, we are looking at moving away from vxWorks and onto a Yocto-built Linux with the RT patches to the scheduler. From a requirements standpoint, the RT patches meet our needs for "real time" since our timings require more of a soft real time than a hard one. There's also way more support out there for Linux, and I am also slightly biased in that I am more of a Linux guy than a vxWorks guy simply due to familiarity and ease of troubleshooting.
This latest news with vxWorks is great, however. I actually sent this to management because we are attempting to figure out what direction to take as our current version is reaching EOL.
>Right now, we are looking at moving away from vxWorks and onto a Yocto-built Linux
This decision is probably out of your hands, but I'm happy to share a few words of caution. I've been the yocto distro maintainer at a large firm for about three years and I absolutely despise it. BMW released a slide deck about their experience with yocto, and it was mostly "it sucks." I read that recently and it matched my experience down to a t.
If you absolutely need third party commercial support, yocto is probably the only game in town, but if you don't, I would highly recommend looking at nix (which recently gained cross compile support) or buildroot.
I'm dead serious about avoiding yocto if at all possible, I've seen things I wish I could unsee. But that sounds about right for a military project.
EDIT: I put my email into my profile if you want to chat about yocto more in depth.
Embedded systems here, several projects using VxWorks through the years. Mostly safety motion controllers, robotics, medical and consumer FPGA processing boards.
Due to cost and availability of modern tools (compilers, testing, V&V, CI/CD and virtualization), there is a big trend right now to migrate those projects into RT Linux, which has come of age. I guess the support for modern languages from Wind River is their attempt to avoid losing customers or attracting new ones. They used to have an edge due to the certificated environments, but modern development and systems are becoming so complex (eg networking, cloud, etc) that using a pre-certified kernel is just a tiny bit of the certification process, and in many cases the complete system has to be validated anyway.
I used to when I worked at C-Cube/LSI Logic. We had a multimedia processor (MPEG-2/H.264 encoder and decoder) that was SPARC based. It was used in DVD recorders, D-VHS decks (if anyone remembers those), set-top boxes, satellite downlink receivers for cable TV headends and other MPEG applications.
My favorite part of VxWorks was WindView. It's an awesome tool that shows task execution.
Tornado. That brings me memories. ;-) IMHO it was much better than the current Eclipse-based IDE they have now. At the time, and especially with custom SoCs or CPUs like the SPARC you were using, there were very few options to do that. These days we have tools like Percepio's Tracealyzer [0] that targets a multitude of microcontrollers and RTOSes, and allows you to decouple a bit your development for a specific architecture. Many vendors also have similar tools for their offerings, for example the RTX viewer for ARM tools.
You at ALT or are there other players in that? In discussions about regulation, I often tell people one of the biggest ways of showing its value is that DO-178B caused a company to attempt a high-quality driver for Radeon that they wouldn't build themselves. ;)
I'm very curious about your experience in that market. Especially whether the GPU vendors give you enough information under NDA to make them reliable, if you mostly black-box it using a subset of their features, and so on. Also, the QA techniques you use in such scenarios. I think some of tricks on your end might be applicable to other types of hardware.
I'm at CoreAVI. As far as I know, ALT had some financial troubles that ruined the company (this was all before my time, I'm too young to have been involved in that :P ). Like half the senior employees are ex ALT employees who jumped ship once ALT started hemorrhaging money. We have a great relationship with AMD, we don't just do certified drivers for their chips, but we also temperture screen some of their GPUs and sell them in the industrial market. They give us a lot of documentation and will answer our questions directly. Although nowadays with all the stuff they release to the open source community I'm not sure if we are getting that much more information compared to everyone else. We do work with other GPUs such as the Verisilicon GC series, Intel HD, and most recently we started working with ARM. For Verisilicon & Intel, we don't really get as much support and we mostly work off the documentation they have released and reverse engineering on our part. The ARM partnership is fairly new so I can't speak too much about it, but from the outset they are looking to be a great partner like AMD.
For graphics APIs we implement a subset of OpenGL or Vulkan listed under their safety critical standards. We are part of Khronos and worked with them to define the OpenGL SC 2 standard and are currently working on defining the Vulkan SC standard. The saftery critical version of the APIs are a reduce subset that eliminate anything that would be more difficult to implement in certifiable code.
As for our QA process, its really split into two distinct things. There is just normal QA side with all the run of the mill testing and verification, and then there is the actual certification team who is responsible for meeting all the DO-178 requirements as defined by our process. Like a lot of other companies who do cert work, we don't create all the artifacts from the get go. There are a lot of defense contracts where they want "certifiable" but don't want to pay for the actual evidence, so we won't go out of the way to make it, if no one is buying it. We do have a strict coding standard and code reviews to ensure that any code checked in will actual be able to be certified, but certification is way more work than just writing the code. Once we have a paying customer for cert we will go through and write all the test cases, fill out all the requirements, and make sure every i is dotted and t are crossed.
I believe both Schneider and Allen Bradley use vxworks, but unless you are developing the PLCs you would never know, as a scada system supplier vxworks is not exposed to you just the features Rockwell or Schneider have built on top
