×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Stress-Testing Software For Deep Space

samzenpus posted about a year and a half ago | from the phone-the-help-desk dept.

Mars 87

kenekaplan writes "NASA has used VxWorks for several deep space missions, including Sojourner, Spirit, Opportunity and the Mars Reconnaissance Orbiter. When the space agency's Jet Propulsion Laboratory (JPL) needs to run stress tests or simulations for upgrades and fixes to the OS, Wind River's Mike Deliman gets the call. In a recent interview, Deliman, a senior member of the technical staff at Wind River, which is owned by Intel, gave a peek at the legacy technology under Curiosity's hood and recalled the emergency call he got when an earlier Mars mission hit a software snag after liftoff."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

87 comments

Gay (-1)

Anonymous Coward | about a year and a half ago | (#41615209)

gay babies

Seems like a rationalization (4, Interesting)

gadzook33 (740455) | about a year and a half ago | (#41615239)

While I buy that the landing systems need an RTOS, I doubt Curiosity does. Image processing that happens with "precision"? Do x86 processors not process images precisely enough? I get the idea of being hardened to radiation but it was my understanding we have newer processors that fit the bill on this. The rest of this seems like a rationalization for using old hardware. However, as an engineer for the government it's possible I'm just old and embittered.

Re:Seems like a rationalization (2, Insightful)

Anonymous Coward | about a year and a half ago | (#41615311)

Remember, too, that Curiosity has been in the works for almost a decade. They had to commit to a spec for the computers a long time ago, so it's no wonder by today's standards things seem out of date.

Re:Seems like a rationalization (5, Informative)

Anonymous Coward | about a year and a half ago | (#41615385)

that's why land-based projects like SKA for example which also take decades to complete are designed taking moore's law into account, leading to a very funny situation in which the project starts, they start building stuff but the computers that will run the thing are still 10 years away... (and I guess everybody just hopes computers will keep up or else...)

Also you must take into account that the actual instruments are being built fairly early i.e. 5 or more years before launch since there is a LOT of testing calibration more testing etc. Additionally, when the stake is a billion dollar project like these you tend to leave fancy new things and favor old proven and well documented tech. Just in case...
If not you just mount two instruments if you have space and money a fancy new one and the old usual thing (such is the case for Solar Orbiter for example)

Re:Seems like a rationalization (2)

Electricity Likes Me (1098643) | about a year and a half ago | (#41616333)

The classic example is the Pentium math error: imagine if you were 2 years into the mission and then discovered that the new high speed chip you put in gives incorrect floating point calculations.

Re:Seems like a rationalization (1)

kauaidiver (779239) | about a year and a half ago | (#41616907)

Exactly, and imagine if we sent people there. It's amazing we sent people to the moon so many years ago!

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41617211)

Yes, it's amazing what can happen when a country decides that a stunt is very important and dedicates vast resources to it. Mars is a hundred times further away.

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41617339)

In the end, if you look at the numbers its not that expensive. A big space project like curiosity or ESA's JUICE or PLANK are in the order of a billion spent over 20 yrs.
Medium and small spacecrafts are from 50 to a couple of hundred millions plus launch costs, also over a period of 10-20 years.
Just compare it with other numbers you saw lately during the elections for example... (a small satellite is one fourth of romneys fortune, the banks bailout in 08 is 700 curiosity rovers)

In terms of scientific results, which I can speak of, since I am in research and not in the industry, it makes all the difference for the scientific output of a lab.
To put it clearly, those little things we send out there are not just for fun and cute pictures to hang on our nerdy offices...

Re:Seems like a rationalization (1)

DickBreath (207180) | about a year and a half ago | (#41619639)

That "stunt", which maybe it was, resulted in driving a lot of advances in microelectronics that led directly to the cool toys we have today. We would have gotten here without the space race, but it would have taken longer.

It's funny how once money is put into developing something 'impractical', that other uses are found for it that lead to it being useful for everyone. Examples are many, but I'll just mention: GPS, Communication and Weather Satellites, The Internet and Xtube.

Re:Seems like a rationalization (1)

0123456 (636235) | about a year and a half ago | (#41620243)

That "stunt", which maybe it was, resulted in driving a lot of advances in microelectronics that led directly to the cool toys we have today.

No, it didn't. ICs existed before Apollo, and the primary benefit was ramping up production and pushing for improved reliability.

We would have gotten here without the space race, but it would have taken longer.

Indeed. We might still be using Core 2s. Not a big deal in the grand scheme of technology when most i5s and i3s spend most of their time idle.

Examples are many, but I'll just mention: GPS, Communication and Weather Satellites, The Internet and Xtube.

None of which have anything to do with Apollo. It was a great achievement with the technology of its time, but the 'spinoff' arguments are just bogus.

Re:Seems like a rationalization (5, Insightful)

Sasayaki (1096761) | about a year and a half ago | (#41615355)

My understanding is that the thinking goes like this.

Sure, there are newer processors that claim to fit the bill. But space hasn't changed so much since the Apollo days that we need all new processors; by and large anything that needs "heavy lifting" CPU wise can be transmitted back to Earth. For unmanned probes, there's very little demand for high speed CPU tasks that can't be offloaded to Earth. And even if there was, when your latency back to your operator is about 14 minutes (with an extra 14 to receive further instructions, plus the time it takes to interpret the previous data set, determine new instructions, then program those instructions), that's a lot of down time to work on various tasks.

The Mars rover CPUs, I imagine, spend the vast majority of their time idling.

However... the old stuff works. It has its faults and flaws, sure, but they're extremely well known and documented. You can work around them. You have the old grognards that have been kicking around since Apollo who know every damn thing about them. They're risky, sure, but it's a managed, controlled, limited and understood risk. But new processors are *new*. You lose that element of certainty, and the CPU is the heart of a probe. You lose it, you're fucked.

You're trusting the mission, a mission that costs billions of bucks, to a new, untested device that hasn't been field tested, hasn't got that certainty, and *you just don't need*.

Re:Seems like a rationalization (2)

Grave (8234) | about a year and a half ago | (#41615465)

That's just it - this sort of computing task cannot get by with 5 9's or 7 9's or a hundred 9's of reliability. It needs to be 100% reliable, which means that every potential hiccup, flaw, or design quirk is understood and documented to the nth degree, and thus can be worked around. It also means you can reliably simulate the hardware and throw all sorts of stress testing at it.

Re:Seems like a rationalization (3, Insightful)

khallow (566160) | about a year and a half ago | (#41616173)

It's worth noting that in the overall mission they generally get by with one or two 9's of reliability. There's no 100% reliability out there and nobody would be able to afford it, if there was.

Re:Seems like a rationalization (2, Interesting)

Anonymous Coward | about a year and a half ago | (#41615489)

Reminds me of about 10 years ago when I was working at Motorola on cell phone base stations. We switched from VxWorks to Linux and got... nothing. No performance gains, no reliability gains. Just a free OS instead of something we had to spend money licensing.

Of course, all the extra time spent switching and testing certainly cost a lot of money in man hours.

Re:Seems like a rationalization (1)

gadzook33 (740455) | about a year and a half ago | (#41615563)

Yeah, I agree except that wasn't really how his argument goes...and yes, old stuff works. But new stuff works too (also, new here could be 5 years old). Anyway, I'm not really (or at least overly) questioning their rationale. I've just seen too many programs where the same people have been there forever and it's easier to keep doing the same thing rather than try something new. Again, hopefully that's not the case at NASA but it's sure as hell the case at the Pentagon.

Re:Seems like a rationalization (1)

Electricity Likes Me (1098643) | about a year and a half ago | (#41616339)

Yeah, I agree except that wasn't really how his argument goes...and yes, old stuff works. But new stuff works too (also, new here could be 5 years old). Anyway, I'm not really (or at least overly) questioning their rationale. I've just seen too many programs where the same people have been there forever and it's easier to keep doing the same thing rather than try something new. Again, hopefully that's not the case at NASA but it's sure as hell the case at the Pentagon.

On the other hand, NASA really doesn't have the budget to spend working up for something new either. A processor switch means new simulators, new architectures etc. I imagine for a space probe - i.e. something you can't get at ever if it breaks down - then you go with the processor you have when you start designing it, and you pick the most reliable thing you can.

Re:Seems like a rationalization (1)

sjames (1099) | about a year and a half ago | (#41623895)

How many hours in space had the new stuff logged when the design of Curiosity was completed?

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41616409)

Thats it. 640KB is enough.

Re:Seems like a rationalization (3, Informative)

lordholm (649770) | about a year and a half ago | (#41616479)

Newer missions collect too much data to transmit everything back to earth. They typically need to do local processing of for example images and other data. There is also AI aspects, for the ExoMars rover (made by Europe), the onboard computer will have a virtual scientist embedded. This virtual scientist look at the camera pictures and decide if something is worth an extra look, and may order the rover to carry out opportunistic science. I am not sure as to whether this is the case with Curiosity, by I could easily imagine this is the case. In fact, newer missions have substantial need for computational power. But, there is no software reason to do these computational tasks on the main computer, the task may as well be sent to a soft realtime helper computer, that may as well run Linux or something else. A lost image is typically not the end of the world.

In many cases the spacecraft and rovers are also not hard realtime, but they are also not soft realtime either (i.e. we compute thruster response for t=0, only to have the thrusters fired at t+0.1 or something in that range, whether they fire within this time does not really matter except during docking, landing and separation), I was trying to push through the notion of firm realtime when I was working in the space sector, but the main problem with this notion is that we do not yet know what effects it has in terms of sw design. Any way...

The primary reasons for running 10 year old CPUs is that, 1) specs are chosen early in the project, this is important as the CPU specs are guiding the development of the SW requirements and the actual implementation of the SW and 2) as you say, the older CPU will be battle tested before they are sent into deep space.

Re:Seems like a rationalization (0, Troll)

Required Snark (1702878) | about a year and a half ago | (#41615451)

Yep, you're obviously correct. Everyone at NASA is stupid, and just by looking at a summery on Slashdot you have reached a conclusion that escaped them. Windows or plain old Linux would work just fine.

If your comment is any indication of you native intelligence I don't know how you manage to put your clothes on by yourself. It's surprising that you haven't wandered into the street and been killed by a car. (That's just my wishful thinking, by the way.)

Any autonomous vehicle, is by definition, a real time system. It's working in a physical environment that requires hard real time response. If the control action is not delivered in a specified interval, it is useless. The result of missing a hard deadline is crashing. Not such a good idea on Mars. The speed of light delay time is 14 minutes one way, and it's going to get longer since the Earth and Mars are now moving apart.

All indications imply that you are as stupid as you look. Your SIG implies that you are a knee jerk right wing asshole, who assumes that all government activity is useless. I worked a JPL years ago, and everyone one I met there was bright, creative and dedicated. There were no slackers. I doubt you would last in that environment for two pay periods. You're not smart enough.

Re:Seems like a rationalization (2)

gadzook33 (740455) | about a year and a half ago | (#41615543)

Oh good, someone more embittered than me. I especially like how rather than provide any sort of argument as to why an RTOS is required (because...a pedestrian might walk in front of the rover?) you'd rather insult me.

Not for nothing but I'm about a step away from being a hippie and I've served the government faithfully for many years. I work with some of the best and brightest and if you weren't able to cut it there, the fact that you're incredibly negative and seem like a jerk would likely only be a few of the reasons why.

Re:Seems like a rationalization (4, Informative)

Meditato (1613545) | about a year and a half ago | (#41616209)

Look, that guy ("Required Snark") might have been an asshole, but you didn't really acquit yourself well either in your original post. I cofounded and work for a real-time telemetry contractor. We use Android, but the Linux kernel isn't built to handle read-time applications reliably. There are too many things to handle in terms of time-safe task-switching, execution, multi-processing, and internal consistency in order for it to be a good RTOS. So keeping that in mind, I had to implement a real time environment in userspace that uses root and some native code in order to collect data, send data, and operate hardware in a safe, timely manner. But this isn't the best solution because I still have to deal with the fact that it's all just a frustrating abstraction sitting on top of a kernel that isn't at all concerned with what I'm actually trying to do, despite my best efforts to single-handedly make the necessary changes.

Your "newer processors" bit is also completely off the mark. Radiation-hardened processors lag generations behind owing to the need for extensive redesign and testing. Complicating this picture is the fact that even then, they still have varying levels of reliability and power efficiency. You don't want a processor that has a microcode architecture that makes your targeted code difficult to semantically evaluate and verify. You don't want (or need) a recent processor that hasn't had extensive real-world user testing. You want a processor in the goldilocks zone, one that you've worked with before and has a community behind it.

Keeping that all in mind, they chose a good processor, and already had an OS largely built for it based on previous missions with earlier versions of the same processor.

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41616445)

Pedestrian crossing?

Do you even know what a real time constraint is used for?

Re:Seems like a rationalization (1)

AmiMoJo (196126) | about a year and a half ago | (#41617809)

TFA explains it: "VxWorks has to react immediately in order to survive while exploring Marsâ(TM) surface."

It isn't hard to imagine why this would be the case. If a sensor suddenly reports a fault you might want to react extremely quickly to prevent the rover being damaged. Say a wheel jams or something like that. Since the rover can't be repaired a great deal of caution is necessary.

Re:Seems like a rationalization (-1)

Anonymous Coward | about a year and a half ago | (#41617475)

Nasa Launched a new deep space probe running Windows 8 on a HP desktop using the i7 processor. when it locked up 6 minutes after launch due to windows installing updates automatically, it crashed killing 30,000 people.

NASA said, "It's worth it, the Metro Interface is so cool"

Re:Seems like a rationalization (3, Informative)

Anonymous Coward | about a year and a half ago | (#41615601)

And that's where you (and most people) are mistaken.

A RTOS is not an OS that acts "quickly", it's an OS which provide a 100% guarantee that a task will be executed in a definite time-frame, whether this needs to be 1 micro-second or 1 hour ; and which provide guarantees if the task can not be completed in this time-frame. A job neither Windows nor any flavor of Linux can achieve.

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41615787)

The cpu they used cost $50,000 plus another $350,000 for the board. This is radiation hardened high grade stuff here. You put that kind of hardware in things that require hardcore components, things like spaceships and interplanetary probes. The cpu was also spec'd out a decade ago.

Any old x86 cpu would not fit that bill, not by a long shot.

Re:Seems like a rationalization (2)

TubeSteak (669689) | about a year and a half ago | (#41616117)

First of all, there's a typo in TFA.
They state the chip is a "RAD760" but they link to the RAD750 wikipedia page.

Do x86 processors not process images precisely enough? I get the idea of being hardened to radiation but it was my understanding we have newer processors that fit the bill on this.

The problem with x86 technology is that it has gotten too advanced.
The chips have become so dense that radiation hardening is much much more difficult than it used to be.
Increased difficulty = increased expense

Further, I don't think you appreciate the specs of that old PowerPC chip.
It's tolerance to 1 megarad of radiation exposure is a lot.
You literally get what you pay for with this cup, ranging from 200 rads to 1 megarad.
Even 500 rads is more than most space applications require.

So in order to save money, some companies use cheaper hardware in a triple redundant configuration, in order to avoid paying out big bucks for radiation hardened boards + chips. But for a mission to mars, where reliability and power usage are critical, two old 133mhz processors are better than any of the other choices.

The rover has just enough processing power to talk to NASA, look around, and do one other thing. And that's just fine.
They've partly split up the workload between two processors, but if one processor failed, NASA could juggle everything with one hand.

Ever seen a time table for a space mission? (2)

dutchwhizzman (817898) | about a year and a half ago | (#41616243)

They start planning this years, years and years ahead. It is not uncustomary to have decided on a hardware platform five years before launch. Since there's a lot at stake for these bigger missions to succeed, they usually don't take risks and put stuff up there that hasn't proven itself. Maybe some evolution like a higher clock rate or more memory or something like that, but a new processor architecture gets tried on other things that have redundancy, lower cost or less exposure and preferably a combination of those.

I have been discussing some technology that was possibly put in an instrument on a weather/climate sat with the primary investigator of the then current mission and named to be the one of the next mission as well. This was around 2007. They had to choose the technology then, so they could work on plans and get funding around now. Once they get their funding, it will still be three to five years before it goes up there. Back then, due to the reliability demands they had for the sensor and the relative unproven state of using CMOS sensors for photon capture (common used in digital consumer cameras in 2007) they chose to go with the previous solution, that was in the current instrument. That means that they will probably launch a pre-CMOS sensor equipped instrument around 2015, because that was the best option available to them when it was decision time.

Unless we change the way we "go to space" in a radical way, I don't see the latest and greatest tech make it in missions like this. It's up there, sure it is, but only a handful people know it is and they don't want their precious black ops budget exposed or taken away from them. Once the statistics they get from the successes and failures (failing in secret "testing missions" once in a while is allowed) to a rating that makes it commercially viable to sell the tech to civilian usage, plus the state of technology used for espionage and military use is such that there isn't any tactical threat to do so, more modern tech will be used for missions like this.

Re:Seems like a rationalization (5, Informative)

Animats (122034) | about a year and a half ago | (#41616503)

I get the idea of being hardened to radiation but it was my understanding we have newer processors that fit the bill on this.

Radiation-hardened processors are hard to get. For one thing, they're export-controlled, so if you make them in the US, you can't sell many. Atmel makes a rad-hard SPARC CPU, and they've sold 3000 of them. Nobody seems to have built a modern x86 design or even an ARM in a rad-hard technology.

There's a basic conflict between small gate size and radiation hardness. The smaller the transistors, the more likely a stray particle can damage or switch them. So the latest small geometries aren't as suitable. Also, the more radiation-hard processes, like Silicon on Sapphire, aren't used much for high-volume products.

As a result, rad-hard parts are an expensive niche product. It's not inherently expensive to make them, but the volume is so small that the cost per part is high.

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41619279)

I know this is a crazy observation, but why get parts that are individually hardened against radiation, and just put them in a DU or lead box?

Re:Seems like a rationalization (2)

chihowa (366380) | about a year and a half ago | (#41620213)

Shielding is heavy and expensive to launch (and to land softly). Then, for every extra mm of lead shielding you add, there's a more energetic photon just waiting to flip a bit. It ends being up cheaper to make radiation hardened electronics than to accommodate for the extra shielding.

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41626977)

NEC V40, a 80188 clone with 8080 emulation, had a radiation-hardened version. http://www.amsat.org/amsat-new/satellites/satInfo.php?satID=48

Re:Seems like a rationalization (1)

hackertourist (2202674) | about a year and a half ago | (#41617097)

The RTOS may not be needed for image processing, but I'll bet it's handy when driving, or running other mechanical aspects.
And once you have an RTOS for those tasks, it'd be silly to add another OS for the non-time-critical tasks.

Re:Seems like a rationalization (1)

jittles (1613415) | about a year and a half ago | (#41617363)

The government LOVES old hardware. Trust me. The AH-64D uses 486 processors. You know what? They aren't the only ones, either. I used to work for a company that designed and manufactured analog and digital video surveillance systems. They are still using 486's in some of their hardware as well (key components that require an insane MTBF to comply with regulations for casinos, military installations, etc). Why? Because it runs nice and cool compared to modern processors, and it is a tried and true processor. Can you imagine launching a robot to Mars with a Pentium chip in it, only to find that Intel still hasn't gotten their floating point right in that old chipset? I'm not saying it's likely that float problems still exist in Pentium hardware, but for the cost you go with what you know works. In 10 or 20 years from now, when Ivy Bridge is the tried and true processor, you can bet that the government and many corporations will be using them in satellites and other mission critical hardware.

Re:Seems like a rationalization (1)

toolie (22684) | about a year and a half ago | (#41623593)

The AH-64D uses 486 processors.

You didn't even get the architecture right, much less the processor.

Re:Seems like a rationalization (1)

jittles (1613415) | about a year and a half ago | (#41624061)

Depends on which aircraft system you're talking about. I can promise you that they have at least one 486 on board. I've dealt with the aircraft for years.

Re:Seems like a rationalization (1)

Lumpy (12016) | about a year and a half ago | (#41617439)

Then as an engineer you understand why it's super stupid to have it all run off of 1 processor.

Guidance and maneuvering is 1 processor/system. Science package another, imaging another, etc... When you cant get to it to press the reset button, you dont do the dumb mistakes done on consumer hardware like automotive industry does.

Example: GM and having 90% of the car run on the BCM, and Honda running the WHOLE car including engine off of the single ECM. My AC quit working because of a faulty sensor shorting out the IO port on the ECM. only fix is to replace the WHOLE ECM for the car at $2200.00

That kind of design is only done by really really dumb engineers.

Re:Seems like a rationalization (1)

gadzook33 (740455) | about a year and a half ago | (#41617459)

Yeah, I agree. In fact, I think that was the point I was trying to make (albeit unsuccessfully as it turns out).

Re:Seems like a rationalization (1)

DickBreath (207180) | about a year and a half ago | (#41619733)

> That kind of design is only done by really really dumb engineers.

Or maybe by mid level managers?

Hey, if this one component goes bad, (A) they MUST fix it because it controls so much, and (B) we make a boatload of money replacing it. Therefore, it's a great idea. Engineer promoted. Everyone happy.

Oh, wait. Not everyone?

Re:Seems like a rationalization (1)

mcgrew (92797) | about a year and a half ago | (#41619999)

Example: GM and having 90% of the car run on the BCM, and Honda running the WHOLE car including engine off of the single ECM. My AC quit working because of a faulty sensor shorting out the IO port on the ECM. only fix is to replace the WHOLE ECM for the car at $2200.00

That kind of design is only done by really really dumb engineers.

At one point I agreed with that sentiment. Hanlon's razor says don't assume malice when stupidity will explain, but mcgrew's razor says don't assume stupidity when greedy self-interest explains.

I once remarked "if the idiots who designed cars had to actuallt work on them, they'd be designed better." It was pointed out to me that the automaker makes more money for their dealerships in repair when they're expensive to repair.

Tell me, why does a car need a $2200 computer for the heater and AC when a couple of potentiometers and switches will do the same job for five bucks?

Don't assume stupidity, the engineers are doing what they're told: Make it expensive to fix.

Re:Seems like a rationalization (0)

Anonymous Coward | about a year and a half ago | (#41620651)

Actually, there is one pair of flight computers that are on Curiosity. That pair of computers has directed everything since the rocket lifted off the launch pad. They ran the flight to Mars, the ED&L, and now they run all the operations on the surface. Surface ops include coordinating the science packages as well as driving the rover, avoiding obstacles, and communicating with the orbiters and with Earth.

There really aren't newer processors that are qualified to do deep-space work, the environment - radiation wise - calls for computers that can tolerate in excess of a mega-rad of exposure. There aren't very many of those available.

For operating on the surface of Mars you could probably get away with a 500K-rad hard computer, but you'd still have to get it there somehow.

"earlier Mars mission" == MER-A Spirit (2, Interesting)

Anonymous Coward | about a year and a half ago | (#41615249)

''recalled the emergency call he got when an earlier Mars mission hit a software snag after liftoff."
From TFA:

Back when Spirit Rover landed on Mars in 2004, it experienced file systems problems. I got a call on landing day while I was in Southern California. I fired up my laptop and worked with three groups who were dealing with a variety of time zones: California, Japan and Mars. Since I had a RAD 6000 systems on my desk running simulations, by the end of first week we figured it out and were able to fix it.

Re:"earlier Mars mission" == MER-A Spirit (5, Informative)

AaronW (33736) | about a year and a half ago | (#41615503)

With my long experience with VxWorks this doesn't surprise me. VxWorks is not the most robust RTOS. Think of it as a multi-tasking MS-DOS. The version they used has no memory protection between processes and I have found numerous areas of VxWorks to be badly implemented or downright buggy. Up through version 5.3 the malloc() implementation was absolutely horrid and suffered from severe fragmentation and performance problems. On the platform I was working with I replaced the VxWorks implementation with Doug Lea's implementation (which glibc was based off of) and our startup time dropped from an hour to 3 minutes. I was also able to easily add instrumentation so we could quickly find memory leaks or heap corruption in the field, something not possible with Wind River's implementation. After reading about the problems with the filesystem I looked at the Wind River filesystem code. It was rather ugly. They map FAT on top of flash memory (not the best choice) and the corner cases were not well handled (like a full filesystem).

Similarly, their TCP/IP stack sucked as well. If you can drop to the T-shell through a security exploit you totally own the box (i.e. Huawei's poor security record).

VxWorks is fine for simple applications, but for very complex applications it sucks. At least the 5.x series do not clean up after a task if it crashes because it does not keep track of what resources are used by a task. A task is basically just a thread of execution. All memory is a shared global pool. At the time it did have one feature that was useful that was lacking in Linux, priority inheritance mutexes. These are a requirement for proper real-time performance and I believe are now included in Linux.

Re:"earlier Mars mission" == MER-A Spirit (4, Interesting)

Jeremi (14640) | about a year and a half ago | (#41616501)

Up through version 5.3 the malloc() implementation was absolutely horrid and suffered from severe fragmentation and performance problems.

I talked to one of Curiosity's software engineers the day it landed... he mentioned that one of their coding rules was: no malloc() allowed.

Re:"earlier Mars mission" == MER-A Spirit (2)

kauaidiver (779239) | about a year and a half ago | (#41616943)

No malloc()? Interesting, I worked on a project at NG and we had same policy. Everything was on the stack or global. We had the chance to run with Monta Vista embedded Linux but someone higher up decided to go with "tried and true" VxWorks. I agree with a poster above about re-training costs and all that adding up.. but if embedded linux became standard with big companies I don't think it would take too long to make-up the costs of re-training and all the other stuff that goes with it.

Re:"earlier Mars mission" == MER-A Spirit (1)

AaronW (33736) | about a year and a half ago | (#41617337)

That is a good policy if you can do it, but in this case it was impossible. We had to use some 3rd party software which used malloc and realloc extensively. To make matters worse, for a long time we could only get obfuscated code to support the network processor we were using, meaning that it was impossible to make changes to it. We also had to make use of it because of the dynamic nature of the software. In our case it really wasn't feasible to avoid mllox. Replacing Windriver's malloc had some huge advantages. Fragmentation was horrible with the VxWorks malloc to the point where there were many tens of thousands of fragments of memory. VxWorks used a sorted linked list from smallest to largest free block. Due to the extensive dynamic reallocs, this linked list turned into a huge bottleneck.

Replacing the code with Doug Lea's malloc eliminated the fragmentation problem completely. By including the task ID and calling function's program counter in each block allocated it made it trivial to find memory leaks and keep track of how much memory and how many blocks were allocated per task or even by function.

There really was no good reason why VxWorks was chosen since there were no hard real-time requirements. The product was a mess (router and broadband remote access server) since each box had to include a Sun Ultrasparc computer running Solaris (we required big-endian) where most of the software ran. Solaris was an even worse choice. Trying to write streams drivers for Solaris was a nightmare compared to Linux drivers, especially when trying to tie into the TCP/IP stack. Not only that, Solaris was quite slow. Give me Linux any day.

The great thing about writing applications in Linux user space is that you can use tools like Valgrind to catch many of these memory leaks, uninitialized variables, etc.

Re:"earlier Mars mission" == MER-A Spirit (3, Funny)

Anonymous Coward | about a year and a half ago | (#41618827)

Malloc is non-deterministic. The request for a pointer to return contiguous free bytes will need to search a fragmented memory map to complete the request. The duration of the search depends upon the algorithms and the amount of fragmentation relative to the size of the request. It is worse if it must rearrange memory to accomodate the request. Thus, use of malloc() is typically avoided for time-critical code in a real-time operating system.

Re:"earlier Mars mission" == MER-A Spirit (1)

dfries (466073) | about a year and a half ago | (#41625839)

It is worse if it must rearrange memory to accomodate the request.

You were going okay until here. You can't rearrange memory, malloc returns pointers, and there isn't any callback to ask for that pointer back to move it to another location.

Byte compiled languages like Java can rearrange memory but you call new not malloc so I know you weren't talking about them. Garbage collection is a much bigger problem especially if you think about mixing Java and real time operations. C/C++ in realtime means following the best practices, but for Java, get a different Java http://en.wikipedia.org/wiki/Real_time_Java [wikipedia.org].

Re:"earlier Mars mission" == MER-A Spirit (0)

Anonymous Coward | about a year and a half ago | (#41616723)

While I agree with most of the sentiments on the 5.x vxworks version, it has to be a said that vxworks is now at version 6.9 and is a much improved beast with a far better IP stack, support for 'proper' processes, etc. Saying that it comes with the cost of dropping or modifying a lot of API's making upgrades difficulty and to be honest looks so like linux once you've finished with it you wonder why you spent $50000 on a developer seat

Re:"earlier Mars mission" == MER-A Spirit (1)

AmiMoJo (196126) | about a year and a half ago | (#41617603)

You probably shouldn't be using malloc() on an embedded system like that anyway. Statically allocate everything. That way you know exactly how much memory will be consumed at any time and can budget appropriately. It also reduces the chance of having a bug malloc() all your memory or running out of stack space.

VxWorks claims to have memory protection, chances are it is the CPU they are using which lacks an MMU to support it.

Re:"earlier Mars mission" == MER-A Spirit (1)

Anonymous Coward | about a year and a half ago | (#41617791)

If your objhective requires an RTOS, you're probably not going to malloc(). There are edge cases, but we've successfully banished them. We don't use VxWorks, thank god, but we do use a real memory machine instead of a virtual memory machine. Getting young programmers to understand that is challenging, and getting CS grads, of all fucking people, to program for a real memory machine is just fucking impossible. We make them managers instead.

Re:"earlier Mars mission" == MER-A Spirit (1)

datapharmer (1099455) | about a year and a half ago | (#41617991)

policy inheritance can be handled through FUTEX_PI. Issues due to a lock-contention can be handled by the kernel via FUTEX_LOCK_PI.

I'm glad I don't program for NASA (1)

Press2ToContinue (2424598) | about a year and a half ago | (#41615341)

The last thing I would want to do is program mission-critical systems. That G*d my programming mistakes are hidden in the mire of a thousand other programmer's mistakes, and never make it to the front page of /.

VxWorks has a nice track record in space (2, Interesting)

Anonymous Coward | about a year and a half ago | (#41615357)

At least one instrument [nasa.gov] running VxWorks has been flying on the ISS since 2001. I'd be surprised if it were the only one.

Re:VxWorks has a nice track record in space (0)

toygeek (473120) | about a year and a half ago | (#41615679)

Hopefully it worked better in space than it did in WRT54G's.

Keeping up with the kardashians... (0)

Anonymous Coward | about a year and a half ago | (#41615427)

If you can survive eight hours, you can survive *ANYTHING*...

My PVR (1)

GrahamCox (741991) | about a year and a half ago | (#41615509)

My PVR also runs VxWorks. Given that it still crashes randomly now and again, I hope they have a better version for space probes.

pshaw, we use RTEMS (3, Informative)

Anonymous Coward | about a year and a half ago | (#41615559)

the other big player in space RTOS: RTEMS.
Free, open source, rtems.org.

Has all the same problems as VxWorks.. no process memory isolation (because space flight hardware doesn't have the hardware to support it usually)....

One thing that VxWorks has that RTEMS doesn't, and I wish it did, was dynamic loading and linking of applications. You're basically back in 1960s monolithic image days, not even with overlay loaders.

Re:pshaw, we use RTEMS (1)

jimmydevice (699057) | about a year and a half ago | (#41615599)

Why not FORTH?
It was the to-go system for exploration satellites for years.
I believe Voyager is running it still.

Re:pshaw, we use RTEMS (1)

Hans Lehmann (571625) | about a year and a half ago | (#41615745)

If a better OS came along since the start of the Voyager program, which I'm sure is true, I highly doubt that the Voyager crafts would get their disks wiped and a new OS installed, so to speak, while on their way to the edge of the solar system.

Re:pshaw, we use RTEMS (3, Interesting)

Anonymous Coward | about a year and a half ago | (#41615869)

FORTH is great. From about 2 dozen core instructions an entire operating environment can be built. Unfortunately, FORTH takes in-depth knowledge of not only the hardware, but also a firm grasp of scope of the tasks that need to be performed. Most programmers today cannot handle FORTH -- imagine building your own TCPIP stack, filesystem, and RTOS operating environment from scratch. That talent is found only in a dying breed of programmers, literally.

For a robust 100% reliable radiation-hardened space environment, even the processor, data paths, and memory need to be self-correcting for each data bit. Sapphire-On-Silicon processors are only the beginning of solving the reliability issues. Commercial Off The Shelf solutions are an invitation to disaster, but nobody wants to invest the time and money for proper solutions any more.

You can thank Just In Time supply chains, quarterly corporate focus on maximizing profits, and Globalization for the current sad state of space exploration. Without a paradigm shift in attitude, there will be no more Voyagers. I know. I used to work for rocket scientists.

Re:pshaw, we use RTEMS (0)

Anonymous Coward | about a year and a half ago | (#41622857)

RTEMS is much better overall. There is hardware flying around mars running RTEMS, because I picked it long ago. There is no upside to a lack of memory protection when the hardware supports it.

Your cpu secretly a radio? (-1)

Anonymous Coward | about a year and a half ago | (#41615607)

ALERT! Is your computer secretly functioning as a RADIO?
-=-=-=-

WebSDR: a Tunable Virtual Radio from Holland

(www) iw5edi (com)/ham-radio/58/websdr-a-tunable-virtual-radio-from-holland

"Thatâ(TM)s become a funny game during our last local net on 2 meters.
The screenshot you can see here behind is an online tunable radio working on 80 40 and 20 meters band from Holland.
We did enjoy comparing the delay beetween our receivers and the live feed coming form this web radio, and it was nice to discover that the difference was just half a second or less.

We did also test transmitting and getting heard by the remote station, and was interesting to see the waterfall differences on incrementing the output power.

But the most interesting thing was understanding what is behind this web pageâ¦.

First of all this is not a common remotely tunable radio as you may have had the opportunity to play, there are many remote radios on the net but rather this is a Software Defined Radio. Letâ(TM)s call it a Virtual Radio.

A Software Defined Radio, is a computer program, that emulate a complete radio, where tipical components like filters, mixers, amplifiers, demodulators, detectors etc have been reproduced by software programmed libraries. Input Output is demanded to commom sound cards, and a minimal RF frontend is provieded.

The University of Twente in Holland has published an SDR web interface that allow you to tune the radio, along to other concurrent users, by simulating a virtual receiver. It means that what you tune, is not heard by others, but only from your pc.

Like other virtualization technologies, a shared host platform is supplied (here is a common old-fashion pentium III PC running linux), and from this host you can load your own virtual exclusive radio and tune the waves via the supplied W3DZZ trap-dipole for the 80 and 40 meters

To load this sdr radio, requirements are Java Browser Plugin installed and access through TCP port 8901.

We can just say Bravo! to the hams that promoted this experiment."

Restaurant removes urinals shapedlikewoman's mouth (-1)

Anonymous Coward | about a year and a half ago | (#41615635)

http://www.france24.com/en/20121010-restaurant-removes-urinals-shaped-like-womans-mouth [france24.com]

AFP - A sumptuous new French restaurant in Sydney said Wednesday it would remove two urinals designed to resemble a woman's lipsticked mouth, apologising for any offence they have caused.

The Ananas Bar and Brasserie said the bright red-lipped urinals shaped like an open mouth were "a commonly used European design piece from female Dutch artist Meike van Schijndel".

"We sincerely apologise if they have caused offence. They are being removed today," a spokeswoman said in a statement.

The stylish restaurant opened three weeks ago, with the Sydney Morning Herald's food reviewer describing the urinals as "no real surprise here at Ananas, merely adding to the extraordinary collision of statements and intent".

But feminist, former political adviser and writer Anne Summers said the design was offensive. "Misogyny is very widespread, and this is just an example of misogyny," said Summers.

"The concept is pretty challenging and confronting. They're asking men to put their d(expletive) in these mouths as urinals."

Australia is the grip of a fierce political debate about sexism after Prime Minister Julia Gillard, the nation's first woman leader, accused opposition leader Tony Abbott of being a misogynist.

The unmarried Gillard said Tuesday she had been personally offended by many of Abbott's remarks over the years -- from urging her to "make an honest woman of herself", to his cat-calling at her in parliament.

"If he wants to know what misogyny looks like in modern Australia, he doesn't need a motion in the House of Representatives, he needs a mirror," she said in stinging comments.

The same VxWorks.... (1, Flamebait)

gQuigs (913879) | about a year and a half ago | (#41615877)

that is (or was?) in newer Linksys routers, that are much less stable than the older Linux based versions..

http://en.wikipedia.org/wiki/VxWorks#Networking_and_communication_components [wikipedia.org]

Re:The same VxWorks.... (1)

Anonymous Coward | about a year and a half ago | (#41616483)

that's actually at the fault of the device drivers and glue code.

Unfortunately vxworks has a small and well understood and deterministic core despite various suck points. Important in many areas in control but it has so many cons. The buggy POS crap all over the world running VxWorks is testimount to that. It's like once you consider the human factor in commanding the beast why bother because it's just going to be less reliable in the end. AFAIK you get to pay royalies too... This is truely one of the things in the world i don't understand. You'd think the world would move on to something like QNX or greenhills... On that topic, linux doesn't actually have my vote in this domain of control and RTOS...

Slightly off thread I know... (0)

Anonymous Coward | about a year and a half ago | (#41616751)

As a slightly off thread, I always wondered why Intel bought windriver. One of the issues we have is that finding someone who knows the OS well is difficuilt because there is no way of getting exposure to it unless you have a lot money.

I can't help thinking Intel have missed a trick here. With the rise of the embedded hobbiest with things like Raspberry Pi, a new generation of engineers are learning, however there experience is based around ARM and linux, so further marginalising Intel in the embedded world, which is likely to be the big growth area in the future.

If intel was smart they would create there own hobbiest board based around an embeeded core duo or the like and provide a free version of vxworks to run on it. It doesn't need some of the high end features, but would provide early exposure to the OS as well as raising the profile of Intel in the embedded space.

Just a thought....

Re:Slightly off thread I know... (1)

gnalre (323830) | about a year and a half ago | (#41616763)

As a slightly off thread, I always wondered why Intel bought windriver. One of the issues we have is that finding someone who knows the OS well is difficuilt because there is no way of getting exposure to it unless you have a lot money.

I can't help thinking Intel have missed a trick here. With the rise of the embedded hobbiest with things like Raspberry Pi, a new generation of engineers are learning, however there experience is based around ARM and linux, so further marginalising Intel in the embedded world, which is likely to be the big growth area in the future.

If intel was smart they would create there own hobbiest board based around an embeeded core duo or the like and provide a free version of vxworks to run on it. It doesn't need some of the high end features, but would provide early exposure to the OS as well as raising the profile of Intel in the embedded space.

Just a thought....

Whoops posted as AC

Re:Slightly off thread I know... (1)

tippen (704534) | about a year and a half ago | (#41618107)

As a slightly off thread, I always wondered why Intel bought windriver. One of the issues we have is that finding someone who knows the OS well is difficuilt because there is no way of getting exposure to it unless you have a lot money.

Intel has loads of cash and a near monopoly on processors in most major market segments. They need somewhere to grow and PCs and servers isn't it. The big segments they are weak in are mobile (or more generally, low power) and networking.

VxWorks is very common in networking equipment and in embedded (low power / low processing capability) systems.

I can see where WindRiver looked attractive to Intel. Of course, the risk is that they scare traditional VxWorks customers off by focusing WindRiver too heavily on x86 processors.

Re:Slightly off thread I know... (1)

default luser (529332) | about a year and a half ago | (#41620293)

Owning VxWorks also gives Intel a way to get into military designs. These are high margin, low-volume parts just like server CPUs, so it's a lucrative market for Intel to get into.

That said, they've only made half-assed commitments, offering just 7 years availability of embedded processors [intel.com] (most places do 10+ years). That works for simpler projects, but the bigger government designs may require a CPU upgrade before the finished product even ships!

And yes in the Windows desktop world it's no big deal to upgrade a CPU,. but in the embedded world where board support packages will vary from one board to another (regardless of processor compatibility), upgrading your computer can range in difficulty from simple to incredibly complex. And since these things are always low-volume, you constantly run the risks of running into driver/hardware bugs on a new platform, so there are lots of reasons to avoid changing the hardware powering a project as much as possible.

Wind River? (1)

Viol8 (599362) | about a year and a half ago | (#41616939)

Didn't they used to do Linux distros back in the day?

Yes , I know, off topic , but just asking...

Re:Wind River? (1)

Anonymous Coward | about a year and a half ago | (#41617149)

Yup. Wind River Linux. I remember using it and thinking what a /big/ pile of shit it was. It was slated as an embedded target but the smallest they could shrink it was a gig or so. Their support had no idea how to shrink it, and I only shaved it down be a couple of hundred meg.

Suffice to say, for what they were charging we were able to build our own glibc based distro with newer, more stable components and cram it in under 200M. There was even change.

It demonstrate how inefficient desktop software is (3, Insightful)

Viol8 (599362) | about a year and a half ago | (#41616999)

An old Power PC can fly a spaceship to mars, execute a difficult landing and now semi autonomously drive a robot across the surface of a planet 30 million miles away , yet its not up to the job of writing documents using the latest word processors. Whats wrong with this picture?

Re:It demonstrate how inefficient desktop software (1)

edcalaban (1077719) | about a year and a half ago | (#41617953)

I wonder what the CPU and memory load graphs would look like for a probe versus some standard desktop applications. Might explain a lot.

Re:It demonstrate how inefficient desktop software (1)

Viol8 (599362) | about a year and a half ago | (#41618623)

I would imagine that landing a spaceship takes a lot more CPU than reformatting some text and drawing a blinking cursor.

Re:It demonstrate how inefficient desktop software (0)

Anonymous Coward | about a year and a half ago | (#41619325)

Why do you think that? Landing a space ship can be done using analogue electronics as a control system in the 60s. That means it was simple and light enough even when analogue in design that it made it into space. The rate on the feedback loops doesn't have to be more than a few khz and the amount of processing per loop is very low. More intensive than blinking a cursor yes but is it more intensive than reformatting text? Perhaps not so.

A PID controller is say 10 arithmetic operations per evaluation and only has to be evaluated at the rate of the feedback loops. No, it's not very much processing to control a spaceship landing.

Re:It demonstrate how inefficient desktop software (1)

Viol8 (599362) | about a year and a half ago | (#41619555)

"Landing a space ship can be done using analogue electronics as a control system in the 60s"

I don't remember any system in the 60s where a skycrane had to hover in place, lower a lander down, release it then fly off. Or navigate using image recognition. If you know otherwise fill me in.

"more intensive than reformatting text?"

Oh please. Reformatting text algorithms were running on 8 bit home computers in the 70s!

Re:It demonstrate how inefficient desktop software (0)

Anonymous Coward | about a year and a half ago | (#41621241)

Oh please. Reformatting text algorithms were running on 8 bit home computers in the 70s!

I'm sure that explains why we still don't have hypenation in web browsers and justified text sucks. Hey, browser guys! This one has a clue! You have to use 8 bit home computers!

Re:It demonstrate how inefficient desktop software (0)

Anonymous Coward | about a year and a half ago | (#41621309)

Or navigate using image recognition.

Well, you can call image recognition anything these days, like what univesity students do in their robotic-fight competitions (based on maybe an 8bit luminosity sensor) or what any laser based mouse does to detect movement across a surface. The devil is in the details, isn't it?

Re:It demonstrate how inefficient desktop software (1)

datapharmer (1099455) | about a year and a half ago | (#41618011)

Your desktop word processing software also didn't have a licensing cost in the hundreds of thousands of dollars...

Re:It demonstrate how inefficient desktop software (1)

mcgrew (92797) | about a year and a half ago | (#41622111)

Your desktop word processing software also didn't have a licensing cost in the hundreds of thousands of dollars

It would if you were the only customer and it was only going to run on one computer. Do you have any idea how many programmers MS has and what it costs for salaries and other overhead?

Secret space designs? (1)

scsirob (246572) | about a year and a half ago | (#41617043)

I find the most revealing part of the interview that he publicly acknowledges his customers working on secret designs for space.
I'm sure those customers will deny any such project exists.

Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...