Bait and Switch

Raising money from any source can be difficult - you have to persuade whoever has money that you're the best place for it to go, and you're up against a lot of competition. Sometimes you can do that by showing preliminary work and persuading your peers you have a sensible rational approach to moving forward, and the end result is worthwhile - most grants from the government are done this way. Sometimes you can show existing revenue and that market growth will easily allow the money you need to expand to be paid back, and a bank or other lender will see the value. When you are a public company, you have to show returns and potential for growth that make your stock look appealing to retirement funds and the public. With Venture Capital, the goal is to provide outsized returns, billion dollar companies that give 10x or 100x gains or more, ideally to make up for all the other bad bets made and have their VC fund be profitable. It's this last one, and the behaviours it encourages, that we are going to delve into a little deeper.

What Drives Cheque Sizes and Valuations?

When raising from VC, there are multiple things a company needs to take into account. First, there's how much you need to raise, (Many Series A round are in the $3 to $10 million range) and how much of your company you are prepared to part with to get that.  The combination of those two tells you where your company needs to be in valuation to make that possible, for example if you want to give up no more than 20% of your company and to raise $10 million, you need a $40 million pre-funded valuation (that's the value before you take the money) - as the post-value will be $50 million ($40m pre + $10m investment) and then that 20% is the $10 million investment compared to the $50 million post.

Most VCs have an expectation of owning a reasonable piece of the company, usually in the range of 20 to 25% with 15% at the low end, and 30% at the high end. The range of cheque sizes they are prepared to write depends on the size of their fund - how many companies they want to monitor sets the lower bound, and spreading risk to be sure not all their eggs are in one basket sets the upper end. For example, a $200 million fund may decide that a minimum of $3 million and a maximum of $6 million for Series A companies is their comfort zone, leaving some cash over for seed investment and reserved for later stage funding. These numbers vary with each VC, and with time as their fund matures - they typically last 10 years and what they do in year 1 is very different than compared to year 7.

While normally you'd want to boost the valuation of your company to minimize the dilution of your company, setting your expected valuation as very high will immediately remove a number of VCs from your possible pool of funders. For example, if you want a valuation of $100 million with a $10 million raise, then you need to find a VC not only able to write that cheque, but willing to accept under 10% of the company in return (actually, more than one since often there is a lead and then additional companies that split the deal). With a raise of $20 million the pool of VCs willing to fund is even less, but the % of the company on offer is much more palatable. If instead the valuation moves to $40 million then you still have the same pool of VCs as originally, but they will be much more interested in owning 20% of your company than 10%.

Incentives for the Founder to Push Valuation
You as the founder/CEO want to get the most money for the least equity so the goal is to match financial needs, with the cheque size of a VC that is in your area and likes you, and with an idea/company that justifies the valuation to keep VC ownership in the 20% ish area. Seeing as there is an inherent need in people to both raise as much money as possible, and believe the external validation that your company is worth an enormous sum of money, there is pressure to give the 'rosiest' view of the possibilities for your company. As I wrote before, I have even been chastised by a possible investor for presenting a 'realistic' view of revenue and told instead to show the most positive view regardless of likelihood.

A CEO is under that pressure to inflate values, particularly one without a real grasp of the realities of their area such as those who have little or no actual experience in a technical field - Elizabeth Holmes and the like, for example. They can claim ridiculous things to potential investors that no-one who actually truly understands would say, and say so convincingly because they believe it themselves. If you read VCs talk of what they look for in founders, it's almost always a 'fundamentalist religion type zeal and belief in what they are doing'. Notice in that article it's the last 2 of 12 characteristics that what most people consider critical - Domain Expertise and Integrity - and I've found (to my own detriment) that most CEOs I've worked for utterly fail in both of those. 'Tenacity' is the most important to VCs apparently, after all you don't want your investment to be thinking about reality, there might be a sucker somewhere who eventually buys the very dead horse you've been flogging. (Sorry, I mean "overcome the great difficulties being a founder entails")

Basically, VCs set the terms and incentivize what would normally be considered lying or fraud - why are we surprised when that's what we get? Moreover, it weeds out those experienced in a field who simply understand enough to put realistic expectations on what's possible, or have the integrity to refuse to lie.

Exaggerating or Lying?
When does 'rosiest view' change to 'lying'? There is no sharp line before which it's exaggeration and after which it's fraud, it's a grey area. Sometimes you are smart (or lucky) and what you claim turns out to be true, sometimes it's completely wrong, sometimes it kinda does but not nearly as well as you hoped - very occasionally it's massively better than hoped and you end up a Facebook or WhatsApp. Sometimes you might believe you can reach a metric you are claiming, and only have the resources to know for sure post Series A, and when you learn what you are truly capable of you have to 'pivot' and refocus the company on a different, usually smaller, market you can actually address.

What is it, though, when the company knew for a long time that what they were claiming was never achievable? Maybe they always knew, maybe they learned later once they had the staff and funding, but they kept going because to do otherwise was to admit defeat, and give up any chance of that greater fool buying you. So what does someone like that do? Well if we look to Theranos and Energous, the answer is 'Bait and Switch'.

Theranos Poisoning the Well for other Blood Test Tech
Theranos recently Punked the AACC and managed to give a marketing pitch for a new platform rather than actually give results on their old one on which they had raised $700 million. The old system was supposed to have been able to run up to 200 tests on mere drops of blood drawn from a finger rather than a vein, which if achievable would have been a huge leap forward. They were the darling of Silicon Valley, with huge coverage in the press for the founder Elizabeth Holmes (and all on her, not the tech). It turns out that they were not being very truthful in their claims, and now both the SEC and FDA are pursuing criminal complaints against the company as well as eight class action lawsuits from patients who received false diagnoses from the company. These exaggerated claims allowed them to raise that $700 million while still allowing the founder to maintain a majority holding of stock, for a while making her a billionaire until the truth came out.

So what did this new system do? Capillary blood from the finger? No. 200+ tests? No. Cheaper than existing? No. Faster than existing? No. More utility from a single box? Maybe. Essentially everything that made the company viable and worth investing in was a lie, and now they are trying to pretend the company is viable with a far less interesting concept, and one that was stated by experts to not having anything that didn't exist elsewhere. Had they done this two or three years ago, before actually providing patients with false diagnoses, then it would have been a 'pivot' - a company that made a noble and commendable effort but didn't quite work out. But they didn't, they kept the illusion of capability going far beyond when any sane person would have dropped it, and fully moved to the realm of "Bait and Switch". Turns out they get to keep that $700 million despite at some point having moved from 'exaggerate' to 'lie' in their claims - way to reward bad behaviour.

That VCs burn their money (some of which comes from pension funds remember) on a stupid bet is one thing, partly that's what they do, but because they both allowed and incentivized Theranos' behaviour, that target of a fast, cheap, small, versatile, consumer friendly blood testing was the norm for anyone else raising money in that area for the last few years. Imagine you had a product that did literally half of what Theranos claimed, and you pitched to VCs who kept rejecting you because they expected and demanded a company that exceeded Theranos. An honest founder couldn't pitch that, a dishonest or naive one could. By Theranos continuing the charade of their viability they made it harder for those legitimate startups to raise anything at all due to unrealistic expectations. It's great for a company to kill their competitors but not what we as consumers or investors (or I assume LPs in a VC) want.

Energous and the Pointless Product
Energous are wireless power company who claim to use RF (like wifi) to power small devices like phones. No independent third party has ever validated their system or performance, and some claim they are simply using a "Time to Carrot" approach to constantly keep investors thinking that the pot of gold is about 12 to 18 months out, and just put more money in. They have claimed up to 4 Watts at up to 15 feet from the transmitter but there are so many skeptics, myself included, that look at the physics and maths and show that what they claim is simply not possible. After going through an IPO, raising millions of dollars and the top three executives paying themselves almost $5 million a year total with no product or revenue, how do they answer their critics? By releasing a product, but so simple and with specs so low that it is pointless, and calling it "mini" - and now they claim they have a product, and it's just "the big pot of gold is just around the corner...".

The mini-WattUp is a small USB sized device claiming to charge devices, that needs to be in contact to an inch away (not 15 feet), and charges at a rate that would take days at best to fully charge a phone, but it does have FCC approval (because it does nearly nothing). It's like making a car for a soap-box derby and claiming that next year you'll be competing with Tesla. It achieves the goal of continuing the illusion that there is real technology, real hope of a full scale version which will always be "18 months away".

Had Energous tried to IPO the company based on the mini-WattUp then they would have fallen flat on their face - nothing interesting, useful, or better than the competition (by far). If the goal though is to raise the cash, milk it for as long as they can, then a "Bait and Switch" keeps the money flowing, and that's the most important thing. <sarcasm>It's a pity as it destroys any market for real at-distance wireless power companies.</sarcasm>

Speaking of Other "At Distance" Wireless Power Companies
For some reason, I wanted to remind everyone of uBeam's claims of how they will be wirelessly charging at a distance, and by the end of 2016 (only a few months left to wait!). Released specs are:

• 4 meter range in every direction
• At least 1.5 Watts and often 'faster than a wire' (that's over 10W if true)
• Charges while in your pocket

I can't find anything on safety or efficiency that's public from uBeam, though there are some well written articles on the safety aspect. Just keeping this in mind for comparing to the product uBeam must be releasing soon.

Bait and Switch
So looking at Theranos and Energous, if you're wondering why they make the claims they do which have never been backed up by evidence, it's because they've been paid millions of dollars to do so. The system simply encourages it, and it's basic human trait that when you reward a behaviour you get more of it. VCs by their funding approach are selecting for founders and CEOs most willing to exaggerate, and in some cases willing to lie. If we want to see less distortions in our allocation of capital and see it more go to genuine, viable technologies, then something has to change. In large part, one of the culprits is the tech media, who simply reprint PR scripts they are handed, and give up actual ability to criticise in return for access. We need more willing to ask the hard questions in the way John Carreyrou did of Theranos or Lee Gomes did of uBeam, rather than just parrot a PR line handed to them without question.

Until that happens, expect more companies to raise large amounts on the unfeasible, and then finish the "Bait and Switch".

And finally
Just as I was to publish this, I read a fantastic piece in the The Atlantic by Adrienne LaFrance about "Access, Accountability Reporting and Silicon Valley" which says a lot of what I've been trying to say on the media coverage of tech firms, but far more eloquently. I highly recommend it.

Pick Two, But Only Two

There's a saying in engineering:

"In any product development project you can have it good, fast, or cheap. Pick which two."

You tell this to most people, even non-engineers, and they get it. Something has to give, and choices get made in development. So why is it so hard to put into practice?

Good and Fast:

If you want a quality product developed quickly, then pay overtime, consultants to help, fab houses to expedite, and develop without consideration of materials or processes used no matter how expensive. There is, however, a limit to how fast some projects can be done, and infinite money won't get a project completed instantly - some things just take time. The 'C' level often fail to grasp this - they say they know it, but actions indicate that very often they don't.

It's common to fall for the "Mythical Man Month", where the expectation is that if you double the staff on a project, development time will halve. At some point, this is like saying "Nine women can make a baby in a month", and in fact the management overhead of too many people on a project can slow things down. This is the equivalent of computing's Amdahl's Law, where some tasks can be parallelized, and some are inherently serial - in the end, it's those serial tasks that limit how fast things can be done even with infinite computing resources.

Under pressure of timeline, management often cut what they are told is critical - design verification testing, engineering verification testing, initial user testing - I could say because they have no understanding of the need, or they don't listen when told how important they are, but often it's because they can hope that things will work out, and the blame will be on others if it doesn't. "You should just do it right first time" I have literally been told when objecting to that route. That some problems just have a minimum time and effort to solve, is a hard reality to accept for some. Blame, ultimately, ends at the C-suite, but until that day of reckoning, the minions are suitable cannon fodder for blame. 

Heavy pressure on management also makes them susceptible to the sell from the outside group who swears they can do it faster. better, cheaper than the internal team - "Just sign this contract and we'll take care of it all!". It's nice to be able to sign with someone who promises a solution, when your own team keep telling you of major issues and delays, but when it does go wrong it's the internal team that's going to clean up the mess, and for a lower hourly rate than the outside group was paid. Importantly, it doesn't go unnoticed by the engineering side that they were not trusted, and that does not help future interactions.

One particularly frustrating situation for engineers is when they deliver what's asked for, a quality product in a timely manner, only to be berated for it being too expensive when only weeks before "cost is no object!". Stating that you were (just) within the cost target set falls on deaf ears, and you're sent back to lower the price - without, of course, sacrificing quality or with a delay (which there will inevitably be). I've walked away from those meetings hearing the remaining executives saying "Engineers just never understand cost, it's beyond them." when in actuality we understand it perfectly. Cost is a constraint, like size, or weight, or performance, and we'll work to hit that. Cost targets can move during development, but poor choice of that target is a failure of the business side, not engineering.

Fast and Cheap:

Now, if you want something fast and cheap, then as long as the product doesn't have to be very good (say, falls apart on second use, or doesn't actually do what is claimed) it's achievable. Engineers hate this one, by nature they want to make a good product, and they know they'll be fixing things later on anyway when it will be much more expensive to do than if it was done correctly the first time. Software is often released this way, with the hope that the application is 'just good enough' or that the customer can be placated while fixes are put in place. 

With a non-critical product, and an understanding customer, this can be a viable route for a business, especially with software where patches and updates can fix problems and add needed features quickly. On the hardware side, this is more applicable to disposable or very low cost products, physical objects don't tend to improve with a patch. Despite this, such a mentality has started to make its way into medical related devices such as with Theranos, and other safety critical areas such as automotive - I've written about this in more detail here and here. 

With hardware this method is almost guaranteed to accrue technical debt - problems that cost a lot more to fix in future that to fix now. Hardware that fails in the field regularly and is returned, is a nightmare for warranty cost as well as reputation and returning customers. And guess who gets the blame for a product like that?

Good and Cheap:

Good and cheap (or affordable) is the rarest combination, it's hard to give the time and resources needed to do a job well first time. The pressure to release products to earn revenue, or placate investors, is intense. A few companies can do this, for example Google or Apple with special projects like autonomous vehicles, but in most cases it's very, very hard for normal companies to do this. If you can give a team time to solve problems and iterate, design for cost from the beginning, investigate alternate routes, multiple rounds of customer testing, and the equipment they need, then you can deliver great economical products. Sadly, it's the exception, not the rule.

Good, Fast, and Cheap:

Think you're an exception to all this? Go ahead, good luck, you may be the exception. Odds are that you're going to end up with a product that's late, not quite as good, or more expensive, than if you'd just picked two. If you're ignoring your engineering team who've already told you this, then maybe you're the CxO of a company I've worked for...

Which Two to Pick?

So what is the solution to this problem? There is no one answer to that, every company, every product, every team, every market is different. Getting a product to market, at quality, at cost, and on time, is a monumental task, for both the business and engineering sides of a company. The companies that I've seen succeed the best at this set a culture that is adhered to even under great temptation to move away from - and that consistency has to come from the CEO. This culture usually revolves around a few key points:

1. The business side sets clear and realistic goals for the engineering team - performance, cost, timeline - and stick with them. Engineering need to be able to trust the business side and what they say.
2. CEOs need to understand that their word does not overcome reality, and that timelines in particular can be impossible to shift. Business needs to trust that when engineering say a timeline or cost target isn't achievable, they need to re-evaluate that metric.
3. Engineering need to communicate realities and issues to business in a clear manner, with an understanding that sometimes terms like 'at risk' don't always mean the same to each side.
4. When things go wrong - either engineering performance or business predictions - there needs to be honest and open communication with a mind to fixing the problem, and not a witchhunt looking for the target to blame.

It seems simple, when it comes down to so few things - each person doing their job, open and honest communication, and teamwork to fix a problem when things (inevitably) go wrong. All it requires is competent leadership with a basic understanding of reality. Why is it so rarely seen?

Follow the Money

It seems I'm not alone in my disgust at the leadership of the American Association for Clinical Chemistry (AACC) and how they allowed Theranos to abuse their conference to try to pretend they have any legitimacy. The FT reports that members of the society are apparently resigning in protest at not just giving Theranos the platform they did, but the way the normal rules were bent to eliminate the chance of actual tough questions.

Dr Andy Hoofnagle, a member of the organising committee, said he and several of his colleagues had “fought really hard to prevent” Ms Holmes from appearing but were overruled by the AACC president, Patricia Jones.

“I’m removing myself from the committee and don’t intend to pay my dues next year,” he added, in effect announcing his resignation from the association.

Others were not happy with the decision and called the society President, Patricia Jones, "stupid" but decided not to resign as she leaves office next year. Jones defended her decision, deriding the very idea that even though she'd had to fight her AACC colleagues and force the Theranos presentation on them, anyone could possibly even consider it inappropriate.

This is the same person who introduced Elizabeth Holmes before her talk on Monday, and as she ceded the podium to the Theranos CEO could be heard to say "You're going to be awesome."

Now some have defended that as something a moderator does to help keep a presenter calm and perfectly normal. As someone who has done that role of introducing speakers, I can tell you that no, it's not normal. You'd be hard pressed to consider doing it for a first year graduate student at their first talk, let alone the CEO of a (once) multi-billion dollar company.

I found myself wondering why Patricia Jones was so willing to sacrifice the reputation of her society, or how she could be so naive as to allow a wolf in sheep's clothing to abuse them so badly. As I was talking this over with a friend of mine, a very Sarcastic Brit, he made the following comment:

Follow the money

And then I remembered that this April, following terrible media resulting from the lawsuits and criminal probes it faces, Theranos created a new Scientific and Advisory Board to aid in "advising Theranos regarding the full integration of its technology into routine clinical practice, and publication and presentation in scientific journals and at scientific meetings.". At the time I wrote a piece stating that it was potentially an attempt to buy legitimacy, but now I wonder if it was much more than that.

Of the eight members of this Board, four of them once held the same position Patricia Jones does now - President of the AACC. The board page at the Theranos website lists them as:

Susan A. Evans, PhD, FACB

Ann M. Gronowski, PhD, DABCC

Larry J. Kricka, D. Phil, FRCPath

Jack Ladenson, PhD, DABCC 

Did any of them apply their influence to get special treatment for Theranos and Elizabeth Holmes at the AACC conference? It wouldn't be surprising to make introductions, it's a common part of the role to help with connections, but getting special treatment from a scientific society and effectively endorsing the company would be going too far.

But most importantly, does Patricia Jones hope that this time next year there won't be four out of eight board members as former AACC Presidents, but five out of nine?

Theranos' AACC Presentation: Lack of Data Proves Doubters Were Right

Yesterday afternoon, Elizabeth Holmes, embattled CEO of Theranos, took to the stage at AACC to finally reveal the science and data behind their products and tests. She started with a heartfelt apology for the mistakes the company had made, for the misleading information and marketing both to patients and investors, and proceeded to finally show the actual data and methodology of their tests, warts and all, and then pleaded with the community to help the company move forward and realise the initial promise of Theranos, of multiple, accurate, rapid tests from a single drop of blood. The crowd rose to their feet at the end, and applauded her honesty and humility.

Or at least that's what happened in an alternate reality where shame over fraud and scientific honesty are foremost in the minds of startup CEOs. Instead, what we were treated to was a marketing presentation for a future Theranos product, no apology, no real data, and evasion of any substantive questions. If you have 90 minutes to spare, watch the whole thing, it's a masterclass in abusing the platform given by a professional organisation to do marketing and try to give the company a veneer of credibility to those not versed in the subject.

They didn't just fail to reach the low bar set for them - they didn't even attempt to reach it. To list out what was wrong with this presentation would take a book to cover fully, and there are multiple articles detailed issues with it such as the Wall Street Journal, Wired, and the New York Times. I'll try and cover the main points below - for those of you looking for a history of Theranos, there are some summaries here, here, here, and here.

As someone who has actively organised and managed scientific conferences and talks, I'm stunned by what AACC leadership allowed Theranos to get away with. Theranos committed multiple major faux-pas, and Holmes' celebrity status allowed her to get away with things no other presenter would have been. In a scientific conference the goal is not just to present data, but to withstand the scrutiny of your peers and as such the question component is just as important. The talk was set for 90 minutes, split into 45 minutes presentation and 45 minutes questions. Holmes gave a very polished and well rehearsed talk for an hour - clearly and deliberately done to cut 15 minutes of time from questions. This is an old tactic, commonly used by presenters who fear questions, and one an experienced chair/moderator knows to look for. The session chair was left with the choice of cutting them short and leaving out the component everyone thought they wanted to see, the promised (though never really delivered) final section on actual results comparison, or allowing Holmes to cut into the question time. They went for the latter option, a difficult call to make and one I can understand them doing.

Holmes also committed one of the worst crimes - not presenting what was claimed in the abstract, and instead giving a marketing talk for an upcoming company product. In the Wired article linked above, Stephen Master, professor of clinical and lab pathology at Weill Cornell Medical College. “This sounded like a talk from a manufacturer.” Conferences have commercial presentation sections for that type of talk, and anyone who gives a marketing talk in a scientific slot will never be invited back. AACC leadership should have made comment on this after the talk, stated it was unacceptable, and in being silent they are effectively endorsing this approach.

Her talk was slick and obviously well rehearsed, and covered things in a way that would be ideal for fundraising, but inappropriate for a scientific talk. She used buzzwords that CEOs seemingly have to say, without apparent understanding of their meaning. "Inflection Point" is one of them, which is code to a VC that "we're about to have massive sales, real soon, honest" but everyone else knows is just cover for poor performance to date. For example, Holmes uses it at 42m40s correctly to describe a test function, but tries to pretend at 56m20s that sales of this future miniLab will soon boom. Meredith Perry, Founder and CEO of uBeam that similarly has been questioned about the validity of its technology, said in September 2015 "We’re at a massive inflection point. We are about to head into a completely new phase of growth.” but nearly a year later no sales, no demonstrations, no products. Beware the CEO who promises an inflection point, it's nearly always a false promise.

Instead of answering the many questions scientists had, Theranos presented their future 'miniLab' device, a 'cloud connected' unit that may play well with VCs but fell flat with doctors and scientists who actually use the equipment. Geoffrey Baird, an associate professor in the department of laboratory medicine at the University of Washington, commented in John Carreyrou's Wall Street Journal article:

Every piece of technology they presented has been known for many years, and exists in other platforms largely in the same configuration, or in some cases in much more compact form in competitor’s platforms.

So even the future (non-existent) product they are trying to distract us with does not even move the current state-of-the-art forward in any way. In the initial questions following the talk, all done in comfy chairs and a collegial atmosphere I have never seen at a post-presentation question session, one of the moderators called Holmes on the pedestrian nature of the equipment and the departure from the world-changing claims Theranos had raised $700 million with. Masters noted that the original claim was "Many broad tests, 70 tests from a couple of drops of blood - this falls far short of that... I can buy a point of care instrument today that does a finger stick lipid panel." and received much applause from the audience. Holmes then did what was standard for the rest of the question session, talk a lot, but fail to actually answer the question asked.

Holmes clearly had memorized the line "What we wanted to do here today was..." followed by a few options such as "begin engaging with the community." or "present our future plans to the audience." No-one corrected her that this differed from the abstract of her conference talk, but you could tell people were unhappy with those non-answers.

This continued with all questions designed to elicit information such as "Why present data for potassium with venous not capillary blood?" (a key question about their methods) the reply was "What we wanted to show was..." followed by no answer. When asked about false positives and concordance rate, the reply was that their technique was "Very good" and then no answer, even when asked a second time. It was clear no real answers would be forthcoming. 

Whenever pressed to answer questions on previous methodology based on their Edison systems (which were never mentioned once in the talk despite being the main company product to date), Holmes gave the most frustrating answer of all:

In the appropriate forum, we’ll address those. But today we’re hoping to be able to engage on a scientific exchange on this platform.

A canned answer, giving no indication of when that data would be forthcoming or what that appropriate forum would be - ridiculous given that there probably is no better place than the AACC, and that audience, to address them. The real data everyone wanted to see, the topic everyone wanted to cover, was dismissed in those two sentences, and not a peep from AACC leadership or moderators on that. 

As Wired magazine points out, Theranos just 'pivoted' - that's when a startup company realises their past approach and technology just can't do what it was claimed, and try to shift to another product or market in an attempt to save the company. It's clear from the reactions that no-one was impressed by the miniLab equipment, and that had this been what was pitched to investors, there would have been no $700 million raise. As one of the moderators, Master, said in Wired:

Does this live up to the hype? The answer to that is no.

Finally, no-one raised the most obvious question which should have been:

"Since you (Holmes) have been banned by the CMS from running a company doing blood tests, and you have made it clear that you will not leave the CEO position, how will the Theranos be able to offer this test and equipment?"

That in itself would have been worth the non-answer answer that would have come.

So in summary, Theranos avoided answering any of the questions scientists wanted answered, have abandoned their old technology, and shifted to a new unproven technology that has no advantages over existing available equipment. They truly live in another reality if they think this will be enough to save the company.

Equally importantly though, AACC allowed its prestigious name to be used by someone being investigated by both the SEC and FDA for fraud, to not present real data but instead a marketing talk and to avoid substantive questions, and to fail to press even on the soft-ball questions it asked. Holmes may not have the capability to show remorse, but the AACC should be ashamed for what it has enabled here.

Inflection Points and Hockey Sticks

The term "Inflection Point" is used repeatedly during startup pitches, and is one of those overused terms you grow a little sick of when you have to deal with that world, in part since it is rarely used to describe an actual inflection point. So what is it? 

In mathematics it's the point at which the curvature of a curve changes direction, like below:

But since we're talking about investing and startups, what's usually meant is that it is a moment of dramatic change, a sudden upturn (or downturn) in sales or growth. (Obviously, when business use the term they rarely want it associated with a downturn, but it is equally applicable!) It's pretty obvious where the inflection point in the below image is, and tends to be more of the business meaning.

When you're doing a pitch-deck in fundraising, you have to come to your predicted sales graph, and if you want funding, it better be a "hockey stick" - that is, flat for a few years, then around year 3 to 4 starts to uptick significantly, then leaps hugely in year 5. Without that, you get no funding. So pretty much every pitch deck for a startup shows a loss in year 1, a smaller loss in year 2, breakeven in year 3, modest profit in year 4, and $10 billion of revenue in year 5. Small exaggeration, but not much. Here's WeWork's numbers, with which they raised a substantial amount of money, nearly $1.5 billion

As an example of how this can play out, in a former life I was pitching for funds for a company that was developing software for retail. Creating a business plan, I thought that conservatively we could be at around $25 million a year in revenue by year 5, and as we were asking for $250k to $500k from angels, it seemed reasonable. Of all the slides in the deck, I got grilled on that on in pitches more than any other, and was flat out told that the company wouldn't grow big enough for it to be of interest. So I reran the numbers, changed some assumptions to be more positive, and pitched at $40m by year 5. Same result, and detailed questions on that slide, and disbelief.

At this point I really was struggling, as I simply couldn't justify moving much past that $40m number (which, for those of you who don't do this kind of thing, is a pretty damn good number to get a return for a ~$500k investment). Without money, we were going to have to stop work, but how to get it without plain lying?

My solution was this - I still showed the "$40m by Year 5" slide, but then had a second chart with the caveat "If extended to multiple verticals" and the numbers reached $150m a year by Year 5, and then made no claims as to the chance of achieving that. This was ridiculous, it was never going to happen, but here was the interesting thing - as soon as I did that, not a single question on the financials, we just moved smoothly on. It was amazing.

Finally, I asked one of the potential investors why he hadn't questioned me on that slide as it was clearly never going to happen and his answer was this:

It's not your role to decide to be conservative for me or to decide my risk. I decide that. You show me as good as it can get, I'll factor in my acceptance for risk. Investors expect to see curves like that, just show them.

So there you go - if you wonder why the funding goes to the charlatans that are prepared to lie or make outrageous claims it's because the people with money demand that you do. If you don't they think there's something wrong with you.

And if you wonder why startup CEOs use "inflection point" for things it's not even vaguely appropriate, it's because saying it got them millions in funding, so they just keep using it on everything else as it must be a Harry Potter-esque phrase of magic power.

Theranos Finally Presents Data - Will They Fail to Meet the Low Bar Set For Them?

A few months ago I started blogging about my experiences in Silicon Valley and at startups, with a focus on the way the startup ecosystem actually works, and how it encourages the appearance of success to raise future funding or sell, rather than on actual technology and business progress. I drew heavily on my experience as VP Engineering at uBeam, and pointed out a strong similarity with another media darling CEO, Elizabeth Holmes of Theranos.

I've watched all the recent publicity surrounding Theranos, about a young, media-darling, Steve Jobs idolizing female CEO, who defies the skeptics and without any training in the field apparently makes strides that promise to change the world. Who raises millions on promises of ground-breaking technology that's just around the corner, yet never makes public demonstrations or subjects their technology to third party audit. A CEO of a company that never answers substantial questions raised by veterans in that field, or criticism that they are doing nothing but playing a smoke and mirrors game to fleece the next round of investors, but is more than willing to speak to the press on their 'vision' (and nothing more).

Well as of later today, all that will change for Theranos as unlike uBeam, who still have not released any details on their technology despite promises for both demonstrations and mass volume product in 2016, the company will finally present data to the public during the American Association for Clinical Chemistry’s (AACC) 68th Annual Scientific Meeting & Clinical Lab Expo in Philadelphia, PA.

This presentation comes at a difficult time for Holmes and Theranos, with the Centers for Medicare and Medicaid Services recent ruling resulting in the company having its licence to operate their main testing lab in California revoked, and Holmes herself banned from the blood testing business for 2 years. Earlier in the year, Theranos revised many test results it provided to patients based. Given this is the entire business of the company, it places them in great jeopardy, and normally would result in the immediate removal of a CEO who not only got them into this position, but whose continuation in that role would result in the closure of all other remaining lab testing components and effectively end the business. 

But this is Elizabeth Holmes and Theranos, and the mentality here is one that mere mortals cannot comprehend. To Holmes, she is not the CEO of Theranos - she is Theranos. There is no company without her, there is no considering anything else. No matter the consequences she will not waiver, she would rather the company die with her at the helm than live under the guidance of another. It's the perseverance VC's say they look for in founders, that's more important than technical or business skills, to stay the course no matter the consequences, that's backfiring on them in the hands of a founder who has a plurality of stock and a majority of board seats. No matter what they think, what they say, their money is now tied up in the hands of someone who they have no control over, and who has decided they don't have to play by the normal rules. 

If normal people's jobs and pension fund returns didn't depend on it, it would be hilarious watching the VC's deal with that monster of their own creation. And understand that people of this mentality care only for their 'vision', and the jobs or savings of others are simply tools for them in what is often considered their 'destiny'.

This mentality is partly why I am very much looking forward to seeing the presentation at the AACC. Founders with this type of thinking do not need facts or evidence to know what the truth is - their belief in what they are doing is so strong that they have convinced themselves it is fact, and that the engineers and scientists they employ are simply there to uncover that truth for others. Failure for systems to work or results to show them right are not evidence of their belief being wrong, but rather that their staff are not working hard enough, or aren't smart enough, or are even out to sabotage them. 

They are used to setting arbitrary goals and deadlines that their teams scramble to meet, despite the workers knowing the impossibility of it and the pointlessness of trying to explain it to their CEO, who considers those who question based on reality as heretical and to be fired or driven out. When the deadlines are missed and goals not met, the CEO fires those who failed and new goals and deadlines are set, and the cycle continues.

These CEOs are not technology and business leaders, they act more like prophets of a new religion, where bringing their imagined technology to the masses is their destiny, their role in life. They believe their own press that they are technology geniuses, even when others did all the actual work in developing and implementing the products, and know in their hearts that if they present the information (rather than those incompetent scientists and engineers) then the public will finally understand their true genius.

It's this thinking that likely propelled Holmes to volunteer to present to the AACC - something the lead engineers in her company probably despaired at when they were told, and tried in vain to prevent happening. They probably understand that the data the public expects to see will not match with what the company can show, and that there will be one of two outcomes - that the public see data that shows the inadequacy of their technology and methods, or the public will see that someone is trying to present data in a way to hide the inadequacy of their technology and methods. 

When presenting a product for the first time, particularly a ground breaking one with extraordinary claims, companies in the medical field collect years of meticulous data, yet it was as late as May this year that Theranos began collecting data for this presentation. As John Carreyrou of the Wall Street Journal reports

The company didn’t have sufficient data to present. The Food and Drug Administration had deemed studies Theranos had submitted—other than one for a blood test to detect herpes—inadequate, two people with knowledge of the matter said. Theranos needed to collect new clinical data to present at AACC, one of these people said.

Theranos started new studies in May and still is in the process of collecting and analyzing data from them, this person said.

I'm imagining the poor technical team from Theranos, up at 1am the night before the presentation, exhausted from weeks of desperately trying to massage the terrible data they have into something that's semi-presentable, and hoping to maintain some shred of professional credibility, yet being scolded by their CEO for the poor quality of the work and the lack of dedication.

And what do I expect to see in this presentation? Well, having been the Associate Editor in Chief of a peer reviewed scientific journal for many years, I can say I've grown accustomed to the various attempts people make to massage or hide bad data, and I am no longer surprised at what some expect they will get away with. I fully expect that's what will be done here. 

They will fail to meet standards typically expected of peer reviewed papers - insufficient information to allow reproduction by another, no statistical data on all tests run and whether they are showing the cherry picked best results, showing results from the easiest tests, deliberately poor labeling of data or systems used to prevent direct comparisons, results from blood drawn in the normal manner and not finger-prick tests, changing the meaning of standard terminology to their benefit, and hiding behind 'company confidentiality' to name a few. Throughout this, the thread that will tie it all together will be an arrogance from the presenter that she is right and the 30,000 years of combined experience in the room are wrong.

But no matter how badly this presentation goes, even if utterly rejected by the scientific community of experts, I don't expect to see Holmes change one little bit. After all, these experts are just trained so much they can't "think outside the box" like she can. It'll just be another hurdle for her to overcome, and that history will prove her right, she just needs to persevere.

Sounds crazy? Yes, it does, but remember, CEOs like that don't live by the same rules as you or I. And then remember that the money that lets them act this way, might have come from your pension fund. But whatever happens at the presentation, the circus that is Theranos is going to provide interesting reading for many years to come, at least your retirement funds bought us all that.

PC Build. New Home Office System. M2 RAID for Boot/OS

It's been a while since I posted anything, it's been a funny month. Work end-of-quarter things (yes, I know the date), personal and work travel, and then following Brexit and its effects it was just so depressing to even write anything. Fortunately on that front there are plenty of far better writers than I arguing with the same kind of intensity I like to see - I'd recommend Alistair Campbell as someone to follow there. I started playing Fallout 4, and there went a couple of hours a night (just one more mission...), but I was also spending some time researching and building my new home office PC. Now for those of you looking for business or startup commentary, stop reading here, because the rest of this post is about building a PC. No kidding.

My current PC is getting a bit old, nearly 5 years. I've gone that length of time between building brand new systems before, but in those cases they have been slowly evolving, single parts changed at a time, until usually by the end it's nothing but the case that remains from the start. In this situation though, it's basically been the same hardware the whole time. It seems to be a sign of the maturing of the industry that the refresh on computer hardware is now moving to more than 3 years, when it used to be that 18 months was enough you felt you were falling behind. Mobile is now where the rapid obsolesence is, though even there the cycle time is slowing and with some phones becoming modular, we could see a similar shift in patterns and move to upgrades rather than new purchases.

My home use involves a mix of the usual web browsing, email, and docs/spreadsheets, but also some work in numerical simulation, CAD, and visualization, and it was in those latter three that I was really noticing the age of my system. That old system is going  to get repurposed as a basic work machine, or maybe a Plex server for the house - even at 5 years old it's still pretty good. The Sandy Bridge chipset was a solid base, an i7 CPU at 3.4GHz, I had one of the early SSDs in it, and a 6 series nVidia graphics card. So where was it falling down? 

Interestingly it was not really in the CPU that the problem lay - the new CPU is 4GHz compared to the 3.4 of the old, both are 4 core with hyperthreading, even with the new IPC improvements you get in each generation, you're looking at only a ~25% improvement in raw power. Pretty much I could have stuck with the old CPU and not noticed too much difference, however it's in all the other parts that I wanted an improvement.

Visualisation needs a good graphics card, CAD seems to eat up any improvements to power (similar to the old "What Intel giveth, Microsoft taketh away"), and the 6 series nVidia card was just not cutting it anymore. Further, GPU computing is really where leaps are being made now, not in CPU, and there have been major improvements here over the last few generations (not just scientific computing, but machine learning/AI), with multiple new architectures. nVidia just released their 10 series with the Pascal architecture, so for the GPU computing and to be working with the newer capabilities, I got one of their 1070 cards (I wanted a 1080, but paying nearly double for about 30% better performance just wasn't going to happen). To get that card I needed a motherboard that could support PCIe 3.0 16x lanes, slots/PCIe lanes for future GPU SLI (a second graphics card in parallel), and a PSU that could deliver all that power - my current system couldn't do that, and any upgrades would have been as expensive, and likely less effective, than going new.

I'm also fedup waiting for large datasets to save and load to disk. While I was using a Samsung 850 SSD, read and write speeds are around 400 MB/s which is way better than HDD (around 100 MB/s), it still gets annoying when you deal with 10GB of data writing to and from the disk. They also connect through SATA III and so are limited to 6Gbps. Fortunately the newer NVMe M2 SSDs now read at up to about 2.5GBps and read at 1.5GBps (yes B for 'bytes' not b for 'bits'), and because they connect through PCIe, a 4x lane 3.0 setup will in theory limit at around 32 Gbps. So a new M2 SSD will at most use around ~35 to 60% of the available bandwidth and not be limited.

Intel's Z170 chipset seemed to be the right choice, with up to 40 PCIe 3.0 lanes (20 chipset, 20 CPU), and support for USB 3.0 and RAID. Looking at the various motherboards that used it, I liked what I'd been seeing and hearing about ASRock, and so looked at their Extreme7+ board. It had all the features I needed for now with room for expansion, an additional USB 3.1 port (doubles the transfer speed), Thunderbolt,  boot from NVMe, and interestingly not one but three M2 sockets for SSDs. Anandtech had a good review of it where I saw this RAID option, but didn't cover how well it worked, so now I started wondering if I could put multiple M2 SSDs in a RAID0 volume and triple the bandwidth? 

Digging deeper, there was nothing that said that it wouldn't work, though also nothing concrete showing that it would. The 4x PCIe lanes are shared, so it would be a 32Gbps limit, likely less than that, and even if more lanes were available it all goes through the DMI which would bottleneck it at <32Gbps (though shared with anything else talking to the CPU so definitively lower than that). Three Samsung 950 Pro SSDs (the 512 GB ones, not the 256 GB, they have 1.5GBps and 0.9 GBps write speed respectively) would theoretically be up to 7.5 GBps read (60Gbps) and 4.5GBps write (36Gbps), while two would be 5 GBps read (40 Gbps) and 3 GBps write (24 Gbps) - so three SSDs would saturate the available bandwidth, two kinda straddles the limit. Given my budget, I decided to go with two of them as three probably would see the full gain, plus it was a risk in doing something I didn't know would work, and with two at worst I'd have a second fast storage drive. So I ordered the parts and built my first PC of the decade.

Motherboard: ASRock Z170 Extreme7+

CPU: Intel i7-6700K 4GHz

GFX: ZOTAC GTX 1070 AMP!

SSD: Samsung 950 Pro PCIe3.0x4 512GB x2

HDD: WD Black 5TB

Memory: 2x16GB PC3400 G.SKILL

PSU: EVGA SuperNOVA 650W Platinum, Modular

Case: Phanteks Enthoo Pro PH-ES614PBK

Fan: CoolerMaster 212 EVO

DVD/CD/BD Drive: LG Black WH16NS40

A few notes on the parts. I made sure not to go with a Founder's Edition of the GFX card, you pay more for lesser performance (yep). Memory matched from the motherboard manufacturers recommended page, and PSU large enough to handle future disk and GFX expansion. The Phanteks case had good reviews, and while larger than my existing case seemed to be designed for good cable management and ease of access when building (which it really was). And finally an anti-static wrist strap so I don't zap any of my new expensive components while handling them. I ordered from NewEgg - I looked at Amazon but the GFX card wasn't in stock until August.

All the manufacturers did a nice job of packaging their products, they put the effort in with presentation and little extras to make them look like premier suppliers, and it worked. I'm too used to just getting brown boxes with nothing but the OEM part like the WD Disk in the middle there. Putting it together was really pretty straightforward, these days all the cables seem to nicely labelled, and I only had to look at the MB manual to see the pin orientation for the power on and reset buttons.

One thing I did get caught with - the graphics card had two 8 pin power connectors (GFX cards draw more than the 75W allowed by PCIe so need additional), while each of the two VGA PSU connectors had an 8 (6+2) and a 6 pin connector. I hadn't expected that, so I ended up connecting to the one GFX card from the two PSU outlets. This is fine for now, but should I later put in another GFX card in SLI, I won't have the power connector. The manufacturer lists it as a 220W card so two 6 pins should be enough, and a 6 and an 8 should be more than enough (300W). Two full 8 pin power connectors for the GFX card seems excessive, so I don't know if it's really needed or not - more investigation needed here.

The M2 slots for the SSDs are on the motherboard, it's not the old world of running a SATA (or IDE) cable to them anymore.They sit parallel with the motherboard itself, inbetween the PCIe slots. I chose to put my two SSDs in slots 1 and 3 as slot 2 was directly under the graphics card and I wanted both easier access and no possible heating issues. I was concerned I might have to move them if the RAID option didn't support skipping a slot, but it worked out fine. Use of the M2 slots does come at a cost - some of the SATA ports get disabled for each one used. By leaving M2_2 unused, that meant I had SATA3_2, 3_3, and EXP1 free, all others were disabled. This was fine, I only had the Bluray drive and the single HDD, but if you're planning on doing this and having multiple extra HDD's, then plan on using the ASMedia chip SATAIII ports (4 of them) or a PCIe add-in card. 

As a side note, it's interesting now how small the storage has become, while the graphics cards are enormous (basically it looks to be almost all cooling that takes up the bulk of the volume, same as with the CPU/cooler combo).

Once it was all put together, I powered it up. Fans spun, a few lights flashed, but nothing else. The nice little LEDs on the motherboard reported error 53, which indicated a memory issue. I checked the memory was in the right slots (A2 and B2 according to the manual), then reseated it. This time the system POSTed and I got to the BIOS, which said I had only one memory DIMM. Power off, reseat the other one again, and restart. This time all memory was seen. And that was it for fiddling with connectors - so far all the USB ports, disks, sound etc have all worked fine, so nicely done to the various manufacturers for making the labeling that good.

Now came the fun part - it is fun, even though you curse it at the time - where I spent near 6 hours trying to get the system to see the SSDs in RAID. I got there eventually but it was a bit of a tortuous route, hopefully what I write here will remind me what I did if I have to do this again, and save someone else the headache of doing it themselves if they find this. First, make sure you have a spare USB drive and another PC to work with as you jump back and forth troubleshooting.

My motherboard was on BIOS v2.0, latest is 3.0 (as of July 16), so first of all flash the BIOS. You can get the latest for this board here, and it not only updates the board BIOS but the Intel Rapid Storage Technology firmware too which is a key part. Put it on the thumb drive, then in the BIOS go to the 'Tools' section and there's a utility to flash. It does both the Intel Management Engine and the BIOS. Reboot and then it should be able to do the next needed steps for putting the SSDs into a bootable RAID.

In the Advanced version of the BIOS, go to the Storage Configuration option. Set 'SATA Mode Selection' to 'RAID', 'OpROM Policy' to 'UEFI Only', and Enable the relevant M2_x slots. I saved and exited at this point (F10), though I don't think you need to, but I was overdoing that to be sure that wasn't the problem.

Then to the Boot menu and scroll all the way down, there a 'CSM(Compatibility Support Module)' entry, go in there and change everything to 'UEFI Only'.

Then back to the 'Advanced' tab, and now into the 'Intel Rapid Store Technology' option, which should list your SSDs and the option to 'Create Volume'. Click that.

Then name your volume, select the disks (X) and it should default to the maximum size available. You have RAID0 (stripe) and RAID1(Mirror) options. Mirror buys no performance or storage gain, but heavy redundancy, RAID0 offers no redundancy (slightly increased chance of things going to hell as when one of the two SSDs dies, that's it for everything), but increased performance and storage size. I went with RAID 0. Click 'Create Volume' and it should be done. I left the strip size at 16kb, what limited information I could find showed that it was not a concern for performance.

Now back to Boot tab, and set the primary boot to UEFI version of whatever you want to boot from, in my case the Bluray drive. At this point I did *not* see the M2 RAID array as available as an option in the BIOS which had me fooled for a bit. Pop in the installation media, and save/restart.

From here I'm going to talk only about Windows10 installation, I didn't do Linux yet so can't make comment. Once it restarts, you should get the 'press any key to boot from DVD' message, do so and the standard Windows installation begins. Walk through as normal until you get to the "Where to install" option - where you will most likely not see the RAID array as an option. Instead, use the option to browse media for drivers, and have the appropriate ones loaded onto the USB thumbdrive. This part caught me out for quite some time - the drivers are here (for Win10 64 bit), it took me a while to work out which were the right ones (SATA Floppy and Intel RST), but every time I tried to use them it would find them, work for a while, then refuse to install. Eventually I noticed that the drivers on Intel's own site, here, were v14.8 compared to v14.5 on ASRock's site. Once I was using the latest Intel RAID drivers with the latest BIOS, then everything went smoothly with the Win10 install.

When installation was complete, I then also had the LAN drivers from ASRock's site on the USB and updated them before connecting to my network. I use wired, the house is wired with Cat5e so I get gigabit and it's much more reliable than WiFi, which I leave for truly mobile devices. Similarly, I had the latest nVidia drivers and sound drivers and updated them, before connecting to the network and downloading updates.

Last steps were then to take a couple of restore points (before and after AV installation), and create a recovery USB drive - while I do have original Win10 media, a recovery drive is kinda handy to have for when things go wrong. I then partitioned the HDD to include a 1 GB section that I'm going to use to regularly backup the contents of the M2 SSD RAID array - remember it's RAID0 so vulnerable to total data loss should one of them fail. This is some backup, but not ideal - however backup should really be on a different system, so for now it will also go to an external NAS (until I build a ZFS NAS, but that's another project for later in the year). So consider yourself warned if you do this that your data is at risk!

So how did it perform? Below is a CrystalDiskMark disk result:

You can see it's topping out near 3GB/s which is what I was expecting, and seemed to point to DMI as the limiting factor. I'll be running more tests later and seeing if I can glean more details. Anecdotely however, this thing is really fast and smooth, large files just go to and from the disk like they were tiny - which isn't surprising when I just had a near 10x improvement in disk bandwidth!

And as to the graphics, well I loaded Fallout 4, set it to native 1440 instead of 1080, turned everything to the max, and it handled it very smoothly - there's a noticeable improvement in quality and experience, so probably a near 10x improvement there too, image below. I have no idea, though, how that pack brahmin got up on the roof...

Update Oct 29th 2017: The Windows 10 "Creator's Fall Update" caused me no end of issues installing. Blue Screens of Death (BSOD) every time. The first was "CLOCK_WATCHDOG_TIMEOUT", which apparently, or so Microsoft tells me, is due to me overclocking my CPU. Except I'm not. I tried a couple of things, first of all updating the motherboard BIOS to the latest, which had no effect, then updating drivers. I had a ton of them to update, and I eventually gave up doing it manually and went with the paid up version of "Driver Easy", which I know I can do it all for free myself, I'm just at that point my time is worth more. That did the trick and cured the watchdog timer, except now I got a different BSOD, this time "SYSTEM_SERVICE_EXCEPTION". It took me a few days, but in the end it turns out that this update has an issue with NVMe boot disks, especially the Samsung 950 Pro - the installer is looking everywhere but there during the reboot, can't find a disk, and so fails. Solution - in the BIOS Boot section, disable all options other than "Windows Boot Manager" and also physically disconnect all other drives (in my case the DVD drive, the HDD, and a USB memory stick I had in there). Maybe just one of those actions would have worked, I did them both. And everything installed first time.

So thanks, Microsoft, for making this a giant pain to install. Oh, and with the Windows 10 Creator's Update, it looks like all privacy settings are reset to the default "Let everything access everything" level - type "Privacy" in the search bar in the bottom left and reset them to your preferred level (mine is "nothing"). So screw you MS for that. Oh, and Paint was deleted, now it's "Paint 3D". Screw you for that too.

Self Driving Cars - It's That Safety Mindset Again

There's been some attention to the safety aspects of technology in the last week, as reports of a fatality of a driver while using Tesla's 'Autopilot' feature in their Model S. The company reported this in a blog post, and they made sure to highlight that Autopilot was still in beta and it was the first known fatality in 130 million miles of driving, compared to an expected death toll of 1 per 96 million miles. They are at pains to point out that the customer's use of Autopilot "requires explicit acknowledgement that the system is new technology and still in a public beta phase before it can be enabled.", and that neither the system nor the driver noticed the truck, so it's clearly not their fault, time to move on.

We've been hearing about self driving cars for some years - Google have been at the forefront of this, with all the other care manufacturers trying to play catchup, Tesla included. The promise of infallible computers taking control and driving us, eliminating human error, drunk driving, falling asleep behind the wheel, and other causes of injury and loss of life is a enticing. It's something we've seen in science fiction films for decades, and at some point in the future it will be a reality - but there are no companies selling cars that are fully autonomous - in fact on the NHTSA scale of 1 to 5 (4 to 5 being what most people would consider autonomous), Tesla's system might rank at '2'.

Combined Function Automation (Level 2): This level involves automation of at least two primary control functions designed to work in unison to relieve the driver of control of those functions. An example of combined functions enabling a Level 2 system is adaptive cruise control in combination with lane centering.

Google's self-driving car is listed as possibly a '3'. Google itself notes that:

There are test drivers aboard all vehicles for now. We look forward to learning how the community perceives and interacts with us, and uncovering situations that are unique to a fully self-driving vehicle.

So even though the NHTSA rates Google's car as more autonomous than Tesla's, Google do not allow the public to drive it, and make sure there is a test driver there at all times (presumably as paying full attention as part of their job) and are trying to encounter all the unique situations the car is likely to find itself in. The LA Times has an article going into more detail on this here, and why calling a feature 'Autopilot' may lull consumers into a false sense of security. I would place a bet that the engineering teams had a fit when marketing decided to use that term, but were not listened to.

So what's wrong with Tesla putting in these new advanced features? Absolutely nothing, in fact it's a great thing that they are looking to improve the technology in their cars, however I feel they have made some major mistakes in their introduction and their marketing that fail to take into account consumer safety. I've blogged before about "disruptive" way to get new products into the hands of consumers, to have them 'beta test' for you, has moved from apps and games where it's non-critical and so a good approach, to healthcare and other critical infrastructure - such as Theranos and their blood testing - where it's not. There's a generation of tech business leaders and investors who have never dealt with safety critical products, and they fail to approach them in the correct manner medical and other safety critical industries traditionally have - industries where the suggestion of 'beta testing' a new feature with impact on safety would be met with horror, and where if a user died there would be no hiding behind statements like "well, we did put a warning somewhere in the manual".

We accept that nothing is 100% safe, even walking down the stairs can result in a fatal fall, eating your dinner can result in choking to death, but the chance is so low that we don't even think about it, and act as if it's completely safe. In the USA in 2010, prior to any autonomous vehicles, there were 33,000 deaths, 1.5 million injuries, from 5.5 million accidents, and around 1,500 car trips per person per year for near 500 billion journeys over 3 trillion miles. That means 99.9997% of all car journeys in 2010 resulted in no injury at all, and 99.99999% of all trips didn't result in death - yet that's still 33,000 people taken from families and an estimated economic cost of $231 billion per year even though we consider driving 'safe'. Autonomous vehicles can reduce that cost, but how to develop and introduce this capability is still in question.

How safe should something be before it is released for use? There is no set number here to work from, but typically you start by viewing the necessity of the product and who will be using it. For example a new drug goes through many medical trials, FDA approval, and still remains under the control of a qualified doctor before being prescribed to a patient, but a new wireless router undergoes basic FCC and UL testing before being released to consumers.  If the feature being introduced can't cause harm, then a risk of failure is reasonable, but for safety critical it must exceed the performance of the existing system and you must be able to prove that.

Tesla state that 130 million miles have been driven on Autopilot, and this is the first death compared to an expected 96 million miles - proof it is more than safe, it has already saved a life! Except it isn't proven. 130 million miles may seem like a lot, but compared to 3 trillion miles driven a year, it's nothing - merely 0.004% of the yearly total. You would have to drive much more than that to get anything statistically significant, and that's what matters here, statistics. You need to prove that it's safe, that more than 99.9997% of journeys result in no injury (at minimum), by gathering the data in sufficient quantity that all possible conditions are covered, and showing your system is better than the existing before replacing it. 

That is an enormous task, requiring many times the data that Tesla has gathered so far, and highlights the difference between Google and Tesla - one is taking their time, gathering information and data before taking steps forward, the other is working from the belief that their product is safer than before (which it may well be), however they do not have the data to prove it. The data they need to prove it, is currently being gathered, though in a somewhat uncontrolled manner, as Tesla owners drive their cars and use their Autopilot. Imagine the following scenario:

A pharmaceutical company creates a new drug that they really believe reduces heart disease after their lab tests on 5 people. They immediately sell it over the counter to millions of people, with instructions in the packet about not taking with antacids and that it's in 'beta', and monitor everyone taking it. After a few years they have the data and show everyone they were right - it did save more lives than died from taking it incorrectly or side effects.

Was the hypothetical drug company vindicated and right to do that, or reckless in their approach and lucky it worked out well? By our current laws, they'd be shutdown and executives in jail for doing that. We need roll back this recent thinking of safety as something that is suitable for a beta test, and make sure that the critical things we rely on in our lives are proven safe and effective before they reach consumers.