Deployment Best Practices

Thanks to the evolution of devops & the emergence of containers, applications are more portable now than ever before. So now the question is, how do startups build products to satisfy the needs of on-prem, cloud SaaS and hybrid cloud customer deployments?

Join HashiCorp CEO Mitchell Hashimoto, Heptio Co-Founder Joe Beda (Kubernetes & Google Compute Engine) and LaunchDarkly CEO Edith Harbaugh as they discuss the impact of architecture on a company’s deployment options, and their visions of the future of deployment.

Mitchell Hashimoto is the Co-founder and CEO of HashiCorp, where he focuses on helping teams of all sizes deploy modern applications into complex infrastructures spanning physical, cloud, virtualized, and containerized workloads.

Edith Harbaugh is the Co-founder and CEO of LaunchDarkly, where she helps customers like Microsoft, CircleCI, and Atlassian serve and manage over 10 billion features per day.

Joe Beda is the Co-founder and CTO of Heptio, where he continues his work to support and advance the open Kubernetes ecosystem.

Joe Ruscio: We’re here to talk about deployment. I wanted to put a really fine point on it. Deployment is how you as vendors figure out how to get value into the hands of your customers, in a mechanism that meets them where they want to be to receive that. Ideally, in terms of the last panel, monetize that in a way that’s beneficial to you. We should start today by running through, historically there’s been a couple of different models of deployment. Whether it’s on premise, there was a brief period of delusional-ness in the 90s with ASPs. We don’t talk about that anymore. Then SaaS of course, which some would say we’re in the golden age of SaaS right now. So we could just run through, start with Joe and go down. What you do for deployment, what models you use?

Joe Beda: We help enterprises make best use of Kubernetes. A lot of what we’re doing is on the ground with field engineers, helping them and training helping them understand and run Kubernetes, providing tool sets there. That’s really the teaching somebody to fish type of thing, but then we’re also building out complimentary services. We’re still pretty early on with that journey, and we’re dealing with really big companies. So we’re still evaluating the spectrum of choices from pure SaaS to shipping people’s bits, and everything in between.

Joe R: Even some exploration going on right now. This panel will be useful to you.

Joe B: Yeah, I’m listening. I want to see what these folks have to say.

Joe R: OK. Great.

Edith Harbaugh: I’m Edith, I’m CEO and co-founder of LaunchDarkly. We are a feature management platform. We’re primarily helping SaaS and IT companies manage getting their own features to customers. By definition we started off as SaaS, so we have customers for example, HashiCorp is a customer. We have customers like Meetup who use us to say, “This is what features that people get in that very agile quick environment.” By necessity we were SaaS. That’s just what we started as.

Mitchell Hashimoto: Cool. We have a number of open source tools and we have enterprise variance of a subset of those tools, and it varies by a tool but we deliver– I’ll break it down by revenue. That’s the easiest way to do it. About 90% of our revenue today comes from on premise installations. When I say on premise it can mean bare metal, or it could just be in private VPC, but you are managing it. Another 10% is SaaS. We do have a SaaS offering that’s growing, and then more recently but too early to talk about revenue numbers, is we’ve been working with MSPs as well. Looking at our own managed services, single tenant. I laugh because we end up doing everything.

Joe R: Yeah. At some point, the whole basket.

Joe B: That’s where we’re headed too.

Joe R: A quick timeline on that. With open source background, naively it seems like the on-prem would be the first move. Did you start– Is the SaaS relatively new for you?

Mitchell: Our first move was on-prem. Because with open source we’re used to delivering packages and binaries to people. It was fairly straightforward to deliver the on-prem, I would say that the hard parts for us were the non-technical parts.

Joe R: It’s interesting because, Edith, one thing we were talking about is there are different classes of SaaS applications. There’s some where we’re just wrapping up business logic in front of a database, it’s conceptually easy to package that up and to shrink wrap. LaunchDarkly is not one of those, right?

Edith: At its most basic you could think of us as a set of SDKs, which our customers actually install in their own software. Then we have a centralized dashboard where their developers, product managers, marketers, customer success are controlling functionality. You could say in some sense that we do have an on-prem, and that we have a lightweight SDK that people have to use.

Joe R: You have an edge piece.

Edith: Yeah, but the heart of it is really our own dashboarding which is in the Cloud.

Joe R: With your 90% of revenue on-prem, what’s the biggest challenge of that model for you?

Mitchell: The biggest model is the variety of the installation platforms that we’re going on. Everyone always says during the sales process, “We’re normal. It’s an easy network layout.”

Joe B: I like to say, every enterprise is their own little Galapagos Islands.

Mitchell: But they think it’s normal, it’s like “Our disks are reliable, everything’s good. It’s easy. It’s just Linux,” or something. And it’s almost never the case. There’s almost always something that’s weird, and that’s been hard. Then the product development gets affected quite a bit too, because on premise customers don’t want to update.

Updating is a thing they have to plan for and do it, and if something’s working why change it if it’s working for them? It’s really hard to get them to update, so you want to move in a more agile fashion.

You want to ship features every week or something, and then on the flipside you have paying customers being like “Please don’t ship any features. Please stop.” It’s funny because it’s counterintuitive, but our contracts have clauses that say you not only lose support but you lose your license to the software if you don’t stay within a certain release period. The support period.

Joe R: So what’s that, when you’re thinking about long term support. What’s your LTS term typically?

Mitchell: Right now it’s very short, and that’s because we say no. A good tip to you is just try saying no. We’ll get customers that come to us and say, “We want three years LTS support.” “No.” “OK.” That’s the end of that. Our current LTS is one year, which is really short. That’s not going to last that much longer, but it’s really important early on in a startup’s life to try to bring that LTS down as short as possible. Because we’ve had some features we’ve deprecated, and the fact that you can’t code remove a feature for what turns out to be effectively two years, is really painful.

Edith: I have all the scar tissue of working as an engineering manager in the 2000s, and it was this immense tax on every release. It was a tax in multiple ways. We had to do a supported platform matrix, of just, what databases and app servers can you install this on? What crazy Sybase version is this one big customer running? That’s its own tax.

Joe R: Kerberos was mentioned earlier.

Edith: Then the other tax of just like, you start to accrue I’d even say technical debt of, “Does this old feature work with the new feature?” You want to do this cool new thing that your new customers are asking for, and there’s one customer desperately clinging to this thing, then you’re like “Please God, no.”

Mitchell: That realizes itself as real cost to the vendor, because you usually have to maintain that test matrix internally. We’re a Cloud-ish company but we have in our office, we have physical racks of servers with clunky pieces of hardware that our customers sent to us just to do validation.

Edith: It just adds up to so much tax on every release that you’re like, “Is it really that hard to test three database versions versus two app servers?” And it’s like, “Yes. Yes it is.”

Joe B: One of the things that– 100% of our revenue is what we call behind the firewall, either about half AWS half on-prem. One of the arguments we use with customers to stay up to date, because we support N-1 with Kubernetes right now. That’s a six month window.

We really want folks to get on that train so that they get to benefit from the larger ecosystem. It depends on what you’re shipping, but if there’s a set of tools and if there’s other things that they want to integrate, if they get too far behind then they get stranded.

That’s one of the reasons why they’re looking to move to Kubernetes, is so they’re getting to take advantage of that ecosystem. We really help them help themselves in that way.

Edith: The getting behind is so dangerous, because what you said about people who are on an old version. It gets more and more risky when they try to upgrade.

Mitchell: One of the challenges is because we’re an open source company, we can’t control the non-customers. We’ll get someone who now wants to become a customer, and we realize they’re on a three year old version. It forces us to have those upgrade paths ready, which sucks. There’s no nice way about it.

Joe R: It definitely seems to be a range of preferences across different– And I’m curious, Joe, in your conversations have you found that verticals impact more than others? I’d assume that healthcare or fintech, we talk at Heavybit a lot about what we call a hipster enterprise for our enterprise facing companies. Selling to Airbnb or Pinterest or Lyft is different than selling to Bank of America.

Joe B: We concentrate on big companies that are not tech forward West Coast companies, because we want to train ourselves on how to deliver to mainstream enterprises. They still have a lot, even Cloud is a force of nature, it’s going to happen. SaaS, whatever, it’s going to happen. But it’s going to take longer than a lot of folks in this world, on this this side of the country, tend to think.

You have to be flexible about meeting customers where they are, helping them along with that journey, and it’s painful.

We have a support matrix and it’s not fun. But it’s something we are also building out on complements. I view whether we do that as SaaS or on-prem, the proprietary complements to Kubernetes, that’s a negotiation with your customers. What you want to do is say, “We’re going to do SaaS, we’re going to ship every week. Suck it up.” What they want is something that is going to have all the latest features and never change, and they’ll never have to upgrade. It’s physically impossible. It’s a negotiation and it’s changing over time as people get more and more comfortable with this. We’re definitely seeing that shift.

Joe R: Edith, on the SaaS side. Someone mentioned something I did want to ask about, but in the last panel, single tenant instances. Which is the spiritual descendant of the ASPs, except the vendors are agreeing to do this. With LaunchDarkly being SaaS only, do you have any of these in the wild?

Edith: Yes. To continue on what Joe Beda was saying, we have a double Joe, I don’t know if anybody has noticed this. A lot of our customers are really large enterprises that are coming to us, and one of my favorite quotes I heard yesterday is they’re coming to us for the harmony between agility and stability. So, to your point–

Joe B: You should tweet that. I’m sure you already have.

Edith: They’re coming to us because they say, “We’re getting left behind by all the hipster companies. We don’t want to be the stodgy hotel chain where Airbnb is worth billions of dollars. We don’t want to be the stodgy bank where PayPal is worth billions of dollars.” They know that they have to move faster.

It’s not 2004 anymore where everybody sells software as a cost center to cut.

Usually they’re coming to us when they’re somewhere in a transformation from, “We’re moving from our data centers, we’re moving to Cloud, we need to move faster but we still need to do it stably.” They’re looking to a vendor like LaunchDarkly to help them with it.

Where single instances come into this, is sometimes we do have extremely large customers who say I really need a single instance. Usually the reasons are pretty much exactly what Marianna gave. Regulatory compliance, a lot of PII. Really spiky workloads that are unpredictable and they want their own instance. Our usual first comeback is we say, “We serve 40 billion flags a day. That’s billion with a ‘B.’ Everybody likes to think they’re huge, but we’re actually serving customers like fuboTV which is the number one sports app. We’re powering Atlassian, we power Hashi, we can handle your workload. They continue and say, “We really want this private instance.” We usually quote them the price and their eyes go, “You know what? SaaS is just dandy.”

Joe R: Yeah, “That sounds pretty good now.” One thing that’s interesting when people talk about on premise in 2018, and Mitchell I’d be interested in how often does on premise mean what you would think of a traditional air gap? I mean, no network connections, or a heavily firewalled data center? Or actually just, “We have our own VPC on Amazon. That’s on premise.”

Mitchell: For us it’s pretty split. I would say we still do quite a lot of traditional on-prem, physical on-prem, but I would say that’s skewed with our company a bit because we sell a lot of security software with Vault. The way a lot of people are deploying Vault is they’re in the Cloud or they’re on Kubernetes or something, but they still want their primary Vault cluster. One of the enterprise features they are buying from us with Vault is multi region. But they want the primary Vault cluster in a physical, not air gapped usually, but in a physical location. So, that breaks down but it’s just a matter of time before it all moves to private VPCs. We’ve designed everything to assume VPC. If you do that, it scales down just fine. Joe R: Another thing I want to get some people’s opinions on, because I have opinions. Multi Cloud is a word that gets me going, because it is a thing, I don’t think it means what most people think it means when they say it. In terms of deployment targets, what does multi Cloud mean for you? What do you see in the wild? Joe, are you talking to people who are multi Cloud?

Joe B: The first thing to recognize is when you’re a startup your failure mode is that you run out of money, and nobody cares. When you’re a big company you have different risks that you deal with. Multi Cloud ends up being a risk mitigation type of thing. Pretty much all of our customers, they don’t want to necessarily run on other Clouds but they want to have the option to run on other Clouds.

They want to know that it’s going to be a two to three month process to migrate some stuff over, it’s going to be painful, but we can do it.

Versus “Let’s rewrite our app, it’s going to be a three year process to move stuff over.” For us, because we’re in this position where we know that the stuff that we’re building, some of it we’re going to run ourselves in a multi-tenant way. Some of the stuff that we’re going to be running behind the firewall for folks, we’re keeping our stuff as plain Jane as possible. We’re not using fancy stuff, your little VMs, run of the mill databases as much as we possibly can so that we have that flexibility.

Joe R: Got it. Mitchell, you have multi Cloud customers. What do they look like?

Mitchell: As of today, roughly 100 out of the Fortune 500 are paying customers. Out of those hundred, 99 or 100 are multi Cloud. The first thing I always say when I say multi Cloud is there’s four totally distinct definitions of multi Cloud. There’s workload portability, which is extremely rare. That means you could run your app on any Cloud, that’s almost never true. There’s workflow portability which means that the way you deploy an app to Amazon is the same tools and process for a developer as the way you deploy to Azure. That’s often true. There’s traffic portability, which means that depending on a geographic region you’re going to a different Cloud provider.

It’s close, but not the same as workload portability. That’s semi-rare but more common. And then there’s data portability, which means that your actual data can move Clouds, which is almost never because speed of light. Those are the four things. Workflow portability is what we sell as a company, and it’s common in enterprises in particular. If only the most basic reason is M&A. When a big company is acquiring another company and that company has chosen to be on Google, and your company has chosen to be on Amazon. The big company as part of the M&A process is not going to price in an infrastructure switch from Google to Amazon.

They’re going to say, “You stay on Google, we’re going to stay on Amazon, and let’s do our own thing.” You end up with business unit level splits between Clouds, so there’s no workload portability. They’re separate apps and they’re totally siloed. The other thing we see is that the Clouds are now starting to differentiate in terms of best in class ML, versus best in class serverless, versus various different high level features. Some teams even in the same view want to leverage different technologies, so we’re seeing that a lot. But for the most part it’s pretty much everywhere. Especially if you think about the on-prem as a different Cloud, more or less, if you just bucket into that.

Because the nature of enterprises is that any paradigm transformation, like mainframe to servers, servers to VMs, VMs to containers, containers to schedulers and schedulers to serverless. All these different transformations are probably 10 years, and the 10 years overlap. They’re always in a state of heterogeneity.

Even if they have these nice new teams that are in a fully Kubernetes workload, they also have these teams that are still in their last two years of the server VM switch. You either as a vendor say, “We don’t help you with that,” and you maybe lower your impact and value to the company, or you try to figure out a way to make it work.

Joe R: Got it. Thoughts?

Joe B: Yes. I think it’s longer than 10 years. Mainframes are still a growing business for IBM. They’re making a lot of money on that stuff. The other point is on the multi Cloud data, there are scenarios where big companies, I’m thinking like movie studios. They’ll backup stuff to multiple Clouds because they don’t trust those eight nines or whatever of durability that Amazon promises. Black Swan events, and such.

Joe R: Edith, feature flags, which are near and dear to my heart.

Edith: Mine too.

Joe R: Do you find, just to take an interesting spin. As a SaaS hosting the Cloud SaaS offering, do you find customers managing features in on-prem products with feature flags?

Edith: Yeah, absolutely. Maybe you should just talk about how you use us.

Mitchell: No, you go.

Edith: No, you go.

Mitchell: All right. I’ll just say that feature flags are, I mentioned that problem earlier of how our on-prem customers don’t want to be upgraded too fast. We deliver features quickly, we just turn them off.

Joe R: You just hide them. Interesting.

Mitchell: They’re getting them. They don’t see.

Joe R: They’re deployed, but they’re not released.

Mitchell: Right. And they’re not running actual runtime components.

Edith: We have a lot of customers who use us to manage their own customers. If they’ve signed a contract and this customer says, it’s common to say, “We do a lot of training on this UI. I do not want new features.” Full stop. Even because there might be something as stringent as union rules, if you’re say an airline, about how often you can change a UI. They’ll have features and then they’ll say, “We were going to do a training on this date,” and that’s the date we want to turn them on.

Joe R: They’re not up on continuous delivery.

Edith: It’s the developers who want it. The developers that these customers are champing at the bit, they’re churning out features, and then what they really love about feature flags is that the developers can move super quickly and then use the feature flags to match with the rest of the work. For example, they can build a feature and then plan all the training.

Joe R: The other interesting thing is, and this overlaps with what the last panel was talking about with SLAs.

You can use feature flags to implement a lot of those premium support offerings, or UIs that you give people. They opt into that platinum plan and you toggle the flag on their account, and they get all the goodies.

Mitchell: A fun one of enterprise in particular is when you realize the world isn’t super flat, and there is really weird laws in different countries. We don’t use LaunchDarkly for this but we use feature flags for example, and when we sell Vault in China we have to disable a bunch of the encryption algorithms. They don’t allow them by law there, and it’s the same thing right? You have to have all these feature configurables that different regions can do different things with your software.

Edith: Another use case like that is some customers will say, “I’m a bank. Do not give me this feature until it has been battle hardened with 98% of your other customers.”

Joe R: Switching gears a little, something I definitely want make sure we talk about today that’s important for enterprise sales is channels. There’s a bunch of different kinds of channels. We could start with Mitchell, you mentioned MSP. First of all, how many people here actually know what an MSP is? OK. That’s all right. You should still define it and how that works in HashiCorp’s strategy for delivery.

Mitchell: Sure. When you get past the hipster enterprises, the hipster enterprises are like the Airbnbs, Netflix, those of the world. They want to run it themselves. Then you get more to the billion dollar beauty product business in the middle of Texas type enterprise, they want it on-prem, but they super don’t want to run it themselves and they don’t want a SaaS. They want something in the middle, and an MSP is a managed service provider, which is a business dedicated to running other people’s software for you on other people’s premises. That’s what MSP is.

Joe R: It has a channel, because there are tens of thousands of these. Then if they get comfortable with your solution they can effectively act as resellers.

Mitchell: Yes, definitely.

Joe R: Or at least you can draft off, they’ll have existing procurement in place.

Mitchell: There’s pros and cons, which is that as a vendor you bait and you really want to control the paper and control the relationship with the customer. Some MSPs are resellers, some are not resellers. Some you could bring in after the fact of your own paper, and they have their own separate service agreement, but there’s pros and cons there.

Joe R: Another, not variant, but Joe I’m curious if the people you’re working with use VARs or value added resellers?

Joe B: We’re not there yet. We still want to control that experience. There’s still a lot of dealing with those Galapagos Islands out there. But as your deployment gets more turnkey, as you understand the patterns, it makes sense to be able to bring in VARs or SIs and be able to have them extend your reach.

Joe R: Generally they tend to add, particularly when we’re going back to verticals, they will specialize in a vertical.

Joe B: They speak the lingo.

Joe R: They speak the lingo. They know the things that matter to the buyers in that vertical.

Mitchell: They’re usually highly specialized. It’ll be like, “AWS only, with security only and finance only.” That’s how big that market is. They could do that and still be a billion dollar business.

Joe R: Some of the big ones, I know some companies have open budget with it. So when you’re talking about, we haven’t even talked about budgeting cycles because this isn’t a procurement panel. But they’ll have actual budget set aside that you don’t have to go through budgeting, you can just go straight into that.

Joe B: We’ve done that. Where it’s like, “We want to get you through procurement but it takes forever. Just give 5% to these folks and make it happen.”.

Joe R: And all the problems go away. It’s not racketeering at all.

Edith: We’ve definitely had, we don’t discount but we’ve had people who say– It was just a large sporting goods company that said, “We want to buy you. We’re not set up to purchase from you. Can you sell to this person who will mark it up two times?” We’re like, “OK.”

Joe R: Other channels go to the way other end of the spectrum. There’s the Heroku add ons marketplace. Amazon has, although it’s been around for a while, especially in the last year or two they’re clearly starting to invest in their marketplace. Are you seeing those tend to be more in the self-serve side? Or bottoms up inside, are you seeing any enterprise adoption from large Amazon accounts going to the marketplace?

Mitchell: It varies by Cloud for us. I don’t want to call anyone out, so I’ll say I just won’t name them, but there’s some Cloud marketplaces that are really useless. They’re demo only, self-serve type things. There’s other Cloud marketplaces where they do a better job of integrating it into how you could better control the control panel flow, and the billing style. And BYOL, Bring Your Own License for enterprise software, that we’ve seen a little better adoption but it’s not as strong as direct sales.

Marketplaces are rough because you have that middleman between you and the Cloud.

You want to have that direct relationship with the customer because, as the previous panel talked about, when you have TAMs and various SLAs and so on, you really want that direct connection.

Edith: It depends on who your buyer is. If you’re selling more, or if your buyers are VP engineering they don’t want to just do a point clickwrap. They want a little bit more white glove service. They want some demos, they want some help.

Joe B: I would say in large part I view marketplaces as vendor introduction services. It’s a good way for them to see what’s there and then they’ll short circuit the marketplace to do the deal. There’s other programs the Clouds offer if you can help bring somebody on, and that drives more spend on their platform. They’ll give you a cut of that. There’s these channel programs that the various Clouds offer because their goal is to land grab as many cycles and bytes and packets as possible. If you can help do that, they’ll cut you in on some of the profits there. Those can be really good channels.

Mitchell: That’s a really cool technical thing too. It’s a lot of people on Cloud will firewall out and you can’t phone home and know if someone’s running your software. We have that problem too. We will do a phone home just to track the number of runs, it’s totally anonymous, but there’ll be firewalls.

If you can get partnerships with a lot of these Cloud platforms, and we have them with all of them, they all have local hypervisor servers running and you can set it up so you could ping that. It’s really hard to block that, and the Cloud provider will tell us how many are running. It’s a lot.

We could go to them and they say, “You ran on 200,000 cores this month.” Even if the network is off you can know that.

Joe R: That’s interesting. It sounds like you definitely need to do your diligence on the different programs. I think we’re pretty much at time, so I just want to do a quick fire. We’ll go right down. One piece of advice for people who are thinking about deploying software in the enterprise. We’ll start with you, Joe.

Joe B: I would say meet customers where they are. Don’t write letters from the future going, “Here’s this beautiful thing.” Instead, recognize that you’re going to be more successful if you bring folks with you.

Edith: We were unexpectedly enterprise. We thought we were going to be a bottoms up sale, and one of our first big sales was five figures. It was like this “Aha!” of we’re solving a big company problem. We need to grow up very quickly. So, realize who your buyers are, and I’ll echo what Joe said. If you’re getting pulled up market, go there.

Mitchell: My advice would be that a lot of the problems with selling to enterprise are non-technical. Focus on customer success, TAMs, solutions engineer, sales engineers. Because they’re going to be working realistically for four to six months before any bits hit the server.

Joe R: Thank you.

Joe B: Thank you for having us.