Ten lessons from twelve years of AWS

Recording and redacted transcript from my keynote at the AWS Community Day Australia and New Zealand.

First of all, I would like to thank everyone in the AWS Community in Australia and New Zealand and the AWS Heroes who have helped put this event together.

In particular, Augustino (aka Gus), Peter, and Nathan , the AWS Heroes, and John our OBS Ninja. All have done an amazing work building this Community.

Thank you!

A little reminder that if you plan to share your day with others on social media, please use the hashtag:

#AWSCommunityDayANZ

A few years ago, I did a talk called ten lessons from ten years on AWS.

That was at the Community Day in Bangalore, India. Back then, there wasn’t any COVID virus, so I traveled there — and I loved it.

I wish I could be on site with you today, but instead, I am virtual with you this early morning, live from Finland.

It is 4 am here, so bear with me if I am a little slow :)

So, I did that talk two years ago, but it feels like it was already ten years ago.

J. R . Rim said

“In the information age, one tech year is equivalent to one person’s lifetime.”

While that is true, I recently found myself highly influenced by a dog.

Her name is Hilma , and as you can see, she likes kissing.

And so, I now see tech years like dog years, with a ratio of approximately 1 to 7.

I am trying to say that time flies — fast — and for today, I thought it would be interesting to look back at the past again, and see what went well, and what didn’t.

So, as I said, I did a similar talk two and half years ago, and looking at it now feels like I need to make some corrections.

That is the summary slide of it.

Do you notice the problem?

The biggest mistake I did back then was to focus mostly on technology.

Not that what is on that slide is wrong, quite the opposite, but these lessons don’t all qualify as life lessons — lessons that make you grow as a person.

So today, I will try to do a better job and share some of the lessons that changed me as a person.

#1 — It’s the customers and the information you give them that’s important, not the technology.

WAP, ASP, Instant messaging, Grid computing, Portals, Biometrics, AR, VR, Web2.0, P2P, IoT, 4G, 5G, InternetTV, DVB, NFC, BigData, wearable, NLP, ML, Autonomous Vehicle, Connected homes, Deep learning, Neural network, Deep learning, digital twin, crypto, serverless, k8s, blockchain, …

These are all examples of hypes I have lived in since 2000.

While hypes come and go, the bigger picture remains the same.

More important than the technology itself are the customers, especially how good an experience you give them.

Technology is irrelevant if your customers aren’t happy!

Whether you use Ruby, Python, Node, Haskell, or Clojure, K8S or serverless, a monolith or micro-services, customers’ happiness is making the final call.

I have seen fantastic customer experience, and genuinely successful businesses run on less than polished software systems.

And honestly, technology is moving too fast always to make sound technological decisions.

How many front-end frameworks will be released during this talk? :)

Most of the time, successful teams guess and take a shot at it. What they do differently is being fast at adjusting the course.

And to me, this has become the best measure for success:

“How fast can you correct course once you have customer feedback.”

Notice that first, you need to have that feedback, and only then can you correct course. Otherwise, you go blind.

So, focus on that feedback loop. And it means a few things:

• Listen to your customers.
• Forget your ego.
• Invest in automation — early! (CI/CD).
• Repeat.

Most of what we build in the cloud is organized around Data, with a big D.

• We store Data.
• We Process it.
• And we move it around.

Easy right? :)

How fast you process it and move it around influences the price and the accuracy of the information you extract from Data.

The rest is mostly operations. How much time you spend on operations influences the price, but most importantly, the amount of time you spend listening and adjusting course for your customers — so choose wisely.

#2 — You can’t know everything.

Technology changes too fast to keep track of all of it while maintaining a balanced life.

Trying to keep up with technological changes is like trying to make sense of AWS services naming convention — it’s impossible :)

As a developer advocate for AWS, I often get asked

“How do you keep up with everything?”

Here is a news for you — I actually don’t :)

Everyone is struggling to keep up, worried about being left behind the waves of technological evolution.

Even people who seem to have it all together. They are more like you and me than you would believe.

There is a weird paradox to knowledge:

“the more you know, the more you realize you don’t know.”

And of course, we feel bad about it. We feel like imposters — I certainly felt and continue to feel like one.

To me, the imposters’ syndrome was and continues to be one of the hardest things to deal with, and I am 40yo white male, so imagine what it must be for women in tech or other minorities.

Communities like this have helped me a lot overcome my fears a lot — Communities where I could discuss and practice, often without being judged.

Another weird paradox exists:

“As you become more senior, you know less and less about new technologies.”

Yet, the most inspiring engineering reviews I have participated in while working at Amazon were with very senior engineers — the senior principals. There are about 70 or so of them at Amazon.

What’s their secret?

• They listen.
• They ask questions — a lot of them.
• And, they challenge ideas and biases.

I will come back to biases later — but do you notice something?

These are all people skills — they listen, are curious, and they dive deep.

Seniors engineers are more people engineers rather than technology engineers, which leads me to lesson three.

#3 — Invest in people skills

… as much as you invest time learning about the latest NodeJS framework, k8s or serverless.

No one’s technical skill is irreplaceable. I learned it the hard way.

One day, after three years of working heart and soul for a startup which I was number two employee, I was fired.

I arrived one morning at work with a smile on my face. A coffee later, I was walking out of the door, with all my books, but without a job.

And it was terrifying.

Sure, we had financial difficulties, but I had built everything from scratch. I knew every line of code, and I never thought that I would be the first to go.

But I was, and my technical skills that I thought were irreplaceable, were in fact, fast covered for by the new, very good and cheaper work force just hired out from college.

The only option you have to stand out and stay relevant is to invest in people skills.

And I get it, for many of us it is difficult, even more now — but it is critical.

There are many skills necessary, of course — but one stands out: Empathy.

Bill Billard correctly said:

“Opinion is really the lowest form of human knowledge. It requires no accountability, no understanding. The highest form of knowledge … is empathy, for it requires us to suspend our egos and live in another’s world. It requires profound purpose larger than the self kind of understanding.”

When I got fired over a coffee cup, I had a lot of opinions. I was a lead developer, so I had a lot of empathy for the team of developers I managed but a lot of opinions towards the management team.

Opinions that cost me my job.

While I understood our developers, I failed to communicate upward and failed to understand the different business needs.

Like most things in life, you have to find the right balance to be successful.

But the more people skills you have, the easier it will be.

#4 — Embrace failure.

I got that one right the first time! :)

While we have learned that almost everything will work again if you reboot it, including you, sometimes, things just fail.

Ask Murphy about it!

Brian Tracy, beautifully, said:

“It is not failure itself that holds you back; it is the fear of failure that paralyzes you.”

Let me start by saying that scared developers won’t:

• Try things out.
• Won’t innovate as fast as your business would need to.
• Won’t dare to jump in and fix things when (pardon my French) shit hits the fan.
• Won’t do more than ask for.
• And, won’t stay long in the job.

I know!! I was one of them.

Failure should not be seen as “you are a failure” but merely as moving along the path of experimentation.

If you don’t fail, you are probably not trying things hard enough nor pushing the limits of the “adjacent possible.”

Innovations flourish in a community where ideas are exchanged, discussed, tried, and improved over time — a community like this one.

But most of all, innovations flourish in an environment that embraces failure.

Remember, Thomas Edison tested more than 6000 different materials before settling on a light bulb using carbonized bamboo.

Failure tends to teach you lessons that reading books or blog posts can’t teach you.

The most successful teams I have worked with are those I failed the hardest with, but we were prepared to fail, and especially we were not afraid of losing our jobs.

The best way to learn from failure is to practice failure — you know I had to mention chaos engineering at least once in this talk :)

You also have to work on minimizing the blast radius of failures. Techniques like bulkheads, sharding, isolation, load shedding, graceful degradation, immutability, etc., will come handy — so you probably should get familiar with them.

But again, technology itself won’t make your system more robust; people will — which brings me to point five.

#5 — Don’t blame people when things fail.

Sidney Dekker, one of the most critical writer in the field of safety engineering, rightly said:

“The question […] is not who is responsible for failure; rather, it asks what is responsible for things going wrong. What is the set of engineered and organized circumstances that is responsible for putting people in a position where they end up doing things that go wrong?”

Everyone screws up, and one day, so will you!

I screwed up several times — big time!!

I deleted databases in production — twice — while trying to recover from an outage. But that is the topic for another talk :)

Do you think I woke up that morning thinking:

“Today, I will come to work, delete some databases, and do a shitty job!”

Of course not — Everyone has good intentions, even when we screw up.

So don’t blame individuals or teams. Similarly, don’t assign or imply blame to others, individuals, groups, or organizations. Instead, identify what happened and question why those things happened.

Stopping at people’s errors isn’t right. It is a sign that you haven’t gone deep enough.

Think about the situation that led the operator to trigger the event? Why was the operator able to do such a thing? Was it a lack of proper tools, a problem in the culture, or a missing process

[…]

Continuing on databases —

# 6 — Route53 is NOT a database.

… regardless of what QuinnyPig says on Twitter or Reddit.

Nor are tags :)

#7 — Watch out for heuristics and biases!

Being aware of cognitive biases when performing your job is a superpower that will often make the difference between becoming an inspiring leader or not.

Cognitive biases impact our perception of reality, driving us into making incorrect conclusions and often, irrational decisions.

While I don’t think removing biases is possible, it definitely will help you if you can identify them and properly adjust your perception and question your assumptions.

Some of the biases and heuristics to watch out for:

• The confirmation bias — “the tendency to search for, interpret, favor, and recall information that confirms or supports one’s prior personal beliefs or values.”
• The sunk cost fallacy   — “the tendency for people to believe that investments (i.e., sunk costs) justify further expenditures.”
• The common belief fallacy   — “If many believe so, it is so.” you know, blockchain …
• The hindsight bias   — “the tendency for people to perceive events that have already occurred as having been more predictable than they were before the events took place.”
• The fundamental attribution error   — “the tendency to believe that what people do reflects who they are.”

And, there is more!

• The optimistic bias
• The overconfidence effect
• Wishful thinking
• The anchoring bias
• The bandwagon effect
• The cargo cult

You get the idea — we are full of imperfections :)

So, spend some time reading about it — it will make you a better developer, a more understanding manager, and eventually an inspiring leader.

It will make you a better person overall.

#8 — Writing.

I started writing my blog on Jan 9, 2018 — after I gave the first version of this talk in India — two and a half years ago.

Julien Simon, the ML principal developer advocate, tried to convince me for a year before I wrote my first words down.

I was worried I had nothing to say. So afraid that I deleted words as fast as I was writing them down.

You know, the fear of failing does paralyze you.

But eventually, I took Julien’s advice, published my first blog post, and the second, the third, etc.

Today, and with approximately 60 blog posts published and nearly 300 thousand visitors a year, I can easily say that writing that first blog post was one of the most important thing I did in my career.

One crucial thing l learned along the way:

Every writer you know writes terrible first drafts. And second drafts. And third.

Every. Single. Writer.

Eventually, after tens of iterations, words start to make sense.

But it takes time! So don’t give up.

The funny thing is that I hated writing most of my school life, and long after.

It still takes me an absurd amount of time to put my thoughts into words. But in the process, it also clarifies them, put them in order, crystallizes them.

The key to writing is to start, and then do it word by word.

If you are like me, you are probably often wondering what to write about.

One of my favorite writer — Anne Lamott- said:

“Remember that every single thing that happened to you is yours, and you get to tell it.”

And that is so true.

What problem did you solve lately? Why? How?

What happened to you a few days, months, or years ago, is probably happening to someone today.

So, tell your stories, for only you, with your own words, can tell it!

My favorite part of writing is the review process.

As soon as I have some sort of draft, I select several victims in my team or broader network, and I ask them to review it — to challenge my ideas, my opinions, and surface biases.

The review process was initially very challenging, as I had to deal with critics.

You know, the same critics when a developer gets her or his code reviewed in a pull request.

“How dare you criticize my baby?”

Getting critics is humbling, very humbling — and for the better — It forces you to listen, learn, and eventually improve.

Of course, you need to find competent reviewers, reviewers that care about your success. Reviewers that aren’t scared of telling you the truth.

So, make sure you have some of them in your network of influence.

And, if one day you decide to start writing, I would happily become one of them.

#9 — Mentoring.

Throughout my career, I’ve had the chance to have amazing mentors.

Whether in my work or outside of my career aspirations, I have had few key people who have helped me move forward.

Sometimes, this is as simple as bouncing my ideas back-and-forth to see a clearer picture.

Other times, it’s getting encouragement, a supportive tap on the shoulder, and advice on what to do next.

A good mentor can help you be your best self.

Everyone needs mentors. All my mentors have mentors themselves.

Mentoring others has been really important to me, especially in the past few years.

Most people enter their professional lives with little understanding of the complex landscape and expectations for excellence required for a successful career.

I certainly had no idea what I was doing when I started!

It is is not a problem per-se, for it is the normal unrolling of life, but it is scary at times, and a rather excellent opportunity for mentoring.

Mentoring is essential, not only because of the knowledge and skills one can learn from mentors but also because mentoring provides professional and personal support that facilitates success.

Research shows that people with good mentors have a higher chance of success and more significant career advancement potential.

So, open yourself for mentoring others and look out for your own, personal mentor as well.

#10 — Learn from others.

Learning from others is the single most important thing I have learned. And I have to admit, sometimes I still have to remind myself to shut up and listen.

There is always someone in the room that knows more than you do. That person is just not necessarily broadcasting it.

Be open and ready to be challenged and change opinion.

Share your ideas with others, challenge them and especially let others do so.

Fortunately, there are myriads ways to do this — participating in this community is, of course, one of them.

A few weeks ago, while preparing for this talk, I asked the developer advocate team to share with me their lessons — and few of them kindly answered.

But as I didn’t really know what I was going to talk about a few weeks ago, I asked my question with a heavy bias towards getting technical answers — so despite my first lesson not to focus on technology, the following is mainly about technology — but I am the one to blame for that :)

Javier Ramirez :

Don’t use the AWS as a traditional Data Center. Have a consistent naming/tagging strategy early on. Especially for everything that has unique names (s3 buckets, for example).

Alex Casalboni:

Learn IAM before you do anything serious. Master Infrastructure as Code (IaC), either CloudFormation or Terraform. Use the management console only to build prototypes, or the first time you try out a new service, switch to IaC for anything else.

Boaz Ziniman:

Account security — don’t use the root account. Always enable MFA. Set IAM users for every developer, with different roles. Tag everything!

Enrique Duvos:

Security, Security, Security.

Dennis Traub:

Turn on CloudTrail. Turn on Guard Duty. Don’t assume that development teams will consider security when building on AWS. So, I’m with Enrique here —think security.

Marcia Villalba:

Try to see if there is a managed service before building it yourself.

Danilo Poccia:

Adopt the right mindset: With on-prem virtualization and hosting, you have a finite set of resources where you try to squeeze as many things as possible. With the cloud, you have access to a virtually unlimited set of resources, and you should use the minimum you need at any point in time.

Sebastien Stormacq:

Use EC2 only if you have exhausted all other possibilities. This is not because of EC2, but because no machine to manage is better than one machine to manage. So, go serverless as much as possible. And serverless is not only Lambda, but it is also RDS, Cognito, S3, API Gateway, etc.

Ricardo Sueiras:

Shift the conversation around AWS from technology to business outcomes (Agility, etc.). It will help you get the exec sponsor/support required for success. From a technical point of view, bet on automation. Resist being obsessed with the console.

Isabel Huerga Ayza:

Governance — everything that is not clearly defined will be done by no one. Or best case it will be, but not consistently. Goes to accountability and ownership, which is not good to leave to good faith when what is at risk is your business. Don’t wait until a production disruption to enable support. Setup budget alerts!

Steven Bryen:

My advice is to think differently about things like the dynamic allocation of resources. An example would be security groups. You can create a rule referencing another security group, which means new instances automatically match the rule and have access as they scale. It is very different from the traditional on-prem mindset but is so valuable.

Cobus Bernard:

Read the pricing page for a service and set up billing alerts.

Feel free to connect with them and challenge them on their ideas. I am sure they would love that!

That’s it for now, folks! Again, thanks for inviting me and have a lovely rest of the day.

-Adrian

https://medium.com/media/4971e7d7ca309f5a367c1cfbbd5651ee/href