Cloud on Andrew Khoury

AWS 2021 Highlights

Fri, 27 Aug 2021 14:37:40 +0000

AWS updates their services so quickly they literally have thousands of updates each year (1,284 the last time I checked): https://aws.amazon.com/about-aws/whats-new/2021/

This blog will highlight some of my favorite AWS updates for 2021.

AWS Network Firewall

https://aws.amazon.com/network-firewall/ — A managed service by AWS that allows fine-grained control over network traffic.

Before Network Firewall was available, customers were left to manage their own squid proxy or similar service if they wanted fine-grained control over their traffic in the Cloud (like many Enterprise customers do). This meant ensuring their service met security and compliance requirements, traffic and scaling demands, and uptime SLA’s, all while adding to the burden for operational teams. Some routed traffic back to on-premise solutions, but this wasn’t always a viable solution for all customers.

AWS Network Firewall became available in Sydney back in January: https://aws.amazon.com/about-aws/whats-new/2021/01/aws-network-firewall-is-now-available-in-the-asia-pacific-sydney-region/

It received a few updates and deployments to other regions. In April, they rolled out to 10 more regions. By June, it was available as part of AWS GovCloud (US), and by the end of July, it was PCI DSS Compliant.

Amazon EBS io2 Block Express Volumes

If you have the need for speed (that is, I/O speed), then this is the update for you. In July, AWS launched a special kind of volume that is perfect for I/O intensive operations. This isn’t intended for your typical web server with a few hundred concurrent connections, but plenty of Enterprise clients may find themselves needing a little bit of extra grunt.

https://aws.amazon.com/about-aws/whats-new/2021/07/aws-announces-general-availability-amazon-ebs-block-express-volumes/

up to 4x higher throughput, IOPS, and capacity than io2 volumes
designed to deliver sub-millisecond latency
99.999% durability
Supports Multi-Attach and Elastic Volumes

Customers can now provision a single io2 volume with up to 256,000 IOPS, 4000 MB/s of throughput, and a storage capacity of 64 TiB.

PrivateLink for Amazon S3

As of February, Amazon S3 supported AWS PrivateLink, providing direct access to S3 via a private endpoint within your virtual private network.

https://aws.amazon.com/about-aws/whats-new/2021/02/amazon-s3-now-supports-aws-privatelink/

Simplify your network architecture by connecting to S3 from on-premises
In AWS, use private IP addresses in your Virtual Private Cloud (VPC) to connect to S3

This eliminates the need to use public IPs, configure firewall rules, or configure an Internet Gateway to connect to S3.

As of July, AWS also lowered the data processing cost for PrivateLink (phew): https://aws.amazon.com/about-aws/whats-new/2021/07/aws-lowers-data-processing-charges-aws-privatelink/.

Summary

Each of these updates has been something I’ve personally been waiting for or had to work around in the past. They also represent a nice cross-section of services that you’re likely to encounter if you dive deep enough into AWS.

AWS continues to roll out updates across its services which reduces the effort required to enter the Cloud. They continue to deliver services that cater to the speed and scale we could only have dreamt of 15 years ago (oh and Happy Birthday AWS/EC2).

Info

Also posted on medium as AWS 2021 Highlights.

How Cloud Transformation at Scale can enable Good Software Delivery

Wed, 04 Aug 2021 22:50:33 +0000

Why should you transform?

Each person, team and organization is going to be on their own journey to cloud, writing good software and attempting to delight customers, or increase profitability. Finding your why can take some time and deep thinking (and deserves a whole blog to itself).

For now here are some common why’s for transformation and good software delivery that might also apply to you:

To enable product (development) teams to ship software with less dependencies on other teams/humans
To speed up delivery
Allow the business to scale products quickly
To build new products quickly
To support the business in getting fast feedback through frequent delivery
To achieve consistency in infrastructure and software deployment, so we can reduce the risk of issues due to manual changes
To reduce the manual but repeatable steps close to production (security and compliance checks) that often slow down or delay releases

A picture

If a picture tells a thousands words, then I present to you Cloud Transformation at Scale:

Cloud Landing Zone

In purple we have the “Cloud Landing Zone” which is the starting point for key cross-cutting concerns that span the entire organization:

Automated Account-Provisioning for shared Cloud Accounts
Account Guardrails
Security & Logging Accounts (SIEM & IAM)
Networking Foundations
Shared-Services Account

This is the foundation of most modernization efforts that depend on the Cloud. Automation of Accounts & Infrastructure unlocks the ability to “move fast” in a way that traditional on-prem data center or operation teams aren’t able to match.

Most organization on their cloud journey are building or buying some variation of this or started off with some manual steps and a few scripts, and are maturing their operations to look at lot like this as they scale.

Services for Developers

In red we have the next set of shared services focused on supporting product (and thus development) teams. This includes:

Automated Infrastructure-Provisioning for Shared Services (think shared k8s clusters)
Common Shared Services (build agents, artifact storage etc)
Self-Service Cloud Accounts for Product teams (self-service being the key)

It’s important that developers can manage their own accounts via a self-service mechanism (if you’re an organization with more than 10 developers). This means automated pipelines for infrastructure and account provisioning that have a well thought out developer-experience that make it easy for them to manage their own accounts, securely (ie with Account Guardrails built into the pipelines and Cloud Accounts), and sans human intervention.

Product Teams — Trust and Verify

In blue we have application and app-related infrastructure pipeline(s). Each commit should result in a single artifact built and tested then promoted into production if it passes the automated tests.

Developers should have access to reference pipelines that show exactly how to pass all the organizational, security and compliance requirements (which should all be automated within the pipeline). It should be easy to start a new project that can quickly pass through the pipeline into production.

Developers should have the freedom (and trust) to do whatever they need to do before they reach production. They should have access to every automated security and compliance check that will be run in production so they can quickly assess how ready they are for production on the very first commit.

The organization should automatically verify every production deployment with the same rules they supplied the developers on day-one, commit-one. No surprises, no extra work to do on production-day.

This seems like a simple concept, and the name I give it is “Good Software Delivery” because it’s simply how we all should be delivering the software we want to ship.

So how does one transform? In practice it’s a journey that individuals, teams and organizations need to be prepared to go down. And that’s a completely different rabbit hole I welcome you to explore: https://faun.pub/one-devops-please-part-1-df7a2787fde8

Appendix

We could go deeper into each area, but here’s a quick teaser that goes into more detail, in particular for the blue Product Team’s Pipeline(s):

Info

Also posted on medium as How Cloud Transformation at Scale can enable Good Software Delivery.

Why Business Value eats DevOps for breakfast

Sat, 11 Jan 2020 22:18:26 +0000

It’s 2020 and my new years resolution is to keep a list of words that deserve to be on the naughty list:

DevOps
Agile
Requirements

It’s not that these words are inherently bad — but my industry has twisted, abused and polluted them leaving behind a trail of over-engineered solutions and misguided teams.

For an in-depth look at “how to do the DevOps” see my post One DevOps Please which talks about how to do transformation and discusses the unicorns that are “DevOps teams”.

First, a little warm up…

I’d like to interrupt this blog to do a warm up exercise. Sorry-not-sorry for anyone I’m about to offend.

Repeat after me:

I am not a DevOps Engineer

Let’s try another one:

I don’t belong to a DevOps team, because I only work with things that are real

One more:

Doing standups every day at 9am doesn’t make me agile

You say these every day along with burpees, planking or whatever your daily routine is. I promise you they’re good for your health.

Why — should I care?

You might have heard the following:

We’ve always done it this way

If “why” is the super hero of this story, then this phrase is the villain.

Like many youngsters I spent most of my childhood asking “why”. Ironically I’m sure my parents asked themselves “why” they had to have one more kid as I pestered them with endless questions about science, the universe, or whatever else was on my mind.

My questions didn’t stop at childhood. I’ve always had an obsessively curious mind and a determination to understand everything inside-out. I did it with helpdesk support, then when I stepped into development, and again when I learnt about cloud and automation.

Helpdesk me:

Why do clients always call with the same MSSQL replication errors

Developer me:

Why does my code work differently in production

“DevOps” me:

Why doesn’t this DevOps thing solve my client’s problems

An important part of what I do today is about helping foster the same kind of curiosity in others, and helping hone in the skills to make the best use of that curiousity.

Examples of times you should ask — Why

Hint: If you’re not sure — always ask why.

You’ve been invited to a meeting, with no description or context, and no idea what the outcome is.

You’re creating a presentation as a team but you haven’t agreed on the audience, the intention or the outcome.

You’ve been told to create a kubernetes cluster.

You’ve been told stand-ups happen at 9am (and they happen to go for 45 minutes, nobody seems to have an understand of what to do for the rest of the day even after the meeting).

You’ve been asked to join a new project, team or a recruiter has told you they think you’re “perfect for Job X”.

Your company has decided they’re moving to the Cloud.

Why we ask why

Okay so that heading was very meta. The intention isn’t to challenge or be confrontational. It’s not a power play or a fun party trick.

Asking why is about having a shared understanding.

In each of the previous scenarios and indeed in many interactions there’s always an underlying intention. Sometimes it’s just about communicating with each other to gain that shared understanding.

Warning: Here be dragons. Sometimes (e.g when people aren’t sure why) asking them can lead to defensiveness and well, make them angry. Other’s don’t like being interrupted or challenged. Some people have values and principals that favor hierarchy over creativity and innovation. Asking why isn’t always a happy path to making friends.

What does all this have to do with Business Value?

Business Value should be well known and clearly articulated in every team of your organization. Everyone should know the why (and how what they’re doing ties back to a very specific business value). Your business should have a way to measure the business value (so you know when you’re done). Everyone should be contributing to the business value metrics, and collaborating to get it done together.

Assuming you’ve done the hard work of figuring out what your business is trying to do (you understand the business value and have some metrics to measure against) the next part can be fun and rewarding!

Why haven’t you talked about DevOps yet?

Let’s assume (like many companies today) you “develop software as part of an effort to provide products & features, to give you a competitive advantage”.

Where I think “DevOps” has failed us is the misconception that it’s “half dev” and “half ops” and that problem lies in the very core of DevOps — It’s name! But in reality without software there’s nothing for “Ops” to do.

Customer leads to Business Value leads to Software. How we build, test, deploy and operate our software is important but we can’t call it DevTestSecQADeployOps so let’s just call it Good Software Delivery Practices.

Good Software Delivery Practices

You’ll likely want to incorporate automation and optimizations that allow you to “produce working (aka releasable) software every sprint” and reduce manual steps in favor of automated ones where rework is likely to occur.

You’ll want to “drive value through the system” and this usually translates into working on new products and features that benefit your customer (hopefully identified by your product team’s close relationship with it’s customers).

Your product/development team might also decide to operate in some flavor of “you build it, you run it” team which encourages them to build good quality software with resilience and alerting that’s actually helpful.

If I know I’m going to be woken up at 3am when things go wrong it’s going to fundamentally affect my choices when designing and operating my software (hint: for the better).

Ultimately when you combine “Business Value” with “Good Software Delivery Practices” every choice you make has clarity. You know “why” you’re doing something and you can trace it back to a specific metric that has a positive impact on your business (and by that I mean improving the lives of your customers in some way).

So when you go to that next meeting, or pickup your next Jira ticket from the “todo” column ask yourself:

Do I know why am I doing this?

Info

Also posted on medium as Business Value eats DevOps for breakfast.

The True Cost of Being Cloud-Agnostic

Wed, 09 Oct 2019 01:41:01 +0000

Let’s talk about the real cost of cloud computing, and by that I don’t mean hourly pricing models, network charges, and licencing implications.

I want to talk about opportunity cost, explore factors like speed to market, how effective our development teams can be, and the operational overheads involved in our technology choices.

The famous Allan Denot once said…

“If you believe in the cloud, you can’t be agnostic.”

…and by famous I mean in DevOps/Sydney, and by once I mean January 2019.

I believe in the cloud! When I make that statement, it doesn’t mean I promote “only using the services that are available in all clouds” or “building all of your own services from scratch so they work identically in each cloud”.

I’m a firm believer in using the right tool for the right job, and the right cloud or clouds for your project, program or organisation. I’m a cloud believer and I don’t have a vested interest in any one cloud.

There are many ways “the cloud” can make your life easier. If you’re looking for a simple way to store and retrieve files, and you’re okay with it taking a few seconds, not a few milliseconds, AWS S3 is a great option. AWS RDS is an ideal way to use the relational database that you know and love without having to be hands-on with installation, upgrades, and replication. Are you a Microsoft shop? It’s hard to pass up Azure for services like hosted Active Directory. Do you only care about writing code? Google App Engine is an amazing service for Planet-Scale web services with all of the infrastructure managed for you.

But what about the other definition of being cloud-agnostic? What about the dream of many organisations that we keep hearing so much about in the industry…

Cloud-agnosticism is great! No vendor lock-in, more customisation, but wait. Think of all those tightly integrated managed services you’re leaving behind.

If you read the article by by Maish Saidel-Keesing called “ Cloud-Agnostic: Friend or Foe?” this provides some great insights into this topic.

Trying to be cloud-agnostic may lead you to “only use features that are available in all clouds”, which is the lowest common denominator, or “deploy in such a way that we can switch a given workload to any cloud in minutes depending on the hourly price of a given service”, which I’ve not yet seen an organisation achieve.

The scary path I’ve seen too many organisations follow is the idea that we should build as much of the system as possible in-house to avoid any sort of lock-in.

The avoidance of cloud vendor products or even third party SaaS products for fear of being locked-in has been a common theme with the organisations I’ve worked with. I’ve never found the right way to describe the flaw with this way of thinking until I came across the following article which compared cloud provider lock-in to databases:

“If we choose to use Oracle as the database for a given application, we don’t usually also build the application on SQL Server to make sure we aren’t locked in to Oracle. We don’t do so simply because we believe that the costs of having two alternative databases for the same application will outweigh the benefits in risk management.”

“ Switching Costs and Lock-In” is a great AWS blog that explains the difference between switching-costs vs lock-in.

If cloud-agnostic is the “wrong strategy”, what is the right one?

It would be foolish of me to prescribe a magic bullet for your cloud journey. Having a well thought out application migration strategy and a shared vision for the future of your application architecture should be a key part of your strategy. If I only had a few minutes to set you on the right path I’d ask you to consider the following:

Some of the most successful projects I’ve had the pleasure of being part of have taken advantage of the hard work of cloud or SaaS providers. They help organisations save time, and allow them to get things done right the first time.

Cloud native, serverless and SaaS are some of the most powerful ways to add value to a DevOps/ cloud transformation, outside of the people-part of the transformation. Services like RDS, S3, Google App Engine, Buildkite, Github, and Slack are among my favourite technical accelerators to help drive success in organisations.

Is missed opportunity the real cost?

At this point you may still be asking yourself, what’s the true cost of being cloud-agnostic? Is it cheaper, or more expensive than the alternative? The cost I’m referring isn’t directly related to your profit and loss statement. In fact, the true cost of trying to be cloud-agnostic can be missed opportunity and revenue, stifling innovation, project delays, and missing out on a selection of some of the most useful services the cloud has to offer.

So here I stand, a Cloud Believer using the “best of breed” way of thinking and continuous improvement to help you get the most out of the cloud. After all if you believe in the cloud, you can’t be agnostic.

Info

Also posted on medium as The True Cost of Being Cloud-Agnostic and Contino as The True Cost of Being Cloud-Agnostic.