Winning with Serverless DevSecOps Pipelines

Winning with Serverless DevSecOps Pipelines

There's little doubt that Serverless Technologies are being adopted at a rapid pace [1]. I have personally seen typically stodgy, old-school enterprises take to building out serverless (especially FaaS) projects to handle some mission-critical operations. I tend to agree with experts like Simon Wardley (of the famous Wardley Maps) that Serverless will edge out Containers as a major driver of application delivery within organizations moving forward.  

TL:DR; If you're already aware of Serverless Tech and Security, I'd recommend you start reading from here

To me, some of the major benefits of serverless apps are:

  1. Leverage any programming language or platform and focus on the code. Developers can write their programs in any language and focus on the business-logic, rather than have to think too deeply about deployment parameters and needs or even about integrating with legacy codebases.
  2. Ability to support multi-source events. Serverless (FaaS) apps can be invoked on different types of events, like Queue events, S3 uploads, Emails and Text Messages, etc, which provides a great deal of operational flexibility.
  3. Scalability. Serverless Apps provide nearly unlimited scalability and better yet, provide specific scalability for functions that are invoked more than others. This is a major factor in terms of cost-savings for organizations running applications that are "serverless-first"
  4. Maintainability. For me, the biggest benefit of serverless, is well, serverless. There's no need to run and manage servers and associated infrastructure. You can just focus on the code and logic. The "Ops" aspect of serverless is limited and that makes it a very attractive proposition.

However, serverless tech is not without its security challenges, namely:

  1. Events - A Major source of security flaws with Serverless apps is the fact that you can invoke serverless functions from multiple types of event invocations. This is a far-cry from traditional web apps, where events are typically user-generated and target the api. With serverless, I can upload a XML file with a XML External Entities payload and trigger a function that would trigger the XXE bug with a Remote Code Execution.
  2. A Remote Code Execution flaw against a Serverless Function is pretty bad. Attackers can leverage an RCE like Insecure Deserialization, etc to execute code on the Function hosting environment and further gain access to the cloud account and other resources under the same cloud account/organization account. One of the common demos in my trainings are those of Remote Code Execution, leading to a near-total takeover of the organization's cloud account.
  3. The other security issue with serverless apps is the possibility of financial exhaustion. Remember that you pay per GB-Second in serverless. So if an attacker runs a memory-exhausting function (ReDOS vulnerability for example) in succession, then you are likely to shell out serious coin for your Serverless Functions.
My talk on Serverless and GraphQL at AppSec California 2019

Serverless DevSecOps - The Why and the How

I dont think I need to go to deeply into why you need to continuously deliver, secure serverless functions. However, one think you should consider is that serverless functions inherently accelerate the process of application development and delivery, which is why, more than ever, you need to implement some sort of continuous security practices for your serverless functions. Let's call this, for lack of a better word, "DevSecOps".

Here's a high-level view of a Serverless DevSecOps Pipeline. Again, please note that this doesn't need to be exactly this way. This is really an example that you can choose to use

Serverless Security Activities in the DevOps Pipeline

Good DevSecOps for Serverless is largely similar to good DevSecOps for any other application/stack, with a few key differences. And most of those differences are due to the inherent speed of delivering and deploying a serverless function AND its deployment environment (your cloud environment). I am going to keep the rest of this article AWS focused as I have more experience with AWS. However, you should (mostly) be able to apply these practices to any other cloud provider as well.

Let's dive in.

First off, I am not going to get into Threat Modeling (Planning Phase) in this article. But needless to say, it needs to be done. In fact, I am planning an entire article only on Threat Modeling for Serverless, which I find to be much easier than threat modeling for a monolith. But I digress.

Pre-Commit and Commit-Time Security

Be it serverless or otherwise, you should strive to have developers identify and rectify security issues as early in the lifecycle as possible. This means that developers should have "in-IDE" processes that identify and flag security vulnerabilities in their code. A good, free and lightweight way of doing this is to find security rules for linters like:

  • ESLint
  • Flake8, pep8
  • and other code quality tools

Developers typically already use these code quality tools. So embedding security rules into these code quality checks ensures that developers never change their workflow and they address security problems, just like they address code quality problems. Both desirable elements to have in any robust, continuous security process.

Tools/Lint rules like DLint and Bandit (python), ESLint Security Rules (JavaScript), SpotBugs (Java) work directly in the IDE and are able to flag security issues along with code quality issues. Consider using them as the developers write code itself.

This is essential for Serverless, especially when you have multiple platforms and languages being used to build serverless apps. The complexity also increases significantly if you use Golang, Java, Javascript and python to write your serverless apps, vs just one of these languages, as the need for Static checks or linters increase.

In addition to IN-IDE checks, there probably will be a need for you to set up a SAST or Static Checks as part of your DevOps pipeline. This is not only required from the perspective of "Trust but verify" but also from a visibility standpoint in the DevOps pipeline. With Serverless Functions, the fidelity of static checks needs to be of a high quality. In addition, I'd recommend the use of simpler Static Scanning tools without the linked call-graph and sinks capability, simply because vulnerable code is typically restricted to a function and will not likely have calls to methods or variables, like in the case of a typical web application.

Build-Time Security

Probably one of the most important practices in the modern DevOps pipeline is Source Composition Analysis (SCA) or Supply-Chain Security Analysis. With Serverless Functions, this is no different. In fact, I would say that with the large number of functions and the sprawl in terms of dependencies per function. Source Composition Analysis becomes not only mandatory, but also an item to be engineered correctly. For example - A policy and practice to use only specific libraries for a given serverless application. Or engineering the serverless functions with minimal dependencies.

With Serverless, SCA in some ways can be more localized, hence more effective. For example, you have an egregious Remote Code Exec flaw in the Java lib that you are using for a function. You can take corrective action far more easily than you can, in say a monolith, where a change to that single dependency might result in the cascading effect of dependency hell, where things randomly break, where you least expect it.

In addition, with the right package management tooling, SCA can be baked into the developer workflow. For example, NPM Audit, which is a (sub)tool from the Node Package Manager allows developers to scan for vulnerable dependencies and even warns on vulnerabilities in dependencies when you attempt to install them.

This approach is being adopted by newer package management tooling like pipenv from python, where the safety library is integrated into the CLI option pipenv check that scans all the existing dependencies for security issues. Rust's Cargo has a cargo-audit library that aims to do the same thing. In addition, you have great free tools like OWASP Dependency Check and Dependency Track that scan package management files across multiple languages for security vulnerabilities.

Test-Time Security for Serverless

Now, this is where things get a little complicated. Specifically with Serverless Functions. Normally, for web apps or mobile apps, you can run some DAST or IAST tooling to identify runtime security flaws against your applications. I am not going to delving into the DAST vs IAST debate in this article, but these are typically the options you have for runtime security analysis in the DevOps pipeline. Both of these options are much more useful if you are running a serverless application that is a WebAPI (RESTFul API), where you can test it, like you would, any REST API.

However, its when Non-API-Gateway event-driven functions start to show up, that you have a non-trivial runtime security scanning problem. For example, lets say that someone can trigger a command injection flaw in your serverless function, triggered uploading a file to Amazon S3, which is named with the payload of a command injection attack. This is clearly not a API Gateway driven event. This can't be tested like a Web API. There's a chance that given certain validation bypasses, this could be easily slipped through a static scanner as well.

Most Serverless Security Solutions (and they are mostly commercial at the time of this writing) perform dynamic security analysis with a combination of instrumentation, whitelisting, blacklisting, static and source composition. However, we've been able to slip some reasonably trivial attacks by many of these solutions, undetected. And in the case of business logic flaws, which are very likely in the case of a technology that touts itself as the focal point of your development efforts, you'd be hard-pressed to find automated tooling to find any security issues.

However, its not all gloom and doom when it comes to dynamic security for serverless, largely because of the possible attack surface of the serverless stack. One of the largest and most significant issues with serverless functions is the misconfiguration of IAM, leading to a large blast-radius of the attack. For example, an attacker could trigger a remote code execution on a NodeJS Function, and due to an IAM misconfiguration (Permissive roles), gain have privileges to destroy your EC2 servers or gain access to DynamoDB database tables that she shouldn't have access to. This is IMO, the largest area of risk in serverless stacks. And with the aid of some good Cloud Security Scanning tooling, you should be able to identify security issues like this quite easily. But more on this later.

When it comes to pure Dynamic scanning, it depends on the way you are using your Serverless Functions. If you are running your serverless stack as a REST API, then you can easily leverage OWASP ZAP or BurpSuite to perform dynamic scanning against your REST API. I highly recommend proxying an E2E test automation script before running the DAST scan, as you are likely to have more success finding vulnerabilities.

The other option that you can choose to use is to leverage one of the commercial solutions like Protego or PureSec to find security issues with your Serverless Functions. I believe PureSec has a free product in FunctionShield. But options are limited, as this space is quite nascent and will see growth in the future.

Deploy-time Security for Serverless

If there were practices (for Serverless) that I would put most of my money on, other than Commit-time or Build time checks, it would be during deployment of the serverless functions.

This is because most of the tooling (especially free), exists in abundance for this stage of Serverless DevSecOps. This is simply because the Cloud API allows for the breakdown of a serverless function by:

  • IAM Role
  • Versions
  • Layers and Layer code
  • Permissions - Invocation
  • Other Function Configurations

As a result, the visibility for tools to perform security analysis against functions post deployment (to staging or dev environments) is great.

Tools like ScoutSuite and Prowler have modules that scan for Generic security issues with AWS Lambda and with secrets in Lambda UserData and so forth.

However, for a more focused Serverless Scanning tool, I recommend using LambdaGuard from SkyScanner engineering. Its an easy to use CLI that be be easily fitted into your CI pipeline and triggered post deployment to identify security issues.

It produces an easy-to-read report in multiple formats (HTML, JSON)

In conclusion

DevSecOps for serverless, in my opinion is very important and a must-have aspect of your SDL from the get-go. However, you'll find that needs for DevSecOps for Serverless stacks have differences from typical web or mobile applications. Also, as I mentioned, Dynamic Security testing for serverless stacks is an area that I hope evolves in the future and there's some useful OSS/Free tooling available for this.


we45 is running a 2 day training @ OWASP GlobalAppSecDC on "Attacking and Defending Containers, Kubernetes and Serverless". Do register if you're interested: https://globalappsecdc2019.sched.com/event/SK9t