Modern server side software faces numerous security challenges from countless directions. Unfortunately, it often takes just one chink in the armor to infiltrate an otherwise secure system, potentially creating massive damage.
At FP Complete, we believe security must be an integral part of all activities around software: requirements, project planning, architecture, development, quality assurance, deployment, and operations. We've worked with dozens of companies to help them strengthen their server and cloud based systems. To help you understand some of the surprising directions that socurity problems - and solutions - can come from, here are some key recommendations.
While you take security seriously, you cannat afford dedicate all your best staff to reinventing the wheel. But you also can't wait to act until after a security breach happens. We're here to help your team, working remotely to add our capacity and expertise to your own. For help planning and implementing some cost-effective, timely improvements to your security in all these areas, or just to discuss what may be possible and get some feedback from our technical team, please contact our team.
We've discovered many surprising, high-impact places where security can be added to - or missing from - an information system. Every one of these is actionable, and you don't have to wait for a grand plan to start making measurable impacts.
Requirements
Security is a broad, encompassing term. One person on a team may think of security as "make sure only licensed individuals are able to use our software." Another may think "ensure no user can read another user's data." And a third may interpret it as "don't let users upload viruses that will infect our servers."
These are all valid security concerns, along with ensuring only authorized users access machines, preventing common web exploits in software, preventing social engineering attacks, and much many more.
During the requirements phase of a project, it is vital to consider as many of the security requirements as possible, and ensure the team leads are all in sync on them. And this list of security requirements should regularly be reviewed as needs change, products develop, and new threats come to light.
Implementation teams will generally omit or short-cut work items that are not part of the spec. Unfortunately half-done security is like locking one door while leaving the other wide open. We recommend gathering best-practices lists of security requirements, and incorporating them into your project specs, typically in the test plans and acceptance criteria.
Working for dozens of companies' project teams, we've been able to learn some trends and patterns. We've seen that a large portion of project success is determined by the thoughtfulness of the up-front discussions. If team members aren't aligned on the security goals and requirements, sort this out up front - not after new technology has been chosen and implemented.
Project planning
When planning a project, most teams know to estimate and schedule time for basic feature development. Well-planned teams will also include budget for debugging, quality assurance, documentation, and project management. However, security is often missing. Many assume this is part and parcel of other activities. After all: developers should be coding with security in mind. The quality team should be exploring potential security vulnerabilities.
A typical IT project should plan to spend 5 to 15 percent of its total budget on issues relating to security. If your plans look much skimpier than this, it's time to get much stronger about your plans. Don't let security remain undone in the interest of a faster (but totally unsafe) outcome.
Skilled and experienced software and quality engineers will often plan for security without being asked. But there are three things to keep in mind:
- Not all engineers have specialized skills in security
- Without sufficient time in the budget, engineers may overlook some details of security
- Managers under pressure to meet commitments won't find time for workers to add security later
We recommend including explicit items in the budget, and ideally specific trained individuals, to focus on security. This usually falls to the Quality Assurance team and the Operations team. On large projects, we would recommend a dedicated security team within the QA team, which typically has the correct purview and attention to detail to bring security issues to light. We'll further discuss quality assurance below.
Architecture
It's much easier to design a secure system than to retrofit security onto an existing system. When there is sufficient time in the budget to address security, and the major security requirements are well known, we can architect correct solutions.
A common mistake is to think of this architecture work as being the responsibility of one team: only the deployment team, say, or only the software engineers. Working in isolation, no one team can architect a secure solution. Detailed discussions of assumptions and expectations prevent "I thought you were supposed to handle that" security failures.
Consider a web development project with infrastructure, front end, and back end teams. The front end team may implicitly make assumptions around cross domain requests they can make. Unbeknownst to them, the infrastructure team may be planning on hosting static files on a CDN served from a different domain name, violating browser security policies. The back end team may end up "solving" the issue by applying Cross-origin resource sharing (CORS) headers. While the software may work, vital security assumptions may have been violated, opening up the software to attack.
Or consider a cloud migration of an app, where remote servers will need to make arbitrary requests into an on-premises database. This software will work, but only after Operations opens up a huge hole in the firewall allowing remote machines to talk to the formerly secured database server. Ops may refuse to deploy this, leading to months of software rework - or they may be bullied into deploying it, leading to blatant security risks. Integrated DevOps collaborations that incorporate software designing and deployment planning can prevent architecture mistakes like this.
Spending an extra few days at the beginning of the project can save months of wasted effort down the line. It can also potentially prevent company-destroying security exploits. Breaking down "silos" by including people from both Dev and Ops may seem like a lot of cultural and management work, but it's a great investment.
Development
By sheer numbers, software is likely the largest source of security vulnerabilities. This can come down to mistakes (bugs), lack of clarity in requirements, poor communication between teams, and lack of understanding by developers. It can also come from not understanding how one person's change affects another person's work, or how the pieces will be integrated. Remember that most developers are not by nature security experts.
Here are three of the best tools for improving development security:
- Provide security training to the team in the form of courses, best practices documents, and code review sessions
- Keep reminding the team that security requirements are mandatory, not to be bypassed or de-prioritized
- Choose tools that support good security practices out of the box
Regarding tools: we too often see companies choosing software languages and frameworks based exclusively on what will get them to market fastest. This is most often a misleading metric. The majority of software development occurs in the maintenance phase, not initial development. Languages and frameworks which focus on easing the maintenance burden, especially around security, are vital. Our Vice President of Engineering, Michael Snoyman, spoke about this in his talk "Functional Programming for the Long Haul".
This is one of the reasons why FP Complete builds software in Rust. Rust provides a great ability to get working code out quickly like Python or Ruby, provides the efficiency at runtime of languages like C and C++, and provides a strong type system to encode invariants like Haskell or Ada. With the right programming languages and libaries, good design helps your time-to-market rather than being an obstacle that you have to trade off.
If you're interested in learning more, please check out our training solutions. Feel free also to check out the resources on this site for more about Haskell and Rust, two of our favorite programming languages.
Quality Assurance
If you have a large enough project, it's worth having a dedicated security team to review the entire software and DevOps stack holistically and bring their experience to bear on recommendations. However, for smaller projects, a dedicated team may not be warranted. Including security overview as part of Quality Assurance's recommendations can be a workable solution. QA already has scope to review the totality of a project. And given clear sets of requirements, a QA team can put together proper acceptance criteria and tests plans to cover these requirements.
Remember that a QA worker is dedicated to finding errors, omissions, and other problems. A quality-driven culture welcomes, rather than suppresses, news about risks and defects. Knowing what you are dealing with allows you to prioritize fixes and allocate time realistically, rather than just hoping that the unexamined "hasn't broken yet" parts are secure.
Even in such a setup, it's a good idea to include some dedicated security investigation via a penetration testing team. This may uncover gaps in the requirements that a quality engineer would not know to look for. Deploying systems that haven't been scrutinized and tested is like owning a house without fire insurance: it saves money, until the day it very much doesn't. For things you can afford to break (like prototypes), QA may be the wrong use of budget, but for production systems it's the wise choice.
We also recommend leveraging multifaceted testing to get the most bang for your quality buck.
Deployment
Many of the best practices FP Complete recommends around DevOps will help ensure a secure deployment. Immutable infrastructure, infrastructure as code, containerization, and CI/CD all help ensure that your software is in a known state, network architecture is configured correctly, and testing matches the production environment.
Another common security hole lies in authentication and authorization. Ensuring secure passwords, non-reused credentials, multi-factor authentication, and deterrents to social engineering attacks are all vital. A common and dangerous shortcut is to have a few super-powerful accounts with full access to all running software, infrastructure, and data. You can do much better. We cover some of these topics in Understanding Cloud Auth.
Keeping in mind that security is a multilayered approach, there are some additional ideas you should consider, such as leveraging application firewalls to automatically detect and protect against some classes of attack. This can be a good double-check on software security, and is surprisingly powerful for fairly little cost.
Operations
The operations team needs to have useful information to help protect a system. Importantly, this means:
- Log aggregation for investigating events
- Monitoring to collect useful information
- Alerts to notify of suspicious activities and enable a response
There are many software packages out there to help with these activities. Once you have the software available and installed, configuring correctly is its own challenge. You can read more in our article on logging, monitoring, and alerting.
You have probably read articles about companies that were penetrated and didn't know it for days or weeks. That's enough time to copy all your sensitive data, install worms and viruses, or even modify your systems.
The operations team is the first line of response against an actual intrusion. While we do everything to prevent a successful attack, it's best to plan in advance for how to respond. Having intrusion playbooks and training for the operations team will hopefully be unnecessary investment. But if there is an attack, you'll be very glad for the insurance.
And finally: while the entire team should have social engineering attack training, the operations team is the most at risk for these attacks. If only one group will receive such training, it should be them.
Learn more
FP Complete specializes in secure, reliable, and efficient server side software, and getting your product to market quickly. Want to learn more? Check out our DevOps homepage.
If you would like help planning and implementing some cost-effective, timely improvements to your security in all these areas, please contact our team.