Secure Software Development and Product Security¶

Last Reviewed: 2025-02-17:19:44:43-UTC

APS development team follows the latest security best practices when developing software, and automates security testing throughout development lifecycle whenever possible.

Remediation:
- Follows defined vulnerability management lifecycle
- Ensures no high risk security vulnerability is in production

Details about the APS software application architecture and security are documented on the product development / engineering wiki.

Policy Statements¶

APS policy requires that:

APS software engineering and product development is required to follow security best practices. Product should be “Secure by Design” and “Secure by Default”.
Quality assurance activities must be performed. This may include

peer code reviews prior to merging new code into the main development branch (e.g. master branch); and
thorough product testing before releasing to production (e.g. unit testing and integration testing).

Risk assessment activities (i.e. threat modeling) must be performed for a new product or major changes to an existing product.
Security requirements must be defined, tracked, and implemented.

All critical or high severity security findings must be remediated prior to each release.
All critical or high severity vulnerabilities discovered post release must be remediated in the next release or within the defined, predetermined timeframe.
Any exception to the remediation of a finding must be documented and approved by the security team.

Controls and Procedures¶

Software Development Process¶

Overview¶

Software development at APS follows a release strategy that provides traceability for production software changes. Features, enhancements, and bugs are written up as Issues in Rally. An engineer on a small team proposes changes necessary and creates a review for the team (GitHub). Continuous integration (Jenkins) kicks off unit and functional tests which pass before changes are deployed to the development environment where Product Owner tests are run before the new code replaces the existing in-service code (test then deploy model).

APS practices continuous delivery of code into production through multiple environments: development and production. The deploy process and infrastructure roll-out are written as code (using technologies such as AWS CloudFormation) and managed under source control.

APS’s lower environment (dev) provides an ecosystem of sample data sets that exercise the application and services when test automation is run. The test environment is where the system is stressed for performance and scalability. Performance and scalability changes are driven by metric data captured through monitoring and logging (metrics before and after change – typically captured as part of the issue description/writeup).

Deployments to production are gated by change control process where an issue is opened which identify what is new/changed (Rally). Sign-offs are recorded by development, product owner, and product management. Production roll-outs happen on a regular basis without impact to service. This continuous process allows for security updates to roll out regularly and with urgency. If there is impact to production, a rollback is performed to restore service and whatever caused the problem is reverted from source. This restarts the re-proposal approval process of source changes. This process keeps the set of differences between the development environment and the production environment as low as possible.

In the continuous delivery mindset, features are not released by the deployment of code into production, instead features are enabled in production at the appropriate time (dark launching). Feature toggle enablement in production is gated by a change control ticket (Rally) that follows the software roll-out approval process. Feature toggle enablement in production can have a few more dependencies than code. Those dependencies include things like external documentation, early access programs, and internal playbooks for supporting the feature.

Secure Development Standards¶

Traceability of code changes allow for our software to be a living entity. Our current system for documenting changes is Rally. Every commit should have a Rally supplied that describes contextually why this change is necessary and reasonable. These artifacts over time allow for one to trace the lineage of why our production software and services change over time.

All APS git repositories have a company standard configuration from a GitHub perspective. This standard is a guideline and can be relaxed when those exceptions are needed. One example of an exception, is the wiki repository, as editing a wiki and always requiring a PR in this setting slows down ‘flow’.

Code: <#>
Repo settings: <#>
Build: <#>

NOTE: Not all projects (and repos) follow this standard, certain projects might be excluded (e.g. wiki).

Developers follow the master branch strategy and code review process below:

All development uses the main branch used for the current release. Any changes required for a new feature or defect fix are committed to that branch.
- These changes must be covered under 1) a unit test where possible, or 2) integration tests.
- Integration tests are required if unit tests cannot reliably exercise all facets of the change.
Developers are strongly encouraged to follow the commit message conventions suggested by GitHub.
- Commit messages should be wrapped to 72 characters.
- Commit messages should be written in the present tense. This convention matches up with commit messages generated by commands like git merge and git revert.
- Additionally, the commit messages should start with the Rally Issue ID when applicable.
Code reviews are performed as part of the peer review swim lane. Once a change is ready for review, the author(s) will notify other engineers using an appropriate mechanism, typically by adding reviewers to the ticket.
- Other engineers will review the changes, using the guidelines above.
- Engineers should note all potential issues with the code; it is the responsibility of the author(s) to address those issues or explain why they are not applicable.
- If changes/commits are made, it should reset previous approvals and require review and approvals again before the code can be deployed.

If the feature or defect interacts with sensitive data, or controls access to sensitive data, or contains security/risky infrastructure changes to the target environment, the code changes must be reviewed by the Security team before the feature is marked as complete.
- This review must include a security analysis for potential vulnerabilities such as those listed in the OWASP Top 10.
- This review must also verify that any actions performed by authenticated users will generate appropriate audit log entries.

Release Strategy¶

Features, enhancements, and bugs are written up as issues (Rally). An engineer on a small team proposes changes necessary and creates a review for the team (GitHub). Continuous integration (Jenkins) kicks off unit and functional tests which pass before changes are deployed to the development environment. Small teams can decide to follow a source-control branching strategy that makes sense: git-flow, github flow.

APS practices continuous delivery of code into production through multiple environments: development then production. The deploy process and infrastructure roll-out are written as code (IaC) and managed under source control.

APS’s lower environment (dev) provides an ecosystem of sample data sets that can be used to exercise the application and services when test automation is run. The test environment is where the system is stressed for performance and scalability. Performance and scalability changes are driven by metric data captured through monitoring and logging (metrics before and after change - typically captured as part of the issue description/writeup).

Deployments to production are gated by change management process where an issue is opened which identify what is new/changed (Rally). Sign-offs are recorded by development, testing, security, and product management. Production roll-outs happen on a regular basis without impact to service. This continuous process allows for security updates to roll out regularly and with urgency. If there is impact to production, a rollback is performed to restore service and whatever caused the problem is reverted from source. This restarts the re-proposal approval process of source changes.

This process keeps set of differences between the development environment and the production environment as low as possible.

Features may be released via code deployments or features may be enabled in production at an appropriate time (dark launching). Feature toggle enablement in production is gated by the same change management ticket (Rally) that follows the software roll-out approval process. Feature toggle enablement in production can have a few more dependencies than code. Those dependencies include things like external documentation, early access programs, and internal playbooks for supporting the feature.

Detailed process and procedures for code promotion and production release: See Configuration and Change Management.

Source Code Management¶

APS development/engineering team uses GitHub for source code management. Access to GitHub and its configuration standards include:

All developers must authenticate to gain access to GitHub and code repos hosted on GitHub according to standards and procedures defined in the Access Policy:
Access control to the GitHub web interface must be enabled, via SSO and/or MFA if applicable
SSH public/private key access may be used for command line or git access to the code repos
All code repos in GitHub follow these configuration standards:
All repos must have an owner identified and listed
All repos are by default private

High Level Application Security Requirements¶

All APS software must be developed to include the following general application security principles and requirements. Web applications must also protect itself against the OWASP Top 10 vulnerabilities.

Protect sensitive customer data such as PII and account passwords. Encrypt data stored (at rest).
Secure data in transit and customer communications via TLS.
Provision strong access control (authentication and authorization). Prevent and report unauthorized access.
Log all transactions and activities to be able to tell who did what, when, where, and how. Mask or remove sensitive data in logs.
Implement client security at application endpoints (e.g. browser, mobile app).
Communicate securely across application endpoints and between service consumers/producers.
Use secure defaults to ensure security when in all error conditions.
Check and maintain the security of all third party and open source libraries/components/dependencies.
Validate all data inputs; encode data outputs when appropriate.
Deploy and configure applications securely to production.
Perform regular vulnerability analysis and apply security patches promptly.
Secure privileged access to production environments and ensure ongoing application monitoring.

All software code must complete a set of security scans/testing prior to being deployed to production, including open source dependency scanning, static and dynamic application security testing, as well as periodic penetration testing.

Pre-production testing is performed with nonproduction data in nonproduction environments. Health checks are performed regularly or automated in production.

Software vulnerability identified through any of the above processes shall be reported and tracked following APS Vulnerability Management process as defined in the Vulnerability Management Policy and Procedures.

Secure Design and Application Threat Modeling¶

APS Security Team in collaboration with development team performs full Application Threat Modeling and Risk Assessment on per-application basis using a custom approach that relies on industry standards and best practices.

Major application updates are captured via an RFC process. The RFC template includes Security Consideration as a required section. This section is used to document abuse cases including:

risks identified,
attack vectors, and
mitigating controls.

Each RFC is required to capture sufficient details of the feature/component to be developed, including use cases, motivation and outcome, and the following design details as applicable:

authentication/authorization mechanisms,
network communications,
data encryption,
cloud services used,
logging/auditing,
data flow diagram/description,
edge cases, drawbacks, and alternatives.

The RFC must be approved prior to implementation. Security team is included in RFC reviews via email.

Platform Design and DevOps Security Details¶

Documentation on the APS Engineering Wiki may include additional security specifications as well as the security design and implementation details of the APS Platform and its supporting operations.

Access Control of the Application (Identification, Authentication, Authorization, Accounting)¶

APS external software application that is customer facing with access to customer specific data, including sensitive information such as PII , implements strong access control, covering the Identification, Authentication, Authorization, and Accounting/Auditing (IAAA) of access and user activity.

The implementation ensures that

the user requesting access is the one claimed (Identification and Authentication);
only users authorized to access specific data are allowed to (Authorization); and
their access activities are logs (Accounting/Auditing) according to the APS auditing standards.

The current implementation leverages AWS Cognito for user identity management and access.

The backend platform implements granular Attribute-Based Access Control (ABAC) for granting access to specific services and data based on the attribute(s) of a principal (i.e. user requesting access – an attribute could be the role or group membership or organization the user belongs to) and the attribute(s) of the requested resource (i.e. data or service – an attribute could be the project this data belongs to).

More implementation details are documented on the internal Engineering wiki.

Penetration Testing¶

External Penetration Testing¶

An external penetration testing is performed at least once a year by a qualified security researcher / ethical hacker on the security team internally and/or with an external security consulting firm.

Outsourced Software Development¶

APS requires all outsourced software development to follow the same rigor and process as internal engineering. Outsourced developers must develop in our secure environment, accept and follow our security policies and procedures, and comply with the same secure coding standards, including:

Receive regular OWASP or equivalent secure coding training.
Follow the same source control, code review, and security code scanning procedures as defined.
Install endpoint compliance agent that checks to make sure firewall, encryption, patching, password policy, screensaver password, and other required protection is properly configured.

Additionally, the third party firm providing outsourced development services must demonstrate that they have conducted the appropriate screening during hiring.

Production System Monitoring and Paging¶

Software and systems deployed in production are monitored 24/7 for health check and other major/critical error conditions. The on call team is paged via PagerDuty in the event an error or failure is detected.

Notifications via additional channels such as Slack and email are also configured.