Managing observability

This is achieved through observability, which is the capability to track the security health of your service through the processes of logging, monitoring, and alerting. This enables you to gain insight into your service’s state, users, and data.

Good observability will allow you to:

• quickly identify and respond to anomalies which may be a sign of potential security vulnerabilities or incidents
• feed information into incident management processes
• monitor the effectiveness of security controls
• identify areas where security could be improved
• help you to meet regulatory requirements and industry security standards

Security observables (including information and events) are identified changes (or triggers) within your system that may indicate a security issue that requires your attention.

You should begin identifying observables (such as security alerts generated by applications and networks) from the service design stage so you can build and integrate these within the organisation’s overall security monitoring capabilities. You should regularly review your observability processes to ensure any changes to the service or new threats have been considered.

Completing this activity will help you to achieve the outcomes included in the Secure by Design principles to build in detect and respond security and minimise the attack surface.

Who is involved

Managing observability should be led by your technical team (such as security and technical architects, developers and DevOps) who will have a detailed understanding of your system’s infrastructure and be able to configure the appropriate data logging, monitoring and alerting capabilities. They should consult with security professionals to ensure information is being collected and interpreted correctly.

There may be dedicated cyber security incident response teams or a Security Operations Centre (SOC) within your organisation who could require access to observable data to allow them to detect potential threats, investigate security events, and take appropriate actions to mitigate risks.

How to manage observability

Your organisation may already have templates for the information and events they expect you to capture. Use these as your starting point, considering what additional information and events are required to provide good observability of the service.

Step 1: Decide what observables to capture

You should begin by identifying what security related information and events to capture within your systems. This may include:

• Metrics – data that can reveal unusual or abnormal behaviour indicating a possible security threat. For example, a sudden network traffic spike might be a sign of a DDoS attack or malware infection.
• Log events – a timestamped record of events that can provide information about attempted and successful access to sensitive data and any changes made to the system. You can use logs to detect and investigate suspicious activity, such as failed login attempts or unexpected access to sensitive data.
• Traces – a process for providing detailed information about the flow of a request through a system. This allows you to identify where there might be security breaches or vulnerabilities.

Your threat assessment, threat modelling and security risk mitigations will help you to identify what information may be useful in the event of a security incident.

Step 2: Set up data collection capabilities

Document the relevant observables for your service assets. You should investigate whether you have the appropriate capabilities set up to collect and monitor observables, and what you may need to implement to give you full visibility. Speak with your organisation to see what capabilities and resources may be available to support you.

Logging and monitoring systems will allow you to track and record activity such as user access and changes made to the system. Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS) and Security Information and Event Management (SIEM) tools can help you detect suspicious activity and create alerts.

If you are using third-party vendors to deliver components within your system, they should provide cyber security monitoring, logging and alerting as part of their service. Explore the options and expertise available to see how it can complement your own observability processes.

All observables should be collected and stored securely. It should also be easy to search and analyse so potential security issues can be identified.

Step 3: Manage access to observable outputs

Your service’s observables and their outputs should be treated with a high level of security so they don’t compromise the assets they’re being used to protect. There are various measures you should put in place to maintain the integrity and confidentiality of this information.

Access Controls

Implement strict access controls to limit access to log files and event data. Ensure that only the necessary authorised personnel have permission to view, modify, or delete logs. Strong authentication mechanisms, such as multi-factor authentication, should be considered mandatory for access to view, modify or delete logs.

Encryption

Log files and event data should be encrypted when being transferred and stored. Use secure transport protocols (such as HTTPS or SSH) when transmitting logs across networks and encrypt log databases to protect data from unauthorised access if storage systems are compromised.

Secure Storage

Consider using dedicated log management solutions or SIEM platforms that provide built-in security features for data storage. Regularly back up information and implement appropriate disaster recovery measures.

Logging Separation

Separate log files from the systems or applications they originate from. Storing logs on a separate server or dedicated log management system helps protect them from being compromised if the source system is breached. It also allows for centralised and controlled access to log data.

Backup and Retention

Regularly back up log files and establish appropriate policies for retaining data. Backups can help restore log data in case of accidental deletion, system failure, or data corruption. Retention policies should align with regulatory requirements and the needs of your organisation.

Employee Awareness and Training

Educate employees about the importance of observable data protection and the role they play in maintaining the security of log files. Train employees on proper handling, storage, and disposal of data.

Step 4: Decide what actions to take

Your observability process should consist of regular checks to identify potential security issues and attacks within systems. The output of these should result in specific actions. These may include:

• Review – for data that does not indicate an immediate threat, a workflow should be set up to investigate and understand the reasons behind data anomalies
• Alert – certain events should trigger communication through the appropriate channels so those responsible for responding to cyber security threats can take the necessary steps
• Automate – your system can be set up isolate security incidents in some circumstances, for example shutting down certain functions or features until they have been investigated and resolved

Applying the right actions to each observable will help to increase visibility of known events and allow retrospective analysis of unknown events.

Integrate your observability processes into your threat model so it can become part of your ongoing plan for identifying and managing cyber security threats.