After onboarding Microsoft Sentinel into your workspace, connect data sources to start ingesting your data into Microsoft Sentinel. Microsoft Sentinel comes with many connectors for Microsoft products, available out of the box and providing real-time integration. For example, service-to-service connectors include Microsoft 365 Defender connectors and Microsoft 365 sources, such as Office 365, Azure Active Directory (Azure AD), Microsoft Defender for Identity, and Microsoft Defender for Cloud Apps.

You can also enable out-of-the-box connectors to the broader security ecosystem for non-Microsoft products. For example, you can use Syslog, Common Event Format (CEF), or REST APIs to connect your data sources with Microsoft Sentinel.

The Data connectors page, accessible from the Microsoft Sentinel navigation menu, shows the full list of connectors that Microsoft Sentinel provides, and their status in your workspace. Select the connector you want to connect, and then select Open connector page.

This article describes supported data connection methods. For more information, see Microsoft Sentinel data connectors reference and the Microsoft Sentinel solutions catalog.

Enable a data connector

The Data connectors page, accessible from the Microsoft Sentinel navigation menu, shows the full list of connectors that Microsoft Sentinel provides, and their status. Select the connector you want to connect, and then select Open connector page.

You’ll need to have fulfilled all the prerequisites, and you’ll see complete instructions on the connector page to ingest the data to Microsoft Sentinel. It may take some time for data to start arriving. After you connect, you see a summary of the data in the Data received graph, and the connectivity status of the data types.

In the Next steps tab, you’ll see additional content that Microsoft Sentinel provides for the specific data type – sample queries, visualization workbooks, and analytics rule templates to help you detect and investigate threats.

For more information, see the relevant section for your data connector in the data connectors reference.

REST API integration

Many security technologies provide a set of APIs for retrieving log files, and some data sources can use those APIs to connect to Microsoft Sentinel.

Data connectors that use APIs either integrate from the provider side or integrate using Azure Functions, as described in the following sections.

For a complete listing and information about these connectors, see the data connectors reference.

REST API integration on the provider side

An API integration that is built by the provider connects with the provider data sources and pushes data into Microsoft Sentinel custom log tables using the Azure Monitor Data Collector API.

For more information, see your provider documentation and Connect your data source to Microsoft Sentinel’s REST-API to ingest data.

REST API integration using Azure Functions

Integrations that use Azure Functions to connect with a provider API first format the data, and then send it to Microsoft Sentinel custom log tables using the Azure Monitor Data Collector API.

To configure these data connectors to connect with the provider API and collect logs in Microsoft Sentinel, follow the steps shown for each data connector in Microsoft Sentinel.

For more information, see Use Azure Functions to connect your data source to Microsoft Sentinel.

 Important

Integrations that use Azure Functions may incur additional data ingestion costs, because you host Azure Functions on your Azure tenant. For more information, see the Azure Functions pricing page.

Agent-based integration

Microsoft Sentinel can use the Syslog protocol to connect via an agent to any data source that can perform real-time log streaming. For example, most on-premises data sources connect via agent-based integration.

The following sections describe the different types of Microsoft Sentinel agent-based data connectors. Follow the steps in each Microsoft Sentinel data connector page to configure connections using agent-based mechanisms.

For a complete listing of firewalls, proxies, and endpoints that connect to Microsoft Sentinel through CEF or Syslog, see the data connectors reference.

Syslog

You can stream events from Linux-based, Syslog-supporting devices into Microsoft Sentinel by using the Log Analytics agent for Linux, formerly called the OMS agent. The Log Analytics agent is supported for any device that allows you to install the Log Analytics agent directly on the device.

The device’s built-in Syslog daemon collects local events of the specified types, and forwards them locally to the agent, which then streams them to your Log Analytics workspace. After successful configuration, the data appears in the Log Analytics Syslog table.

Depending on the device type, the agent is installed either directly on the device, or on a dedicated Linux-based log forwarder. The Log Analytics agent receives events from the Syslog daemon over UDP. If a Linux machine is expected to collect a high volume of Syslog events, it sends events over TCP from the Syslog daemon to the agent, and from there to Log Analytics.

For more information, see Connect Syslog-based appliances to Microsoft Sentinel.

Common Event Format (CEF)

Log formats vary, but many sources support CEF-based formatting. The Microsoft Sentinel agent, which is actually the Log Analytics agent, converts CEF-formatted logs into a format that Log Analytics can ingest.

For data sources that emit data in CEF, set up the Syslog agent and then configure the CEF data flow. After successful configuration, the data appears in the CommonSecurityLog table.

For more information, see Connect CEF-based appliances to Microsoft Sentinel.

Custom logs

Some data sources have logs available for collection as files on Windows or Linux. You can collect these logs by using the Log Analytics custom log collection agent.

Follow the steps in each Microsoft Sentinel data connector page to connect using the Log Analytics custom log collection agent. After successful configuration, the data appears in custom tables.

For more information, see Collect data in custom log formats to Microsoft Sentinel with the Log Analytics agent.

Service-to-service integration

Microsoft Sentinel uses the Azure foundation to provide out-of-the-box, service-to-service support for Microsoft services and Amazon Web Services.

For more information, see Connect to Azure, Windows, Microsoft, and Amazon services and the data connectors reference.

Deploy as part of a solution

Microsoft Sentinel solutions provide packages of security content, including data connectors, workbooks, analytics rules, playbooks, and more. When you deploy a solution with a data connector, you’ll get the data connector together with related content in the same deployment.

For more information, see Centrally discover and deploy Microsoft Sentinel out-of-the-box content and solutions and the Microsoft Sentinel solutions catalog.

Data connector support

Both Microsoft and other organizations author Microsoft Sentinel data connectors. Each data connector has one of the following support types:

Find the support contact for a data connector

To find the support contact information for a data connector:

1. In the Microsoft Sentinel left menu, select Data connectors.
2. Select the connector you want to find support information for.
3. View the Supported by field on the side panel for the data connector.The Supported by field has a support contact link you can use to access support and maintenance for the selected data connector.

Security Orchestration, Automation, and Response (SOAR) in Microsoft Sentinel

Azure Sentinel is now called Microsoft Sentinel, and we’ll be updating these pages in the coming weeks. Learn more about recent Microsoft security enhancements.

This article describes the Security Orchestration, Automation, and Response (SOAR) capabilities of Microsoft Sentinel, and shows how the use of automation rules and playbooks in response to security threats increases your SOC’s effectiveness and saves you time and resources.

Microsoft Sentinel as a SOAR solution

The problem

SIEM/SOC teams are typically inundated with security alerts and incidents on a regular basis, at volumes so large that available personnel are overwhelmed. This results all too often in situations where many alerts are ignored and many incidents aren’t investigated, leaving the organization vulnerable to attacks that go unnoticed.

The solution

Microsoft Sentinel, in addition to being a Security Information and Event Management (SIEM) system, is also a platform for Security Orchestration, Automation, and Response (SOAR). One of its primary purposes is to automate any recurring and predictable enrichment, response, and remediation tasks that are the responsibility of your Security Operations Center and personnel (SOC/SecOps), freeing up time and resources for more in-depth investigation of, and hunting for, advanced threats. Automation takes a few different forms in Microsoft Sentinel, from automation rules that centrally manage the automation of incident handling and response, to playbooks that run predetermined sequences of actions to provide powerful and flexible advanced automation to your threat response tasks.

Automation rules

Automation rules are a new concept in Microsoft Sentinel. This feature allows users to centrally manage the automation of incident handling. Besides letting you assign playbooks to incidents (not just to alerts as before), automation rules also allow you to automate responses for multiple analytics rules at once, automatically tag, assign, or close incidents without the need for playbooks, and control the order of actions that are executed. Automation rules will streamline automation use in Microsoft Sentinel and will enable you to simplify complex workflows for your incident orchestration processes.

Learn more with this complete explanation of automation rules.

 Important

• The automation rules feature is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Playbooks

A playbook is a collection of response and remediation actions and logic that can be run from Microsoft Sentinel as a routine. A playbook can help automate and orchestrate your threat response, it can integrate with other systems both internal and external, and it can be set to run automatically in response to specific alerts or incidents, when triggered by an analytics rule or an automation rule, respectively. It can also be run manually on-demand, in response to alerts, from the incidents page.

Playbooks in Microsoft Sentinel are based on workflows built in Azure Logic Apps, a cloud service that helps you schedule, automate, and orchestrate tasks and workflows across systems throughout the enterprise. This means that playbooks can take advantage of all the power and customizability of Logic Apps’ integration and orchestration capabilities and easy-to-use design tools, and the scalability, reliability, and service level of a Tier 1 Azure service.

Learn more with this complete explanation of playbooks.

Next steps

Identify advanced threats with User and Entity Behavior Analytics (UEBA) in Microsoft Sentinel

What is User and Entity Behavior Analytics (UEBA)?

Identifying threats inside your organization and their potential impact – whether a compromised entity or a malicious insider – has always been a time-consuming and labor-intensive process. Sifting through alerts, connecting the dots, and active hunting all add up to massive amounts of time and effort expended with minimal returns, and the possibility of sophisticated threats simply evading discovery. Particularly elusive threats like zero-day, targeted, and advanced persistent threats can be the most dangerous to your organization, making their detection all the more critical.

The UEBA capability in Microsoft Sentinel eliminates the drudgery from your analysts’ workloads and the uncertainty from their efforts, and delivers high-fidelity, actionable intelligence, so they can focus on investigation and remediation.

As Microsoft Sentinel collects logs and alerts from all of its connected data sources, it analyzes them and builds baseline behavioral profiles of your organization’s entities (such as users, hosts, IP addresses, and applications) across time and peer group horizon. Using a variety of techniques and machine learning capabilities, Microsoft Sentinel can then identify anomalous activity and help you determine if an asset has been compromised. Not only that, but it can also figure out the relative sensitivity of particular assets, identify peer groups of assets, and evaluate the potential impact of any given compromised asset (its “blast radius”). Armed with this information, you can effectively prioritize your investigation and incident handling.

UEBA analytics architecture

Security-driven analytics

Inspired by Gartner’s paradigm for UEBA solutions, Microsoft Sentinel provides an “outside-in” approach, based on three frames of reference:

• Use cases: By prioritizing for relevant attack vectors and scenarios based on security research aligned with the MITRE ATT&CK framework of tactics, techniques, and sub-techniques that puts various entities as victims, perpetrators, or pivot points in the kill chain; Microsoft Sentinel focuses specifically on the most valuable logs each data source can provide.
• Data Sources: While first and foremost supporting Azure data sources, Microsoft Sentinel thoughtfully selects third-party data sources to provide data that matches our threat scenarios.
• Analytics: Using various machine learning (ML) algorithms, Microsoft Sentinel identifies anomalous activities and presents evidence clearly and concisely in the form of contextual enrichments, some examples of which appear below.

Microsoft Sentinel presents artifacts that help your security analysts get a clear understanding of anomalous activities in context, and in comparison with the user’s baseline profile. Actions performed by a user (or a host, or an address) are evaluated contextually, where a “true” outcome indicates an identified anomaly:

• across geographical locations, devices, and environments.
• across time and frequency horizons (compared to user’s own history).
• as compared to peers’ behavior.
• as compared to organization’s behavior.

Scoring

Each activity is scored with “Investigation Priority Score” – which determine the probability of a specific user performing a specific activity, based on behavioral learning of the user and their peers. Activities identified as the most abnormal receive the highest scores (on a scale of 0-10).

See how behavior analytics is used in Microsoft Defender for Cloud Apps for an example of how this works.

Entity Pages

Learn more about entities in Microsoft Sentinel and see the full list of supported entities and identifiers.

When you encounter a user or host entity (IP address entities are in preview) in an entity search, an alert, or an investigation, you can select the entity and be taken to an entity page, a datasheet full of useful information about that entity. The types of information you will find on this page include basic facts about the entity, a timeline of notable events related to this entity and insights about the entity’s behavior.

Entity pages consist of three parts:

• The left-side panel contains the entity’s identifying information, collected from data sources like Azure Active Directory, Azure Monitor, Microsoft Defender for Cloud, CEF/Syslog, and Microsoft 365 Defender.
• The center panel shows a graphical and textual timeline of notable events related to the entity, such as alerts, bookmarks, and activities. Activities are aggregations of notable events from Log Analytics. The queries that detect those activities are developed by Microsoft security research teams, and you can now add your own custom queries to detect activities of your choosing.
• The right-side panel presents behavioral insights on the entity. These insights help to quickly identify anomalies and security threats. The insights are developed by Microsoft security research teams, and are based on anomaly detection models.

 Note

The IP address entity page (now in preview) contains geolocation data supplied by the Microsoft Threat Intelligence service. This service combines geolocation data from Microsoft solutions and third-party vendors and partners. The data is then available for analysis and investigation in the context of a security incident. For more information, see also Enrich entities in Microsoft Sentinel with geolocation data via REST API (Public preview).

The timeline

The timeline is a major part of the entity page’s contribution to behavior analytics in Microsoft Sentinel. It presents a story about entity-related events, helping you understand the entity’s activity within a specific time frame.

You can choose the time range from among several preset options (such as last 24 hours), or set it to any custom-defined time frame. Additionally, you can set filters that limit the information in the timeline to specific types of events or alerts.

The following types of items are included in the timeline:

• Alerts – any alerts in which the entity is defined as a mapped entity. Note that if your organization has created custom alerts using analytics rules, you should make sure that the rules’ entity mapping is done properly.
• Bookmarks – any bookmarks that include the specific entity shown on the page.
• Activities – aggregation of notable events relating to the entity. A wide range of activities are collected automatically, and you can now customize this section by adding activities of your own choosing.

Entity Insights

Entity insights are queries defined by Microsoft security researchers to help your analysts investigate more efficiently and effectively. The insights are presented as part of the entity page, and provide valuable security information on hosts and users, in the form of tabular data and charts. Having the information here means you don’t have to detour to Log Analytics. The insights include data regarding sign-ins, group additions, anomalous events and more, and include advanced ML algorithms to detect anomalous behavior.

The insights are based on the following data sources:

• Syslog (Linux)
• SecurityEvent (Windows)
• AuditLogs (Azure AD)
• SigninLogs (Azure AD)
• OfficeActivity (Office 365)
• BehaviorAnalytics (Microsoft Sentinel UEBA)
• Heartbeat (Azure Monitor Agent)
• CommonSecurityLog (Microsoft Sentinel)
• ThreatIntelligenceIndicators (Microsoft Sentinel)

How to use entity pages

Entity pages are designed to be part of multiple usage scenarios, and can be accessed from incident management, the investigation graph, bookmarks, or directly from the entity search page under Entity behavior analytics in the Microsoft Sentinel main menu.

Entity page information is stored in the BehaviorAnalytics table, described in detail in the Microsoft Sentinel UEBA enrichments reference.

Querying behavior analytics data

Using KQL, we can query the Behavioral Analytics Table.

For example – if we want to find all the cases of a user that failed to sign in to an Azure resource, where it was the user’s first attempt to connect from a given country, and connections from that country are uncommon even for the user’s peers, we can use the following query:KustoCopy

BehaviorAnalytics
| where ActivityType == "FailedLogOn"
| where ActivityInsights.FirstTimeUserConnectedFromCountry == True
| where ActivityInsights.CountryUncommonlyConnectedFromAmongPeers == True

User peers metadata – table and notebook

User peers’ metadata provides important context in threat detections, in investigating an incident, and in hunting for a potential threat. Security analysts can observe the normal activities of a user’s peers to determine if the user’s activities are unusual as compared to those of his or her peers.

Microsoft Sentinel calculates and ranks a user’s peers, based on the user’s Azure AD security group membership, mailing list, et cetera, and stores the peers ranked 1-20 in the UserPeerAnalytics table. The screenshot below shows the schema of the UserPeerAnalytics table, and displays the top eight-ranked peers of the user Kendall Collins. Microsoft Sentinel uses the term frequency-inverse document frequency (TF-IDF) algorithm to normalize the weighing for calculating the rank: the smaller the group, the higher the weight.

You can use the Jupyter notebook provided in the Microsoft Sentinel GitHub repository to visualize the user peers metadata. For detailed instructions on how to use the notebook, see the Guided Analysis – User Security Metadata notebook.

Permission analytics – table and notebook

Permission analytics helps determine the potential impact of the compromising of an organizational asset by an attacker. This impact is also known as the asset’s “blast radius.” Security analysts can use this information to prioritize investigations and incident handling.

Microsoft Sentinel determines the direct and transitive access rights held by a given user to Azure resources, by evaluating the Azure subscriptions the user can access directly or via groups or service principals. This information, as well as the full list of the user’s Azure AD security group membership, is then stored in the UserAccessAnalytics table. The screenshot below shows a sample row in the UserAccessAnalytics table, for the user Alex Johnson. Source entity is the user or service principal account, and target entity is the resource that the source entity has access to. The values of access level and access type depend on the access-control model of the target entity. You can see that Alex has Contributor access to the Azure subscription Contoso Hotels Tenant. The access control model of the subscription is Azure RBAC.

You can use the Jupyter notebook (the same notebook mentioned above) from the Microsoft Sentinel GitHub repository to visualize the permission analytics data. For detailed instructions on how to use the notebook, see the Guided Analysis – User Security Metadata notebook.

Hunting queries and exploration queries

Microsoft Sentinel provides out-of-the-box a set of hunting queries, exploration queries, and the User and Entity Behavior Analytics workbook, which is based on the BehaviorAnalytics table. These tools present enriched data, focused on specific use cases, that indicate anomalous behavior.

For more information, see:

• Hunt for threats with Microsoft Sentinel
• Visualize and monitor your data

As legacy defense tools become obsolete, organizations may have such a vast and porous digital estate that it becomes unmanageable to obtain a comprehensive picture of the risk and posture their environment may be facing. Relying heavily on reactive efforts, such as analytics and rules, enable bad actors to learn how to evade those efforts. This is where UEBA comes to play, by providing risk scoring methodologies and algorithms to figure out what is really happening.

Notebook Environment Setup

This notebook takes you through detailed setup of your settings for Microsoft Sentinel Notebooks and the MSTICPy library. It covers:

• Setting up your Python environment for notebooks
• Creating and editing your msticpyconfig.yaml file
• Understanding and managing you config.json file.

If you are using notebooks in the Microsoft Sentinel/Azure ML environment you can skip the first section “Configuring your Python Environment” entirely.

The main part of this notebook involves setting up your msticpyconfig.yaml. While many of these settings are optional, if you do not configure them correctly you’ll experience some loss of functionality. For example, using Threat Intelligence providers usually requires an API key. To save you having to type this in every time you look up an IP Address you should put this in a config file.

This section takes you through creating settings for

• Microsoft Sentinel workspaces
• Threat Intelligence providers
• Geo-location providers
• Other data providers (e.g. Azure APIs)
• Key Vault
• Auto-loading options.

You’ll typically need the first three of these to use most of the notebooks fully.

Section 3, “The config.json file” can also be ignored if you are happy using msticpyconfig.yaml. It is included here for background.

<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Strict//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd”>