Last updated
Last updated
Cloud Insights allows you to discover your cloud infrastructure and monitor the flow of traffic within or across your virtual private clouds. The discovered cloud infrastructure and traffic flow is time-correlated with your Cloud and Enterprise Agent views to combine performance measurements captured from your Cloud or Enterprise Agents with cloud infrastructure configuration changes and traffic flow. To learn more about how Cloud Insights can work for you, see .
In this article, we’ll explain how to create the integrations you need with Amazon Web Services (AWS) using their Console to get your ThousandEyes Cloud Insights up and running. For instructions for creating the integrations using the AWS Command Line Interface (CLI), see . AWS for Cloud Insights consists of two separate integrations per each AWS account you want to monitor. Inventory Monitoring is mandatory and allows you to monitor your AWS account assets, topology and config changes. Flow Logs Monitoring is an optional add-on and allows you to monitor your AWS account traffic using the AWS VPC (Virtual Private Cloud) flow logs.
You must have the following AWS account and ThousandEyes organization requirements and permissions to successfully set up your Cloud Insights integrations.
To use the features described in this document, you must have both of the following:
An active account for Amazon Web Services.
An active organization for ThousandEyes.
For the , you need AWS permissions to create roles and policies. You can find the complete list of permissions within the permission policy that ThousandEyes generates for you at step 6 of below. For more information about AWS permissions, see .
For the , you need the above, plus (if not already permitted):
to enable/configure VPC flow logs.
to create the S3 (Simple Storage Service) flow logs bucket and enable event notification to the SNS topic.
to create an SNS (Simple Notification Service) topic to which the S3 events will be sent and to create the role for ThousandEyes to register with.
We currently support the onboarding of up to 200 AWS accounts, where each account is integrated across up to 17 regions (AWS enables up to 17 regions per account by default). If you need to onboard more accounts or more regions per account, please raise a support ticket with us.
To take full advantage of Cloud Insights, you need to create two distinct integrations which capture two types of data and serve different purposes. These are:
Inventory monitoring is used to collect inventory and configuration information from your AWS accounts over time. We use this data in Cloud Insights to:
Display your AWS network assets, including their types and locations within Cloud Insights > Inventory.
Track asset configuration changes over time in the form of events, viewable in both Cloud Insights > Inventory and Cloud Insights > Views.
Display within Network & App Synthetics > Views your cloud network topology and events.
Enrich the flow log data with resource information once the flow logs integration is set up.
Flow logs monitoring is used to track real traffic flows in your AWS network. We use this data, for example, to:
Display outbound/inbound throughput for each entity in Cloud Insights > Views and Network & App Synthetics > Views.
Highlight rejected traffic, visible in Cloud Insights > Views.
In ThousandEyes, each organization is divided into account groups. The Cloud Insights integrations are account group-specific. This means that an integration created in one account group is not shared with another account group. Each account group can set up integrations to monitor AWS accounts in your AWS network.
The monitored AWS accounts in your network can span one or more regions. The flow-logs used for Cloud Insights are fetched per AWS location, which is a combination of the AWS account and region, for example: 351945360856, us-west-1.
In the image below, three different account groups within a ThousandEyes organization each monitor one or more AWS accounts. Each monitored AWS account spans one or more regions. Account Group 1 monitors three AWS locations (two AWS accounts, the first in two regions and the second in one region), Account Group 2 monitors one location (a single AWS account in one region), and Account Group 3 monitors two locations (a single AWS account in two regions).
ThousandEyes uses inventory monitoring integrations to access the AWS accounts that you want to monitor with Cloud Insights. These monitored accounts may also contain VPCs that generate traffic flow information via AWS flow logs. The logs from the VPCs are published to S3 buckets, which can reside in the same or different AWS accounts. You configure event notifications for these buckets to be sent to SNS topics, which we subscribe to be alerted whenever a new flow log files is available.
The purpose of the flow logs integrations is to access the log files in these buckets for ingestion by ThousandEyes. You should aim to surface only traffic from accounts and VPCs explicitly selected for monitoring through inventory integrations. Other traffic will be filtered out or marked as external traffic.
Therefore, for Cloud Insights to work optimally, you need to identify the AWS accounts you want to monitor and create inventory monitoring integrations for them. You then locate the S3 buckets where the corresponding VPC flow logs are stored (or create them) and set up the flow logs monitoring integrations accordingly.
Flow logs are typically published from VPCs in a given region to an S3 bucket within the same region to avoid extra AWS fees associated with cross-region data transfer. Customers with multiple accounts across various regions may choose to create a local bucket for each account in each region, as in the image above. In this example, because separate integrations are needed per account containing flow logs, the customer would need to set up two flow logs monitoring integrations.
Alternatively, customers can send flow logs from monitored accounts to a separate account with buckets in each region, potentially reducing the number of accounts and integrations that need to be managed, as in the image below. In this example, because all the flow logs are contained within one account, the customer only needs to set up one flow logs monitoring integration.
Flow logs monitoring itself is done in two steps. These steps are:
ThousandEyes receives notification of a new flow log via subscription to your SNS topic.
In your AWS network, event notifications are published from the monitored flow logs bucket to an SNS topic in the same region.
We subscribe to your SNS topic to get a notification whenever a new flow log is created.
ThousandEyes downloads the flow log files from your S3 bucket.
You create an IAM Role that we can assume and use to fetch the flow log files from your S3 bucket.
This means that for each monitored AWS account, we need access to both the S3 buckets that contain the flow logs and the SNS topics that alert us to their existence.
The image below shows the flow logs:
Originating in your VPC.
Getting sent to the S3 bucket for storage.
Triggering an event notification into an SNS topic.
ThousandEyes receiving the event notification through a subscription to that topic.
ThousandEyes then retrieving the flow logs from the S3 bucket.
Each integration connects us to a single account where your flow log buckets are stored. For each integration:
We need an SNS topic per region (in the same account) to get notifications of new flow logs.
We need an IAM Role to enable read-only access to the buckets.
The integration for AWS inventory monitoring gives ThousandEyes secure access to your AWS account information and data.
To set up inventoring monitoring:
Go to Manage > Integrations.
Click + New Integration in the top right.
In the Add New Integration side panel that opens, select Amazon Web Services.
The resulting < Add AWS Integration screen defaults to showing the fields required for the “Test Recommendations” service.
Select “Inventory Monitoring” from the ThousandEyes Supported Services dropdown.
Name your integration.
Give your integration a unique name. Duplicate names are not permitted.
You must then use the generated trust and permission policies to create the IAM Role that will give ThousandEyes access to your AWS account.
At the end of this process, you should have pasted the ARN for the IAM Role into the corresponding field in the ThousandEyes < Add AWS Integration screen.
Note: The trust policy for the inventory monitoring integration will look the same as the trust policy for the test recommendations integration. However, the permission policies differ. Do not paste a permission policy from a test recommendation integration into an IAM Role intended for an inventory monitoring integration. Selecting the “Inventory Monitoring” service automatically updates the code (or commands, depending on which interface you choose to create the integration) in the required permission policy.
Click Test.
Note: The Test function only validates the trust relationship between AWS and ThousandEyes; it does not validate the permission policy.
If testing was successful, click Save.
Before you begin to create the integration, you first need to have several items set up within your AWS account for the integration to be a success. Follow the links in each step to find instructions within AWS.
Under the Flow log settings, set:
Filter to “All” to see accepted and rejected traffic flows.
Maximum aggregation interval to either 10 minutes or 1 minute, depending on the granularity of data you need. Flow logs with a maximum aggregation interval of 1 minute produce a higher volume of flow log records than flow logs with a maximum aggregation interval of 10 minutes.
Destination to “Send to an Amazon S3 bucket”.
S3 bucket ARN to the bucket where you’d like the flow logs to be published from this VPC – you retrieve this from step 1, above.
Log record format to custom (because not all the fields use the default setting), and containing the following fields as mandatory:
<account-id>
<action>
<bytes>
<dstaddr>
<dstport>
<end>
<flow-direction>
<interface-id>
<log-status>
<packets>
<protocol>
<srcaddr>
<srcport>
<start>
<tcp-flags>
<traffic-path>
<version>
All other fields are optional, while users can include any Standard attributes.
Repeat this step for every VPC you want to monitor in this region.
Note: It is typically best practice to configure all your VPCs in a particular region to publish flow logs to the same regional S3 bucket to avoid cross-region traffic costs.
Select “Standard” under Type of topic.
Skip Step 1: Create an Amazon SQS queue; it is not required for this integration.
In the Event types section choose “All object create events (s3:ObjectCreated:*)”.
In the Destination section, choose “SNS topic” and enter or select the SNS topic ARN where you want to send the notifications, retrieved from step 3, above.
You will come back to some of these steps as you create the integration in the next section, so it’s best to have both accounts open during the following steps.
This not only updates the permission policy for the flow logs integration and creates the IAM role that will give ThousandEyes access to your S3 bucket resources, but reveals the additional fields required to successfully implement this integration.
Update the access policy of your SNS topics to allow ThousandEyes to subscribe to these topics.
In each one, click Edit and open the Access policy dropdown to reveal configuration options.
Under the Advanced option, paste into the JSON editor the Access Policy generated within the ThousandEyes < Add AWS Integration screen.
Replace <TOPIC-ARN> with the ARN of your specific SNS topic.
Add all relevant SNS topic ARNs to the fields at the bottom of the < Add AWS Integration screen. You can add as many as necessary for each account.
Click + Add SNS Topic ARN to add a topic.
Click the “minus” sign to the right of each topic to remove it.
Click Test.
Note: The Test function only validates the trust relationship between AWS and ThousandEyes; it does not validate the permission policy.
If testing was successful, click Save.
After saving your integrations, ThousandEyes will start monitoring the resources specified in their policies. During this process, each integration status displays as “Pending”.
Inventory and flow logs are monitored every five minutes so you can expect to see the "Pending" state updated within five to ten minutes. Any flow log errors will be detected once we receive flow log files. If we encounter any issue that causes the monitoring to be unsuccessful, we will surface the error on the integration screen and move the status to “Failed”. In most cases, you need to fix the permissions and click Save to restart the monitoring.
Some inventory monitoring integrations might display the "Connected" state but with a red warning symbol alongside it.
Constantly seeing a warning message next to your integration status could be disconcerting, and what's more, you may be quite aware that certain AWS resources are unavailable to ThousandEyes. Therefore, you can remove the warning symbol in the following way:
Navigate to Cloud Insights > Settings.
Open the Integration Policies tab.
Uncheck any of the resource groups or regions that ThousandEyes doesn't or shouldn't have access to. Resource names between AWS and the Integration Policies tab are the same or similar.
Note: You cannot disable the "EC2 (including load balancing)" resource as this is essential for the integration to work.
Click Save Changes.
Similar to Inventory Monitoring integrations, Flow Logs Monitoring integrations can also enter a partially connected state. In this state, the integration will appear as ‘Connected’ but will display a red warning symbol to indicate an issue.
In this state, the integration is connected, but errors have been encountered, preventing some flow logs from being successfully ingested or processed.
The most common errors you may encounter are:
Failure to subscribe to one or more of your configured SNS topics. If ThousandEyes fails to subscribe to all of the topics, the integration will be shown as "Failed" and no flow logs will be ingested. If we are able to subscribe to at least one topic, the integration will move to the partially connected state.
Failure to process incoming flow logs. In this case, we successfully subscribed to the SNS topics defined in the integration and we receive flow log notifications from the topics. However, we encounter errors with downloading or parsing some or all of the incoming flow logs.
The Integration Logs tab under Cloud Insights > Settings provides detailed information about important events associated with each integration. These events can include informational messages, such as successfully subscribing to an SNS topic, or error messages, such as failures to subscribe to an SNS topic or to download a log file. You may use these logs to identify and troubleshoot integration errors.
When you open the Integration Logs screen, a summary table appears with each row displaying the integration name, type, category, and severity for events from a given hour (if any exist). At the top of the page, you’ll find various filters that allow you to narrow down the information displayed. For example, you can filter to show only errors for a specific integration name.
Clicking the arrow button on the right side of a row opens a side panel displaying detailed information about each event that occurred.
The above image shows a failure to subscribe error due to missing permissions. The specific error is displayed under the Details column.
ThousandEyes unable to assume a role for your monitored account:
If ThousandEyes cannot assume the IAM role for your monitored AWS account, you will encounter the following error if you try to either test or save the integration.
Note: if you disable the trust policy in AWS at some point, but do not delete the integration within ThousandEyes, the integration will transition out of the status of Connected, and will not resolve back to Connected until you re-instate the trust policy.
Cannot duplicate the integration:
Each inventory monitoring integration is associated with a unique role ARN, and the same role ARN cannot be used across multiple inventory integrations per account group (note that role ARNs can be used for corresponding inventory monitoring and flow logs monitoring integrations). If you try to duplicate an inventory monitoring integration using the same role ARN as an existing inventory monitoring integration, you will encounter the following error and be unable to save the new integration:
Cannot access certain AWS resources:
The permission policy lists a number of API types to which you grant ThousandEyes read access. You define in AWS which specific APIs fall into each API type. If you decide to limit ThousandEyes’ access to a subset of APIs within these types, your integration will still work, but you will encounter error messages in the sidebar describing which APIs we have been denied access to (note, each error message is specific to the AWS resource being restricted – the message below is an example):
You can restrict read access to other resources in AWS, such as regions. If you disallow read access to ThousandEyes for any regions, ThousandEyes will have access to all other regions, but you will receive an error message explaining that a region(s) was restricted:
Sometimes the error banners will appear accompanied by the message below directing the user to the Integration Logs tab under Cloud Insights > Settings for more detailed information.
Cannot duplicate the integration:
Each flow logs monitoring integration is associated with a unique role ARN, and the same role ARN cannot be used across multiple flow logs integrations per account group (note that role ARNs can be used for corresponding inventory monitoring and flow logs monitoring integrations). If you try to duplicate a flow logs monitoring integration using the same role ARN as an existing flow logs integration, you will encounter the following error and will not be able to save the new integration:
SNS topic ARNs belong to a different account than the role ARN:
Each flow logs monitoring integration is associated with a single AWS account, which corresponds to the account of the role ARN. The AWS account of the SNS topics specified in the integration must match the account of the role ARN. If you attempt to create an integration where any of the topics belong to a different account than the role ARN, you will encounter the following error and will not be able to save the integration. In this example, the second SNS topic ARN belongs to a different account than the role ARN.
SNS topic ARN appears in more than one integration:
The same SNS topic ARN cannot appear in more than one integration. If a duplicate topic ARN is detected, you will encounter the following error if you try to either test or save the integration. The message includes the ARN of the topic(s) which is already defined in another integration.
ThousandEyes is unable to assume a role for accessing your S3 buckets:
If ThousandEyes cannot assume the IAM role provided in the integration, you will encounter the following error if you try to either test or save the integration.
ThousandEyes is unable to subscribe to SNS topics:
Once an integration is configured and saved, ThousandEyes will attempt to subscribe to the topics listed in the integration. If subscribing to all of the configured topics fail, the integration status will change to "Failed" state. If at least one subscription attempt is successful, the integration will change to a connected state. In either case the following error message will be displayed in the integration sidebar with the list of topics for which the subscription failed.
The common causes of subscription failures include:
A topic with the specified ARN doesn’t exist.
The topic’s access policy is missing permissions to allow ThousandEyes to subscribe to the topic.
The Log Details page for the integration displays a message of category subscribe with the specific subscription error message for each topic, as shown in the image below.
When the integration state changes to “Connected”, it indicates that ThousandEyes is able to assume the IAM role and that it has successfully subscribed to at least one of the topics in the integration.
ThousandEyes is now receiving messages from the subscribed topics. This section outlines potential errors related to the retrieval and processing of incoming flow logs messages. If such errors were seen in the last 30 minutes the following error banner will be shown in the corresponding integration:
If you do not see flow logs traffic from your monitored accounts, consider checking for the errors listed below.
There is no associated monitored account:
The Log Details page for the integration displays a warning message of category logFilter for each log file that has been filtered. The Details column provides the S3 bucket and object key of where the log file is stored.
Flow logs aren’t downloading from the user’s S3 buckets:
The Log Details page for the integration displays an error message of category logDownload for each log file which failed to download. The Details column provides the S3 bucket and object key of where the log file is stored. The Error field contains the download error encountered.
Mandatory log format fields are missing:
$(flow-direction)
$(tcp-flags)
$(traffic-path)
You must choose the custom setting for the log record format to add the additional fields. If any of the default or additional fields are missing, Cloud Insights will not process the logs.
You can verify the current log format by checking your AWS flow logs table for your VPCs.
Note: It is not possible to add or remove fields from an existing flow log configuration. To make changes, you must create a new flow log configuration with the required fields and then remove the old one.
The Log Details page for the integration displays an error message of category logParse for each log file which we failed to parse. The Details column specifies the S3 bucket and object key where the log file is stored. The Error field contains the parsing error encountered. In the image below, parsing the log file failed due to missing mandatory fields.
To create the integrations, you must have an Organization Admin or Account Admin role in the ThousandEyes platform. For more information about ThousandEyes roles, see .
You do this the same way as described in .
Follow steps 1-5 under , except don’t yet test the integration.
See for information about what happens after you save your integration.
to collect flow logs in each region that your account covers in the .
Set your to this bucket in the .
in each AWS region where you are monitoring a bucket in the .
for each regional SNS topic in the .
Follow steps 1-6 in , except choose "Flow Logs Integration" from the ThousandEyes Supported Services dropdown.
In your AWS account, find the SNS topics you created in step 3 of for each region in the .
This means that, of the permissions ThousandEyes requested through the permission policy (see for the complete list of permissions requested), which are data resources accessed via APIs, some permissions were not granted. This is often due to those permissions being disabled within your AWS account. The integration remains usable, but some data will not show up in either your Cloud Insights > Views or Network & App Synthetics > Views.
To see which permissions were not granted (i.e., APIs given no access), open the partially connected integration. Error messages appear at the top of the resulting side panel informing you of which API resources have not been granted. See #3 in for example error messages.
In the image above, we successfully subscribed to three out of five SNS topics, while the remaining two failed. Since there is a mix of subscribed and unsubscribed topics, the integration is considered partially connected. The error message shows the list of topics to which we failed to subscribe. To find detailed information about the failures, please visit the Integration Logs tab under Cloud Insights > Settings. Please refer to in the troubleshooting section for additional information on resolving these issues.
If flow log processing errors occur within the last 30 minutes, the integration will transition to a partially connected state and display an error message. For detailed information about these errors, navigate to the Integration Logs tab under Cloud Insights > Settings. Please refer to in the troubleshooting section for additional information on resolving these issues.
Either a role with this ARN doesn't exist, or the role's trust policy is not configured to allow ThousandEyes to assume the role. In the latter case, you must re-paste the trust policy into AWS, as outlined in step 3 under .
Important: When you deny ThousandEyes access to some AWS resources (but not all), the integration status shows as Connected since all other resources granted read access permission will continue to work as part of the integration. However, the Connected status is accompanied by a red warning symbol. You must click open the integration's editing sidebar to see the error messages listing out which APIs you have denied us access to. Due to space, only a certain number of errors reported are displayed in the sidebar. To remove the warning symbol, see .
Common errors you may encounter while setting up the flow logs integration in ThousandEyes are explained below. For common errors when configuring your AWS VPCs, S3 buckets, and SNS topics or events, visit the relevant topics at .
Either a role with this ARN doesn't exist, or the role's trust policy is not configured to allow ThousandEyes to assume the role. In the latter case, you must re-paste the trust policy into AWS, as outlined in step 3 within .
To resolve this error, update the access policy of the SNS topic as outlined in step 2 of .
ThousandEyes will only consume and process flow logs from accounts that have an associated inventory monitoring integration. Ensure that you create inventory monitoring integrations for all accounts you wish to monitor using Cloud Insights. See for more information.
When ThousandEyes receives a notification about a new flow log, it will attempt to download the file from the S3 bucket using the IAM role specified in the integration. The most common reason for download failures is improper configuration of the role’s permission policy. To resolve this, update the role’s permission policy as outlined in step 2 within .
Step 2 of lists the mandatory fields that must be contained within the log format. All but the following three fields are part of the default format:
Now that you have completed your Cloud Insights integrations, see to learn how to view and manage your Cloud Insights data.
