With these integrations, #LetsData will now:

• Crawl Data: Automatically configure and run AWS Glue Crawlers on a dataset’s S3 files

• Discover Schema: Discover the file data schema, partitioning and metadata

• Manage Data Catalogs: Add the schema, partitioning and metadata to the AWS Glue Data Catalog

• Data Governance: Automatically configure access and permissions for the catalog and S3 files

• Data Warehousing: Enable Distributed Querying support via AWS Athena.

The details of the S3 Write Connector Data Catalog integrations are at our Data Catalog Docs

Overview

The LetsData Write Destinations are categorized into either:

• permanent, durable data store (think databases such as S3, Dynamo DB, Vector Indexes)

• ephemeral data containers (such as streams, queues etc)

The permanent, durable data stores can be enabled for analytical workloads (OLAP). LetsData enables these by running AWS Glue Crawlers on these data stores to discover the data schema, partitioning and metadata. These are then added to data catalog / meta stores (AWS Glue Data Catalog). Customers can run AWS Athena queries using the AWS Glue Data Catalog - these are distributed queries that enable on demand, massively parallel processing of large datasets.

See  AWS Athena Docs to understand the AWS analytics services and their usage,  Redshift Docs about Columnar Storage to understand OLAP / OLTP differences.

Highlights

Highlights about LetsData's Data Catalog implementation:

• Automatic Data Cataloging: LetsData can automatically catalog datasets that write to S3 destinations by running glue crawlers on S3 data files and creating databases and tables for the LetsData users.

• Primitive Data Lake: Adding to a Data Catalog, defining a permissioning model for access and enabling Distributed Queries is essentially forming a Data Lake. (There are different terms that one comes across when defining a data strategy for OLAP workloads - Data Warehouses, Data Lakes and Data Lakehouse etc. This  external link explains the different terminology). The LetsData's data catalog is primitive data lake (not a datawarehouse).

• Access and Permissioning: The AWS Lake Formation is a permissioning model over AWS Glue Catalog and vends automated temporary credentials to different services - Lake Formation has different user roles and allows for permissions delegation and sharing as well. As of now, LetsData permissioning model allows data catalog access to data owner only and we do not offer permission sharing / permissions delegation yet.

Architecture

Automated Data Catalog

Here is a high level architecture of how automated data catalog is implemented:

Distributed Querying

Here are a high level steps to query a dataset's data catalog tables.

Details

Availability

Automated Data Catalog is currently available for S3 Write Connectors only (S3, S3 Aggregate File, S3 Spark). Automated Data Catalog is currently available for resourceLocation: LetsData only. resourceLocation: Customer requires some additional architectural redesign which we've deferred for now.

Enable Config

To enable automated Data Catalog, add the "addToCatalog": true attribute to the dataset's write connector config.

Catalog Details

The dataset's initialization creates the catalog database, the glue crawler and sets up the crawl configuration. These dataset's write connector node is updated with these catalog details. You'll need these in Athena queries. Use the CLI datasets view command to view the dataset details. Here are the example details:

Here are what the different attributes in the configuration mean:

• awsGlueCrawlerName: The name of the AWS Glue crawler that was created by dataset. You can use aws cli / sdk to query details about the crawler and its crawl runs. See additional details in the crawler section below.

• awsGlueCatalogName: The name of the AWS Glue catalog that the crawler will store the schema in. This is currently set to the default AWS account catalog. You'll need this to query the catalog details about the tables and when running actual athena queries.

• awsGlueDatabaseName: The name of the AWS Glue catalog database. Every LetsData user gets their own database accessible only to them. The database is named as <stack_name>tenant<tenant_id>. For example tenant with tenantId (d5feaf90-71a9-41ee-b1b9-35e4242c3155) on the prod stack would be assigned a database prodtenantd5feaf90-71a9-41ee-b1b9-35e4242c3155

• awsGlueTableNamePrefix: The crawler sets up the crawl configuration and specifies the tableName prefix. The actual tableName(s) are obtained later by calling the AWS Glue GetTables API using credentials from the the dataset's customerAccessRoleArn and awsGlueCatalogName / awsGlueDatabaseName as the catalog name / database name respectively.

• athenaQueryOutputPath: To help with querying that works without any additional setup requirements, we provisioned an S3 Bucket path for each dataset configured for Athena queries for that dataset. You can run Athena queries with the S3 Bucket path as the results output path. The contents of the path are deleted upon dataset deletion (so do save any results as needed).

Crawler Configuration

We've configured the crawler by default with settings that should work out of the box for most use cases. However, for those who'd need advanced options, here is how the crawler is setup:

• re-crawl everything on each run

• schema change policies of update / delete in database on updates / deletes to schema

• crawler lineage settings are disabled.

• tableGroupingPolicy to combine compatible schemas (and create a single table)

If we should add some classifier support / additional options out of the box, do let us know we can enable these (mailto: support@letsdata.io). ( Crawler in AWS Docs)

Here is an example crawler configuration:

Running Athena Queries

Here are a few example commands that can be used to get details about a dataset's catalog and run athena queries.

Access Credentials

Get dataset's access credentials:

• We need the customerAccessRoleArn, createDatetime from the dataset to be able to access the bucket files.

CUSTOMER_ACCESS_ROLE_ARN=`./letsdata datasets view --datasetName <dataset_name> --prettyPrint 2> /dev/null | grep customerAccessRoleArn| sed 's/.* "//g'| sed 's/",//g'`

EXTERNAL_ID=`./letsdata datasets view --datasetName <dataset_name> --prettyPrint 2> /dev/null | grep createDatetime|sed 's/^.*: //g'|sed 's/,//g'`                    

• Suppose that for the current dataset, the customerAccountForAccess is 308240606591. This has an IAM Admin user for this account whose credentials are stored in the ~/.aws/credentials file as the profile IamAdminUser308240606591. Run the following AWS CLI command to get time limited credentials and save them to the ~/.aws/credentials file in the stsassumerole node

output="/tmp/assume-role-output.json"

aws sts assume-role --role-arn $CUSTOMER_ACCESS_ROLE_ARN --external-id $EXTERNAL_ID --role-session-name 'AccessToDatasetCredentials' --profile IamAdminUser308240606591 > $output

cat $output

    # output should have contents similar to the following
    # {
    #    "Credentials": {
    #        "AccessKeyId": "ASIATIBD24ZTWCHOBDHH",
    #        "SecretAccessKey": "W09pNZg...",
    #        "SessionToken": "FwoG...",
    #        "Expiration": "2024-02-08T01:29:20+00:00"
    #    },
    #    "AssumedRoleUser": {
    #        "AssumedRoleId": "AROATIBD24ZTZWJR5CZEW..."
    #    }
    # }

# copy these the ~/.aws/credentials file    
cat ~/.aws/credentials

[IamAdminUser308240606591]
aws_region = us-east-1
aws_access_key_id = ASIATIBD24ZT3CRVCKVD
aws_secret_access_key = Bx8RVkAXfoGh5ERnL90ceftz5GJMWnxHx27uxTpS

[stsassumerole]
aws_region = us-east-1
aws_access_key_id = ASIATIBD24ZTWCHOBDHH
aws_secret_access_key = W09pNZg...
aws_session_token = FwoG...              

Crawler

Get the status of the crawler using get-crawler CLI command, start a new crawler run using the start-crawler and stop a running crawler using stop-crawler CLI commands. (The crawler name is in the dataset config as the writeConnector.catalog.awsGlueCrawlerName attribute)

  aws glue get-crawler --name testcrawler5855ca1062420abf35ed85b8c6eda82a --profile stsassumerole --region us-east-1

  aws glue start-crawler --name testcrawler5855ca1062420abf35ed85b8c6eda82a --profile stsassumerole --region us-east-1

  aws glue stop-crawler --name testcrawler5855ca1062420abf35ed85b8c6eda82a --profile stsassumerole --region us-east-1            

Table Details

The table(s) are created when the the crawler has run successfully. You can get the table details for the created tables using the get-tables CLI command. (The database name is in the dataset config as the writeConnector.catalog.awsGlueDatabaseName attribute). Additionally, you can get the database, partitions and partition indexes details as well.

  aws glue get-tables --database-name testtenantd5feaf90-71a9-41ee-b1b9-35e4242c3155 --profile stsassumerole --region us-east-1

  # This should return table details including the schema
  # {
  #     "TableList": [
  #         {
  #             "Name": "b8e4b3e8...",
  #             "DatabaseName": "testtenant...",
  #             ...
  #             "StorageDescriptor": {
  #                 "Columns": [
  #                     {
  #                         "Name": "language",
  #                         "Type": "string"
  #                     },
  #                     ...
  #                 ],
  #                 "Location": "s3://tldwc.../",
  #                 "InputFormat": "TextInputFormat",
  #                 "OutputFormat": "HiveIgnoreKeyTextOutputFormat",
  #                 "Compressed": true,
  #                 "NumberOfBuckets": -1,
  #                 "SerdeInfo": {
  #                     "SerializationLibrary": "JsonSerDe",
  #                     "Parameters": {
  #                         "paths": "samples,percentile,language"
  #                     }
  #                 },
  #                 "BucketColumns": [],
  #                 "SortColumns": [],
  #                 "Parameters": {
  #                     "sizeKey": "9953",
  #                     "objectCount": "2",
  #                     ...
  #                 },
  #                 "StoredAsSubDirectories": false
  #             },
  #             "PartitionKeys": [
  #                 {
  #                     "Name": "partition_0",
  #                     "Type": "string"
  #                 }
  #             ],
  #             "TableType": "EXTERNAL_TABLE",
  #             "Parameters": {
  #                 "sizeKey": "9953",
  #                 "objectCount": "2",
  #                 ...
  #             },
  #             ...
  #         }
  #     ]
  # }
  #
            

Run Athena Query

Using the schema from the get tables, construct a query to run. You can run the query using the start-query-execution CLI command. (The database name, catalog name is in the dataset config as the writeConnector.catalog.awsGlueDatabaseName, writeConnector.catalog.awsGlueCatalogName attribute. Additionally, use the writeConnector.catalog.athenaQueryOutputPath to store your results.)

  # run the query 
  aws athena start-query-execution --query-string \
 "SELECT language, COUNT(*) as recs \
 FROM b8e4b3e... \
 GROUP BY language \
 ORDER BY recs desc" \
 --query-execution-context Database=testtenant...,Catalog=223413462631 \
 --result-configuration OutputLocation=s3://test-.../athena-queries/ \ 
 --profile stsassumerole --region us-east-1

  # {
  #   "QueryExecutionId": "8ce557cd-39eb-414d-8cbd-104a1223b009"
  # }

  # use the query execution id to get the query execution details
  aws athena get-query-execution --query-execution-id 8ce557cd-39eb-414d-8cbd-104a1223b009 --profile stsassumerole --region us-east-1

  # use the query execution id to get the query results 
  aws athena get-query-results --query-execution-id 8ce557cd-39eb-414d-8cbd-104a1223b009 --profile stsassumerole --region us-east-1

  # list the athenaQueryOutputPath and download the results as needed
  aws s3 ls s3://test-.../commoncrawladdtocatalog19/ --profile stsassumerole --region us-east-1
  
  # copy the results file from S3
  aws s3 cp s3://test-.../athena-queries/8ce557cd-39eb-414d-8cbd-104a1223b009.csv . --profile stsassumerole --region us-east-1

Conclusions

With automated AWS Glue integrations and AWS Athena enablement:

• we’ve defined a primitive Data Cataloging, Data Warehousing and Data Governance capabilities on #LetsData

• the AWS Glue Data Catalog enables a large number of AWS Data scenarios such as Redshift Datawarehouse, EMR processing and Quicksight analytics to name a few

This is a huge space and while what we’ve built barely scratches the surface, it is quite powerful. We’d love to know the customer scenarios around Data Cataloging, Governance and Datawarehousing and how #LetsData may help solve some of these challenges.

The Control SDK is what the LetsData CLI and the LetsData Website call to orchestrate and manage data pipelines on LetsData. Now, we’ve made our REST API available publicly for customers. Customers can build automation and develop their apps on Lets Data using familiar REST API semantics.

The documentation and runnable postman collection are available on Postman (LetsData Control SDK on Postman). The permalink on the LetsData docs website is at https://www.letsdata.io/docs/sdk-control-api/ - however, the docs page is mostly a redirect to Postman.

The Control API

The Control SDK API at a high level has the following resources, some high level details and the different supported actions are as follows:

• Dataset:  Datasets are collection of data tasks grouped together as a logical entity. They can also be called Data jobs / Data tasks that the user needs to run. A dataset will have tasks for the work items in the dataset. The SDK has a number of APIs to manage datasets. For example, you can create datasets, list datasets, view datasets, manage the dataset's lifecycle (descale / freeze / delete), manage the dataset’s code artifacts and update a dataset’s compute configuration etc.

  • Datasets API on Postman

• Tasks: Tasks are the system's representation of a work item that is executed by the compute engine (Lambda, Sagemaker, Spark etc). When a task executes, it reads from the read destination, calls the user data handlers and then writes to the write destination. During this execution, a task emits metrics, logs and errors records. Each of these is a separate resource in the SDK and these resources tie in together with tasks (taskId). As of now, the SDK defines APIs for listing and filtering a dataset’s tasks, redriving error tasks and stopping tasks.

  • Tasks API on Postman

• Errors: Errors during the task execution in parsing records, record transformations or writing records are archived as error documents and classified as 'Record Errors' in Let's Data. The SDK supports the list errors API to list the errors for tasks and a view API to view each individual error record file. (Do note that the Let's Data infrastructure errors and unhandled exceptions are classified as 'Task Errors' and are handled differently. Error Handling docs have additional details: https://www.letsdata.io/docs/error-handling/)

  • Errors API on Postman

• Logs: Each task writes its logs to files which essentially become a task's execution trace that is useful in debugging and monitoring. The SDK supports a view API to view a task’s log file.

  • Logs API on Postman

• Metrics: Dataset execution comes with some system defined metrics dashboards that can be used to monitor a dataset’s progress, debug performance issues and take corrective actions. The SDK supports the view API to view different metrics dashboards for a dataset.

  • Metrics API on Postman

• Usage Records:  Datasets execution initializes different AWS resources - write connector (e.g. kinesis stream), the error connectors (e.g. S3 bucket) and different internal components such as queues, database tables, compute resources etc. We meter the usage of these resources for each dataset and create usage records. These usage records are then used in determining costs. The SDK supports an API to list these usage records.

  • Usage Records API on Postman

• VPC: Datasets that read / write to destinations where a cluster of machines is managed, Lets Data manages these machines in a Virtual Private Cloud. The VPC resource supports a list API to list a dataset's VPC details. The VPC resource also supports a 'vpcPeeringConnections' resource that has commands to accept (create) and delete VPC peering connections to the customer VPCs. 

  • VPC API on Postman

• Users:  A Tenant (company) in #Lets Data can create different user accounts (login credentials in #Lets Data) for different users in the company. These users can be assigned administrator / user roles and can run datasets individually. The SDK also supports user management APIs such as list, create, update and delete users.

  • Users API on Postman

• Costs: Costs are a collection of the company's the billing account information, the payment method on file, the pricing details, a list of invoices, their payment status and links to pay the invoice /download the invoices as a PDF file. The SDK supports an API to list these cost details.

  • Costs API on Postman

Setup

To start using the LetsData Control SDK, you need to:

• Login Credentials: Have a valid LetsData account (username and a password). You can sign-up for a LetsData account at: https://www.letsdata.io/#signup

• ClientId: For any serious calls, you'll need a valid Client Id to use the LetsData Control SDK. You can request one by emailing at support@letsdata.io or logging an issue at https://www.letsdata.io/#support - we'll enable Control API access for the username/password. For any testing and experimentation, you can use the testing and experimentation clientId 6ent0fqtc4v5ud6i8o41ado8rj. Being a multi tenant system, the clientId helps us differentiate API calls from different clients.

Authentication

You'll need to obtain AccessToken and IdToken by calling AWS Cognito.

Here is a sample request and response:

• Save the post in a file auth_data.json

{
    "AuthParameters" : {
        "USERNAME" : "{{LetsData Username}}",
        "PASSWORD" : "{{LetsData Password}}"
    },
    "AuthFlow" : "USER_PASSWORD_AUTH", 
    "ClientId" : "{{LetsData ClientId}}"
}

• Make a POST request to AWS Cognito and save the output to creds.json file

curl -X POST --data @auth_data.json \
    -H 'X-Amz-Target: AWSCognitoIdentityProviderService.InitiateAuth' \
    -H 'Content-Type: application/x-amz-json-1.1' \
    https://cognito-idp.us-east-1.amazonaws.com/ \
    --output creds.json

• Example response is saved in the creds.json - a quick cat creds.json | jq show the following json - copy the AccessToken and IdToken, you'll need these for API calls. Also note the expiry which is the duration the token is valid for.

{
  "AuthenticationResult": {
    "AccessToken": "eyJraWQiOiJaQk...<redacted>",
    "ExpiresIn": 3600,
    "IdToken": "eyJraWQiOiJuSktcL1JN...<redacted>",
    "RefreshToken": "eyJjdHkiOiJKV1Qi...<redacted>",
    "TokenType": "Bearer"
  },
  "ChallengeParameters": {}
}

API Calls

• You can call any of the LetsData Control SDK API by adding the "Authorization: Bearer IdToken" and "LetsDataAuthorization: Bearer AccessToken" headers. Here is an example API call that does a GET to retrieve a dataset's details.

curl "https://www.letsdata.io/api/dataset?tenantId={{tenantId}}&userId={{userId}}&datasetName={{datasetName}}" \
    -H "Authorization: Bearer IdToken" \
    -H "LetsDataAuthorization: Bearer AccessToken"

• Almost every Control SDK API requires the tenantId, the userId for the authenticated user (TenantAdmins can pass a different userId to retrieve data for other users in the organization - see user roles documentation) and the datasetName. You can find your tenantId, userId via the website, CLI (docs) or decode the IdToken to get the tenant and user ids ( jwt.io has a decoder). Here is a decoded Id token - the sub field is the userId and the custom:tenantid is the tenantId

{
  "sub": "accb3567-2b6e-41ae-b00d-6ce1f9a58d94",
  "custom:companyaddress": "{\"addressLine1\":\"1234 Some Street\",\"addressLine2\":\"Apt F8\",\"city\":\"Bellevue\",\"state\":\"WA\",\"country\":\"US\",\"postalCode\":\"98006\"}",
  "cognito:groups": [
    "Tenant-d5feaf90-71a9-41ee-b1b9-35e4242c3155-Users"
  ],
  "custom:userrole": "TenantAdmin",
  "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_asdjery68Ts",
  "cognito:username": "user@letsdata.io",
  "custom:companyname": "LetsData IO",
  "origin_jti": "1a0038a2-f8dd-4a71-996e-e64cde31003c",
  "custom:tenantid": "d5feaf90-71a9-41ee-b1b9-35e4242c3155",
  "aud": "11bbm85f3niuukca8su98dqc2t",
  "event_id": "978d2a81-fa94-4fdf-a5ce-52240c02aaaa",
  "token_use": "id",
  "auth_time": 1708220818,
  "exp": 1708224418,
  "iat": 1708220818,
  "jti": "194095f3-f350-425b-ad84-c5f8e1dd67fc",
  "email": "user@letsdata.io"
}

Improvements

For the longest time, we had resisted publicly releasing the Control SDK API, primarily because we had full featured CLI and Website clients that were adequately serving the current needs. The CLI is automation friendly, so building automation scripts using the CLI has been the recommended automation way. We also provided private API access where needed.

So why are we publicly releasing it now?

I was recently in a technical conversation with a respected software architect about designing an API for a usecase and we had quite a thorough discussion about the API design and implementation. His experiences in API design helped me disconfirm my beliefs about API design. This led me to review the LetsData Control API and measure it against a higher API Software Design bar.

Here are some issues that we believe need to be discussed in context of the Control API release and possible improvements to a vNext API.

• HTTP Verb Abuse - the API uses a total of 2 Http Verbs, GET and POST to get everything done.

  • Updates and Deletes are done via POST on update/delete sub resources. We need to use PUT for creates, POST for updates and DELETE for deletions.

  • Get single item vs Get list of items is via GET on the resource and /list sub resource etc. Item and List GET needs to be disambiguated with the GET parameters.

  • Getting these right from the get go should not have cost any additional time, so this is a miss (sigh, what was I thinking at that time!)

• Authentication - The API uses both the IdToken and the AccessToken for authentication, which seems a little non standard. Why are we doing this this way? I believe this is our security paranoia kicking in - on each API request, we do deeper validations on the tenant details, the user details, the clientId - essentially doing some redundant validations but making sure calls authenticated are not going to be a security issue. (and we did not find a way to add custom attributes such as tenantId in the AWS Cognito access token, hence using both tokens). While this decision was reviewed for security, we could re-review this for simplification.

• API Simplification - There is a case for parameter simplification that can be made - since tenantId and userId are already present in the authentication tokens, they can be removed from API parameters (or made optional to override the auth token values if needed). For example, api/dataset?tenantId={{tid}}&userId={{uid}}&datasetName={{dName}} could be simplified to api/dataset/{{dName}}. The latter does seem simpler, but I ambivalent about this as of now - maybe we’ll run into some edge case that requires the tenant and user ids. Deferring to vNext for now.

• API Keys (lack thereof) - since we are allowing each customer a separate clientId, we believe as of now, the clientId is a sufficient in place of the API Key. APIKeys have a very nice integration in AWS API Gateway, where usage plans can be specified and the API Gateway can enforce these even before the request hits the web service code. However, we do not have such large scale quota or usage plan needs at this time - we can add any API Key specific logic (if needed) against the clientId (DDOS or other operational issue) and then have API Key in the vNext of the API. As of now, there is no API usage rate limits and restrictions.

• Language Client SDKs - When we did our earlier startup (LetsResonate), we had defined our API model using the API Gateway model. One nice thing about this definition was that the API Gateway generates Language specific clients for different languages. However, that definition and its request and response mappings was time consuming and frustrating. This time around, we by-passed the model definitions and defined the request and response as pass through strings, which were serialized/deserialized by our web service code. As a result, we don’t have the rich language specific clients that are auto generated by the API Gateway. We will try and get our API definition in some defined format and generate language clients for customers. Until then, REST Http API can be used to call the API.

• API Regional availability - API Gateway supports regional API availability. When we went from a US-EAST-1 region service to a multi-region service with availability in 6 regions around the globe, we made a conscious decision to keep the Control SDK API available only in the US-EAST-1 region. For a truly global service with optimal latencies, regional deployments of the API should be done, but this requires either:

  • DBs being in the same region as the regional API deployment (requires multi master replication and conflicts resolution which is easy to get wrong)

    OR

  • having the API servers have persistent connections to the DBs in the different regions (this is a challenge because our API servers are themselves Serverless and are initiated on demand)

  • See the Edge Proxy Design for benefits of API regional availability.

Some of these were conscious decisions because being a small startup, we prioritized for a bias for action rather than a perfect API and we do believe the decisions we made were correct - the API works well and these fixes would be work with lesser benefits. We’ll fix these when they become an issue / re-evaluate for vNext.

Conclusion

With the CLI, Website and API availability, it becomes very easy to get started and automate your data workflows on LetsData. We’d love to learn more about your data usecase and your thoughts on how we can improve the API / developer experience.

Here are docs to get you started on LetsData Spark.

• Spark Compute Engine Docs, S3 Spark Read Connector Docs and S3 Spark Write Connector Docs

• LetsData Spark Interfaces: Java, Python

• LetsData Spark Interface Implementation Examples: Java, Python

• End to End Example - Spark Extract and Map Reduce

How it Works?

LetsData's Spark interfaces are inspired by the original Google Map Reduce paper. We have defined a MAPPER interface and a REDUCER interface.

Here is a Spark Dataset's Architecture Diagram and how the mapper and reducer tasks process a dataset using Spark.

Recall that a dataset's amount of work is defined by a manifest file. For example, for S3 read destination, this is essentially a list of files that need to be processed.

The manifest file above specifies 4 files that need to be processed.

Mapper Tasks

• Each manifest file becomes a separate Mapper Task in LetsData.

• And each Mapper Task runs the mapper interface code.

• The mapper interface implements single partition operations (narrow transformations)

• The Dataframe returned by the Mapper Task is written to S3 as an intermediate file by LetsData.

• In this example case, 4 manifest files -> 4 Mapper Tasks -> 4 intermediate files.

• The interfaces themselves are quite simple, here is the mapper interface:

• Here is a sample implementation that reads the web crawl archive files to extract document which is written to S3 as an intermediate file:

Reducer Task

• LetsData creates a reducer task for any reduce operations for the dataset.

• The intermediate files from the mapper phase are read by the reducer, any multi-partition operations, shuffles, aggregates, joins or similar wide transformations are performed on the intermediate files.

• The dataset returned by the Reducer Task is written to the dataset's write destination.

• In this example case, 4 intermediate files from the mapper tasks -> 1 output file by the reducer task

• Here is the reducer interface:

• Here is a sample implementation that reads the intermediate files from S3 computes the reduce operations and writes the results to S3:

Config

You can configure the LetsData datasets to run both MAPPER_AND_REDUCER tasks, MAPPER_ONLY tasks or the REDUCER_ONLY tasks. Here is the config schema for Spark Compute Engine:

High-level Overview of How Spark works on Clusters

• At a high level, Spark is usually run on a cluster of machines where the cluster resources are divided as spark master, driver and workers processes.

• Each of these processes is allocated resources (memory, cpu etc) according to their expected workload.

• A spark cluster can run many applications, each application is scheduled by the master to have driver and some workers.

• Spark cluster can run as many applications as the driver / worker node divisions allow. For example in a 4 machine cluster, where each machine memory is divided to have 4 workers, the cluster will have 16 workers (assuming that drivers and masters have already been accounted for). Now if the applications that are being run request 8 workers, we can run 2 applications concurrently on such cluster. Additional applications will wait for resources to be available.

• The application's driver will coordinate the application run / worker management. It will send tasks that need to be accomplished to the workers. For narrow transformations (single partition), the task will be contained to a worker. For wide transformations (shuffle, repartition) it coordinates data from different partitions and assigns these reduce tasks to workers. Essentially, driver is the brain that coordinates the data processing and efficiently manages the workers.

How LetsData does it?

• LetsData uses the same primitive constructs, but stitches the dataset workflow a little differently. Here is what we do:

  • Scope Work: Since the files that are to be read are specified in the manifest, at a high level LetsData knows the number of readers (tasks) that it needs to initialize. We currently map each file to a single mapper task.

  • Mapper Task: Each of these tasks (mapper) runs as a DataTask function on AWS Lambda. We install spark on each lambda function in standalone mode. This essentially means that each function (10 GB memory, 10 GB disk) now has a master, driver and workers for this task (which maps to a single file). Since each task is a self contained spark instance, it can only perform narrow transformations (and will lead to incomplete results for any cross partition actions such as re-partition etc). User's spark code is run to compute the intermediate result for the file / partition. This intermediate result is stored in S3. This will be read by the reducer task.

  • Reducer Task: We create a reducer task that runs user's wide transformations (cross partition code). This task again is a DataTask on AWS Lambda where we've installed spark (with 10 GB memory, 10 GB disk) which now has a master, driver and workers for this reducer task. This task reads all the intermediate files and computes the final result using user's code. This final result is written to the write destination

• We've traded the smarts that Spark has built that compute an efficient query execution plan for a dataset with only task level efficiencies for now. As of now, we believe that with availability of workers elastically, parallelism can possibly be adequately sufficient in terms of performance. With usage based costing, we don’t pay for idle clusters, we do not have waits for a fully saturated cluster and on surface the cost efficiencies seem beneficial. Theoretically this seems like a good architecture (and we've seen good results on the tests that we have done), however, we'd like to run it with larger datasets to see how the system performs and what we can improve. In future, the standalone spark instances can be replaced with workers and LetsData could become the brain (driver) for the dataset and send work to the workers. (bigger project for another day).

• Defaults: We've built in default spark sessions, default read and write code and task management that should make getting started really easy. JAR incompatibilities, CLASSPATH issues, other setup and runtime issues have been standardized in our images and users should not have to deal with these.

• Errors, Logs, Metrics and Checkpointing: Spark tasks do not support the LetsData record errors (where individual record errors are sent to the error destination as json error detail files) and task checkpointing infrastructure yet. (where tasks periodically checkpoint their progress and support resume from last checkpoint semantics). So to answer the general question about errors, logs, metrics and checkpointing in spark tasks:

  • Logging: Logger is available, and logs would be sent to Cloudwatch and made available.

  • Metrics: No metrics support within the mapper interface yet. We generate some high level metrics. We will hopefully enable the spark metrics soon.

  • Errors: The LetsData dataset errors are currently not available for spark interfaces. You decide how you want to deal with errors. (Write separate error files (using same write credentials), log them to log file etc.). Any unhandled / terminal failures can be thrown as exceptions and the task will record these and transition to error state.

  • Checkpointing: The LetsData checkpointing and restart from checkpoints is not available yet for spark interfaces. Interfaces run either completely or fail, in which case the intermediate progress isn't used for reduce. If rerun, the tasks will run from beginning and overwrite any intermediate progress.

Example

A working step by step example for Spark on LetsData is available on the LetsData Docs Website. Here is the problem description that the example solves:

“we'll read files (web crawl archive files) from S3 using Spark code and extract the web crawl header and the web page content as a LetsData Document. We'll then map reduce these documents using Spark to compute the 90th percentile contentLength grouped by language and write the results as a json document to S3”

Conclusions

We are amazed by Spark and the scale, ease and facilities it provides for data processing. With the simplifications and Serverless support that we’ve added, we are actually fairly excited about what we have built.

We’d love customers to try it out and let us know what works and what doesn’t so we can help improve. We’d also want to know how folks have productized Spark for their data processing and any pros and cons of their work with Spark.

Thoughts / comments? Let us know.

Existing S3 Write Connector and Aggregate Files

#LetsData already supported an S3 Write Connector where each processed record is written as an individual file on S3. Here are what the files from this Write Connector look like:

While this works great, a large number of scenarios require aggregate files. It is common to have large files in S3 which have multiple records. For example, the individual files from S3 Write Connector above can be coalesced into an aggregate file as follows:

This is what we are launching today - you can now seamlessly create aggregate files with the #LetsData S3 Aggregate File Write Connector.

S3 Aggregate File Write Connector

S3 Aggregate File Write Connector is simple to get started with - define a write connector configuration and create a dataset - the processed records will automatically start getting aggregated into files on S3. Here is the write configuration schema:

The notable configuration elements are:

• fileRecordsSeparator: String - (Required) The string delimiter to separate the records in the file. For example, "\n".

• aggregateFileSizeInMB: Integer - (Optional) aggregateFileSizeInMB is the write connector's output file size in MB in S3. Allowed values: [10-128]. Defaults to 128 MB.

Implementation

The implementation had a few different challenges which should make for an interesting reading.

Streaming uploads on S3

One interesting challenge was how to do streaming uploads to S3 files - here are some issues:

• S3 Objects are Immutable: The S3 Objects are essentially immutable. Once you’ve created a file, it cannot be modified, only overwritten by a new file or deleted. This works great for data uploads where data files already exist, but doesn’t work quite well with streaming scenarios where data is continuously being created.

• Streaming scenarios require storage that supports appends: In streaming scenarios, the content arrives with time and needs to be appended to some storage container (file, database table etc). With S3, you need to have the content to be able create an object - appends don’t work as is. (For some background in distributed file systems, maybe look at the Google File System paper)

• Multipart Uploads: S3 does have a very feature rich API that allows for almost all different kinds of scenarios - in this case, the multipart uploads API seems like the recommended option - relevant snippets from the docs:

  Pause and resume object uploads – You can upload object parts over time. After you initiate a multipart upload, there is no expiry; you must explicitly complete or stop the multipart upload.

  Begin an upload before you know the final object size – You can upload an object as you are creating it.

In our implementation, we simply store the data on the local filesystem and initiate an upload once the file has reached the aggregation threshold. Although this seems to work well for now, we might see:

• some issues from staleness in true streaming scenarios (as opposed to batch streaming). These are because we are aggregating records for a certain size before making them available in the datastore (S3)

• delayed checkpoints - although we are writing to the local filesystem, we aren’t checkpointing until those records are in S3. This can cause delays in checkpoints (checkpoints happen after say 300 MB data has been processed). This can also make the solution prone to wasteful work and larger recovery scenarios. More on this follows.

Enhanced Checkpointing

In our data task, we have reader threads that read the data, compute threads that perform computations and writer threads that write to the write destination. Each of these functions is decoupled from the other - readers could have read 100 records (read pointer at 100), compute could have processed 50 records (compute pointer at 50) and writers could have processed 30 records (write pointer at 30). Here is a pictorial representation:

In this case, checkpoint calc is simple - when the writer flushes the 30 records in its buffer, it sends an ack back to compute and readers to trim those records from their buffers and record a checkpoint at 30.

In the S3 aggregated file writer case, there are two different flushes that need to happen before a checkpoint is created. 1.) the flushing of write records to the local filesystem 2.) flushing of the file to S3

We’d be writing records to local file and since we are aggregating, the flush to S3 thresholds are larger (for example 300 MB). If we wait for S3 flush to create the ack and checkpoints, The reader and compute buffers would be holding 300MB data each unnecessarily. (Contrived example, our implementation is quite efficient). Ideally, we can remove these 300MB records from all buffers once they have been flushed to the local file. The checkpoints can be created once the flush to S3 has completed. (So if the process crashes before the S3 flush is complete, we’ll redo the 300 MB work which is okay for now.)

This meant that our Acks and Checkpoints needed to be separated - the local filesystem writes drive the acks and the write to S3 create the checkpoints. A bit of rework and some testing later, the enhanced checkpointing is working quite well.

Conclusion

We now support writing aggregate files as well as individual files to S3 and have a variety of stateless and stateful readers to read data from S3. In addition, we are low latency and high throughput and are built around the AWS best practices for S3 - features such as backups, versioning, secure transport, audit trails, access controls etc are available by default with the #LetsData pipelines.

We’d love to know how customers might be using S3 in their data architectures and how we can evolve our S3 connectors to better serve the customers.

Thoughts / Feedback? Let us know!

• Dynamo DB Table Read Connector: Scans the table and send the records to compute engine / write connectors.

  • LetsData Docs

  • Interface (Java, Python, Javascript) and Example Implementations (Java, Python, Javascript)

• Dynamo DB Streams Read Connector: Similar to the Kinesis Read Connector, this read connector reads from the Dynamo DB Streams and processes the records according to the customer’s dataset configuration.

  • LetsData Docs

  • Interface (Java, Python, Javascript) and Example Implementations (Java, Python, Javascript)

Details for these connectors are as follows:

DynamoDB Table Read Connector

DynamoDB Table Read Connectors Scan the DynamoDB Tables. You can use #LetsData connector to scan DynamoDB tables for scenarios such as populating caches, running Gen AI pipelines or any aggregation jobs.

Here are some notable implementation highlights are:

• Configurable Scanning Speed: Customers can configure the scanning speed by specifying the number of reader tasks (parallel scan segments), which in conjunction with the configured concurrency can aggressively scan the table or be set up as a low priority scan

• Item Filtering: We’ve enabled support for DynamoDB’s filter expressions to filter items that are read

• Scan Completion Conditions: we’ve enabled a couple of modes for scan completion

  • SingleTableScan: Dataset completes once a single scan of the table is completed

  • Continuous: Dataset continuously scans the table until the dataset errors, is stopped or is deleted.

• Simple Data Interfaces: we’ve defined a simple interface that customers can implement and it is available in our supported languages:

Getting started with the Dynamo DB Table Read Connector is simple.

• Implement the interface, here is a simple python implementation:

  

• define the read connector (tableName and artifact) and manifest for the Dynamo DB Table - in this example, we are selecting english language items from the DynamoDB Table by specifying the readerFilterExpression

  

  Scanning a table reliably at scale is an interesting challenge and what we’ve built is pretty compelling in our humble opinions. We have some futuristic ideas on how to further improve the DynamoDB Table read connector - we’d love to hear user’s experiences so that we can iterate and improve.

DynamoDB Streams Read Connector

DynamoDB Streams are fantastic! When they were launched, I remember being amazed by the new development scenarios that streams enabled - database architectures had instantly transformed from querying models to event driven architectures. With the #LetsData DynamoDB Streams read connector, you can process these events at scale. Here are some notable callouts:

• Database Changelog: DynamoDB Streams are essentially a changelog where every insert/update/delete is available as an ordered stream of records.

• Kinesis / Dynamo DB Streams: The #LetsData implementation for the DynamoDB Streams Read Connector is very similar to the Kinesis Read Connector with the notable difference being the availability of before modification and after modification item images in the stream record. This lends to a slightly verbose #LetsData interface

• Designing for Ephemeral Shards: One interesting challenge in developing for Kinesis / DynamoDB streams was that #LetsData defines a single task for each Kinesis / DynamoDB shard. However, the shards are ephemeral - a shard can split into child shards or adjacent shards can be merged into a single shard. This essentially means that existing tasks complete and new tasks need to be created for the new shards. We’ve added code upon each tasks completion to detect if there are shards that haven’t been assigned to tasks and create new tasks if needed.

  • In all our use cases thus far, we’d defined the complete work specification at dataset creation time. These instances have made us monitor for changes in work specifications and add / subtract tasks as needed.

• LetsData Interfaces: The #LetsData interfaces are defined in different supported languages as follows:

• Configuration: The read connector configuration is simple - the read connector requires the streamArn and the manifest defines 1.) configuration for the start point (Earliest / Latest) 2.) completion conditions such as StopWhenNoData or Continuous. Here are some example configs:

• Performance: We’ve seen quite nice throughput and latency numbers which are comparable to similar read connectors.

Conclusion

With the Scan and Stream read connectors, any data that is in a Dynamo DB table can be processed by LetsData using different models - data pull using querying and data push with event streams. In addition to our already existing Dynamo DB Write Connector, and the nice performance numbers that we see, you can reliably use #LetsData and Dynamo DB as the central components in your data architecture.

Let us know, we’d love to work with you and onboard you to #LetsData!

Customers implement the KinesisRecordReader interface to transform Kinesis records to output documents. #LetsData handles reading from the stream, managing compute and writing to your selected write destination.

Implementation

While we discuss the implementation, there are a few interesting discussion points that should be explored:

Queues vs. Streams

When I first started looking at streams I grappled with the obvious question - What is the difference between a queue and a stream? I had worked with SQS and was now looking into Kinesis, and on the surface both seemed to have the similar sendMessage and getMessage semantics. For those who don’t work with queues / streams in their everyday jobs or are at the periphery of these technologies, do research this simple question. The internalizations of the differences that have stayed with me are:

• Queue messages are ephemeral - you consume them and they are gone (messages are deleted / dequeued). Streams tend to be more like a database records - you consume them and they are still there, available for different consumers to read as well (when message are read, your read pointer is advanced to next record).

• Queues are like glue that connect two systems and are created specifically for that system’s use-case example, sendMessage(“a task has been created, kindly assign to a worker”). Streams are more akin to a broadcast - where the stream can be used by many different systems, example sendMessage(“a task was created”). The many stream readers could do different actions for the same message. A monitor process will create alarms on the task, a scheduler will create a worker to assign task and a reporting process will generate a report of the number of tasks created per hour.

• Ordering guarantees are much stronger in streams vs queues. Queues are mostly unordered or at best offer FIFO ordering semantics. Streams on the other hand allow for sequencing records, which allows for distributed events reasoning such as happens-before relationships and ordering of events.

• The common stream implementations have stronger facilities for key-space control and sharding vs common queue implementations which do not have as flexible key-space / partitioning control.

This lends queues to be a great fit for message passing scenarios. Streams, on the other hand, are best for real time data processing and analysis.

Architecture

The architecture of a Kinesis Stream Reader Task in LetsData is similar to an SQS task, with the implementations being quite different. Here is a high level task architecture:

Following are some interesting implementation details:

Read Acknowledgements

SQS read connector is complicated in that it:

• reads the message from the queue which essentially is a lease for a certain duration

• maintains the lease and renews it if needed

• when processing is complete, deletes the message (read acknowledgement)

• there are no offsets, messages have been deleted, so checkpointing is mostly for progress reporting.

The Kinesis read connector read does not need any lease management and there isn’t any read acknowledgement needed. You read the record, process and then checkpoint the offset (sequenceNumber). This simplifies the reader and also increases the read throughput (one less call to make).

Batched Reads, Network Calls and Concurrency

In Kinesis Stream Reader, we use the batched read API to read 30 records and then for each record we call the user’s interface implementation to get the transformed doc. While testing, we ran into a couple of very interesting issues:

• The calling user’s interface implementation was essentially serial, which meant that we loop over the 30 records and call them one by one. This worked great when the user’s interface implementations were in-proc (java). With python/javascript, we added network calls which meant that user’s interface implementations latency was 30*latency of single network call. We fixed this to be done in parallel and saw significant performance increase (100 ms → 20 ms).

• When we added the above parallelism, we started seeing Out of Memory - Unable to create a native thread errors. Lots of debugging, stack dumps and code analysis later, it turned out that we were using the Executors.newFixedThreadPool(1000) method to create our threadpool. This had worked great but with the recent large increase in threads, it looked like the thread reclamation was somehow not happening fast enough in certain scenarios. We switched the threadpool to a ThreadPoolExecutor - specifying custom values for core pool size (100), max pool size (1000), a relatively aggressive thread reclamation timeout (500 ms), a custom thread factory that assigned stacktrace names to the thread and added threadpool statistic logs. We’ve not seen any issues since this change, threads are constantly around 100 (the core pool size) and thread reclamation is working quite well.

Offsets

Our S3 readers were file based, so our offsets were integers i.e. number of bytes into the file. With SQS, we didn’t need offsets. With Kinesis, we started seeing offsets(sequenceNumbers) which were very large integers, for example, 49647350647089693080323675671827096928861743259506966562. The engineer in me is curious as to how such numbers might be generated and if they are increasing numbers or not. However, that is not important. What is important is that integers were probably not a great idea for offsets especially when we are dealing with a myriad of read resources. So the offsets needed to be string.

We reworked our stack to store offsets as strings - the each destination’s readers can make sense of the offset according to their own logic, but to the overall system, these are opaque strings. This did require an update to our data interfaces as well (breaking change).

These are some interesting implementation tidbits that I thought readers might like and it might help them avoid similar issues / design better systems. Now let’s look at how we can actually develop on LetsData using this new read connector.

Getting Started

Getting started with the Kinesis Stream read connector requires implementing the interface (Code), defining config (Config), granting #LetsData access (Access) and then running the dataset using CLI (CLI).

Code

The KinesisRecordReader is a simple interface that has a single parseMessage method that needs to be implemented.

You can look at the interface definition on GitHub (Java, Python, Javascript) and example implementations that simply echo the incoming record on GitHub as well (Java, Python, Javascript). These are also available on our read connector docs.

Here is the example code in different supported languages that shows simplicity and ease of integration.

Config

Define the read connector config which is simply the interface implementation details and the Kinesis Stream ARN. Read Connector Config on LetsData Docs has details and examples.

Access

• Trust #LetsData to access the Kinesis stream AWS account by creating an IAM Role that trusts the LetsData account. Details at Access Grants Docs

• Create a policy to allow access to the Kinesis stream. Access Grants Docs

• Create the policy and attach to role. Access Grant Docs

This should allow LetsData to access the Kinesis stream in the Customer’s AWS account. You may also want to look at the Examples to get a step by step of the access grant process, be sure to replace the policy with the Kinesis stream policy.

CLI

• Okay, now let’s create the dataset using the LetsData CLI. Download and Setup Instructions

• Create the dataset configuration incorporating the read connector configuration from the Config section above. Examples have a step by step rundown.

• Create the dataset, monitor execution via CLI or via the LetsData Website

Conclusion

Kinesis streams is a very powerful technology and can be used in a variety of different use-cases. #LetsData simplifies reading and writing to Kinesis - user has to write their business case logic only while #LetsData manages the read, write and processing infrastructure.

Thoughts? Let us know. Happy to chat!

Customers can now implement the LetsData interfaces in Java, Python and Javascript languages and package their implementations as docker containers (in addition to the existing JAR Files) for a simplified development experience.

Here is a high level support matrix:

For those interested in digging immediately into the code, Here are the interface and example packages and the #LetsData docs:

• Python: Git Hub Interface Package, Git Hub Interface Implementation Examples

• Javascript: Git Hub Interface Package, Git Hub Interface Implementation Examples

• Java: Git Hub Interface Package, Git Hub Interface Implementation Examples

• #LetsData Docs:

  • Read Connector Docs: https://www.letsdata.io/docs/read-connectors

  • SDK Interface: https://www.letsdata.io/docs/sdk-interface/

Dev Experience Overview

We’ve translated the existing Java interfaces to Python and Javascript and packaged them as buildable, ready to deploy projects on docker container images. The developer workflow looks like as follows:

• update the interface files with the implementation

• build a docker image and upload it to ECR

• reference this ECR Image in their datasets

Architecture

Internally, we package the ECR Image as a “Language Bridge” Lambda function with http request-response web methods that invoke the user’s implementations. Our existing Data Task Java functions can then call these language-bridges with the data that needs to be processed. This is essentially a micro-service architecture with the Data Task micro-service calling the Language-Bridge micro-service.

In-Proc vs Micro-service

An interesting difference between the existing Java language interfaces and the new Python / Javascript language interfaces is that the existing Java language interfaces do not use micro-services, they are essentially in-proc calls in the same JVM. This is primarily because all our existing code is implemented in Java and we take your implementation JAR, build it with our code and execute it as a single JVM process. This results in the following differences:

• Stateless vs Stateful interfaces: Java interface implementations can be stateful - the code can easily maintain a thread to stateful implementation mapping. When we move to Python / Javascript using micro-services, we lose the stateful support - the interfaces need to be stateless. This is because subsequent calls from a thread could land at different micro-service endpoints ( lambda functions) which may not have state from the prior call. Statefulness can be implemented using persistent sessions, but that becomes overly complicated in terms of an implementation and becomes a micro-services anti-pattern IMHO. So is the move to micro-service a step down? Not necessarily - here is to micro-services defense:

  • Most stateful parsing and transformation implementations can be simplified to stateless implementations (in similar parlance to SQL to NoSQL transitions)

  • The loose coupling of services affords higher, horizontal scalability and increased performance.

  • A simplified overall architecture

• Latency: Java interface implementations are blazingly fast since they are within the same JVM, whereas the Python / Javascript micro-service implementations require network transit. These are within the same network and from experience we’ve seen that Python / Javascript interface implementations are adequately fast - we’ve seen parse message calls latency of ~1-2 millisec (inlcuding the network) for python and slightly higher (~5 millisec) for Javascript. With all the other responsibilities of the Data Task function (network reads, destination writes, serialization / deserialization, compute etc), these difference in latencies seem to get amortized (or shows slight perceptible increase). We were initially skeptical about the network latency for such an architecture, but we’ve been very happy with the out of the box performance that we are seeing. With additional tweaking, we should be able to improve performance even more if needed.

• Dataset Initialization / Provisioning: Our provisioning of Java interfaces creates a new java build that packages the customer’s JAR with #LetsData interfaces. This decision was because we wanted to i.) make sure that JARs play nice together and that there are no compile time failures during interface implementations etc. ii.) we believe that this would allow us extensibility where we can customize the workflow to run some focused tests during the build to find any possible issues at initialization time. While all this is great, the downside to this is that this java build takes around 3-5 mins to complete and becomes the long pole in the dataset initialization workflow. The python/javascript implementations using pre-built ECR container images simplify this very nicely - we do not have to do any additional builds and the dataset initialization becomes quite short - we’ve seen datasets start processing data within 1-2 mins of creation. While we love this container packaging as of now, however, if we were to build similar focused tests to find issues at initialization time, we might see increased runtimes. However, this is unlikely, since the ECR container being a separate deployment unit can be deployed and scaled independently and therefore any issues can be treated separately as well.

• Developer Experience: The overall developer experience when developing with java JAR and LetsData is somewhat complex when compared to python/javascript containers and LetsData.

  • Python/Javascript containers and LetsData simplify the development quite a bit. We’ve implemented the Document interfaces by default in python / javascript - with support for key-value map, these are flexible for most use-cases. Python / Javascript developers do not have to write their own Document classes and do additional tests around serialization / deserialization. We should probably implement the same for Java as well.

  • The Http request-response semantics on the python/javascript containers allow you to end to end test your code trusting that as long as your implementation does the right thing given the right inputs, the end to end will work quite well. Docker’s superior development infrastructure (IMHO) has a lot to do with this delight as well. With the java implementations, as we package the java project infrastructure today, the end to end testing and how everything fits in isn’t quite clear. This does mean we need to do more work to make the java experience as facile as python / javascript, but we are not there yet IMHO.

    

• Security: Running user’s lambda containers in your AWS account does increase the security risk - customer’s code could be malicious and could attempt to do all sorts of incorrect things, stealing credentials, generating malicious data etc. This concern is similar to running customer code in-proc, the in-proc code could be malicious as well. The way we mitigate in-proc code is that we run the in-proc code with a scoped execution role - which limits the code to be able to do only what we’ve determined should be accessible by the dataset. Similar security fencing should carry over to the language-bridge lambda functions. While we’ve not limited that execution to the dataset’s execution role yet, we’ve granted the language-bridge function a bare minimum set of permissions that are needed for lambda function to work. However, some accesses are for ‘*’ resources.

  

  My understanding of the system suggests that this should allow user code to be able to create log groups, put logs and metric data to namespaces they don’t own and do similar malicious activity with xray. I haven’t tested this yet and would be surprised if a lambda function malicious code can do these, I am sure there are some internal safeguards that should disallow a lambda function to run amok with respect to these blanket accesses. However, this is okay for now, but is on our minds as a security issue that we may need to fix.

Ideally, we’ll want all languages to be i.) in-proc AND ii.) have the scalability and dev experiences similar to micro-services. But duplicating all our existing code that is currently in Java seems like a larger effort - the read connector code, the #LetsData writers and all the orchestration code. To have a true in-proc, we’d have to rewrite the stack for each different language, which seems untenable as of now. The micro-services approach is a very nice implementation that should cater to the approx. 95%+ of the cases (unscientific gut number).

Implementation

Config

Here is an example read connector dataset configuration using Python.

• The JAR attributes have been replaced by ECR Image attributes

• Since the project structure and docker image is a known template, #LetsData infrastructure knows where the interface locations are. This allows us to simplify the configuration where we don’t specify the implementation class names anymore.

Code

The interface code is simple - for example, we converted our stateful Common Crawl java interface implementation to a stateless python/javascript implementation.

Conclusion

We’d love to learn how SAAS platform owners have dealt with the challenges of multi-language support - separate technology stacks might be the inevitable reference architecture.

Despite the cons and issues mentioned above, simplified development in popular programming languages and zero infrastructure management while processing data are compelling reasons to give #LetsData a try. We’d be happy to work with folks to onboard their data use-cases to #LetsData.

#LetsData is now available in the following AWS regions:

• us-east-1 (N. Virginia)

• us-west-2 (Oregon)

• us-east-2 (Ohio)

• eu-west-1 (Ireland)

• ap-south-1 (Mumbai)

• ap-northeast-1 (Tokyo)

You can read about the feature details in our our docs [1].

Background: Data Locality

Before we get into the details of the #LetsData multi-region feature, let’s look at why multi-region support is important in modern day data processing.

Data and Compute Locality

Google’s Map Reduce [2] and Apache’s Hadoop / HDFS [3] are seminal technologies for large scale distributed data processing. One of the key design tenets from these systems is about the data locality.

From the HDFS architecture:

A computation requested by an application is much more efficient if it is executed near the data it operates on … The assumption is that it is often better to migrate the computation closer to where the data is located rather than moving the data to where the application is running.

From the Map Reduce paper:

The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data. Failing that, it attempts to schedule a map task near a replica of that task’s input data (e.g., on a worker machine that is on the same network switch as the machine containing the data). When running large MapReduce operations on a significant fraction of the workers in a cluster, most input data is read locally and consumes no network bandwidth.

These works are from ~ 2004/2005, when cloud computing either didn’t exist or was in infancy. Google Map Reduce paper cites commodity hardware, IDE disks and 100 Mbps - 1 Gbps ethernet.

AWS, EC2 and Network Bandwidth

With the likes of AWS and EC2 instances with 25 Gbps - 100 Gbps bandwidths available to services like S3 at reasonable costs, in my humble experience, the code optimizations for data locality to be on the same machine, rack or network switch might not be necessary. For example, here are the m5n instance types and their available network bandwidths and hourly costs.[4]

(The costs by themselves are affordable in my opinion. For example the m5n.25xlarge / m5n.metal costs are around $4,100 per month. These can also be decreased drastically if one were to use reserved instances or spot instances).

And we see very nice performance even in higher level services such as AWS Lambda (that might be built on top of this EC2 infrastructure) when reading / writing to AWS services such as S3 and Kinesis etc [5].

So, to re-iterate my humble observation from above, the code optimizations for data locality to be on the same machine, rack or network switch might not be necessary.

What about same AWS availability zone? Same AWS region (different availability zone)? Same geographical region (different AWS region)? Different geographical region? (For those who need a primer about these concepts, see this page about AWS Global Infrastructure [6] and about regions and availability zones[7] )

Lets look at some latencies [8]

So within the same region, we can expect a p99 latency < 10 milliseconds. The website doesn’t breakdown numbers for within an availability zone or across availability zones, but Michael Haken’s blog post [10] suggests that sub millisecond latency within same availability zone can be expected. Within the same datacenter latency and its comparison with disk [11]

Round trip within same datacenter 500,000 ns

Disk seek 10,000,000 ns

To reason about the latency and throughput, refer to Brad Hedlund’s post [9].

Reduce latency? How is that possible? Unless you can figure out how to overcome the speed of light there is nothing you can do to reduce the real latency between sites. One option is, again, placing a WAN accelerator at each end that locally acknowledges the TCP segments to the local server, thereby fooling the servers into seeing very low LAN like latency for the TCP data transfers.

So, the reality here is that you probably cannot be faster than the speed of light - instagram animation, speed of light does 7.5 orbits around earth in 1 second, so to travel across the world it would need 1000/7.5 => 133 milliseconds. Light needs 133.33 milliseconds to travel around the world.

With this reasoning, in my humble opinion, you’d get acceptable performance by being closer to your data, in-region vs. cross region and can probably get away with not being on the same machine, rack or network switch.

So, in case your data is in Europe, you’d now benefit from the #LetsData compute availability in eu-west-1 (Ireland) !

#LetsData Dataset Types

Okay, enough about the background, what can we do with #LetsData’s availability in multiple regions?

With multi-region support, Datasets on LetsData can now either be in a single region or cross-region.

• Single Region Datasets: Single region datasets are completely located in a single region and all resources required for the dataset processing (read destination, write destination, error connector, artifacts and compute engine) are located in the same region

• Cross-Region Datasets: Cross region datasets are where the dataset resources (read destination, write destination, error connector, artifacts and compute engine) are located in different regions.

Dataset configuration defines whether a dataset is single region or cross region. The dataset configuration supports specifying the region either at the dataset level (single region) or individually at each resource level (cross region). This gives flexibility to create configurations according to the dataset needs.

Here is an example schema for dataset configuration with regions.

The Compute Engine region is probably the most important region for the dataset - this is the center of the dataset and where all the processing happens, essentially all distances (in region, cross region) have this region as its origin.

Interesting Dataset Configurations

Ideally, all processing should be done in-region for best performance and costs, but with multi-region support, one can now create a number of interesting different configurations to prioritize reads, writes or errors. For example:

• Prioritizing for In-Region Reads: By selecting the compute region to be the same as the read region, one can prioritize reads to be in-region reads whereas writes can be in a different region. This is essentially pinning the compute region with the read region. This configuration is useful when the data read is much much greater than writes (and reads and writes need to be in different regions.)

• Prioritizing for In-Region Writes: By selecting the compute region to be the same as the write region, one can prioritize writes over reads - essentially pinning the compute region with the write region. This configuration is useful when the writes are significant in comparison to the reads.

• Degenerative case: The academic degenerative case for dataset component regions is where every component is in a separate region. We've tested this scenario, and it works.

#LetsData Regions Implementation Design

How have we implemented #LetsData regions? We’ve partitioned a dataset into two obvious components: a control component and a data component.

• Control Component: The control component is the dataset’s creation, initialization and management. These are low volume, one time / few time operations where we setup resources for data processing. Since these are one time / low volume, we run these in us-east-1 region. Again, since these are infrequent, even a 250 ms cross region call from ap-south-1 should be okay and scalable as of now.

• Data Component: The data component is the Data Task Lambda Function and the Sagemaker endpoints (if any) and they are created in the Compute Region. All the data processing happens in the Compute Region, data access, checkpointing, task processing etc. The only call across region is at the completion of tasks when we let the control process know that the tasks have completed.

Here is the dataset and task lifecycle and the split between control and data components:

Learnings

Here are some learned best practices from experience:

• perfecting a task in a single region and then replicate it across geographies

• question every task being replicated as to whether it is needed in replicating region or not? Managing 1 region vs 6 regions is a very different effort - you’d have copies of credentials, storage, compute, logs, metrics and other such infrastructure. Try and see what the system performance would be if something is not regionalized.

• deep dive into components when replicating across geographies, its very easy to add regions with a sub optimal architecture - beware of the hidden cross region calls and that non regionalized component that you didn’t know was also part of the architecture.

• run tests in different regions, analyze usage and look for any regional / cross region usage / data transfer. You might find a bug or two.

• One thing we did was that instead of replicating databases across regions for low latency, we clearly separated the control databases and data databases. Control databases are only in the us-east-1 region and are accessed by control components only. Data databases are in the data regions and are accessed by data components only. This helped us get away without having to have global tables and cross region replication. While these features are great, as of now, not having to deal with such complexity is better architecture IMHO.

A few things we would like to do but have deferred:

• API availability in each region with regional api and code deployments - add management overhead as of now

• Separate AWS Accounts for each region - currently we are using a single AWS Account for all regions - again, having separate accounts is probably better scaling decision, but its a lot more work to manage

Additional Reads?

Our Distributed Computing field is quite large and while I have read some papers and tried to form an opinion, I am sure many important papers since the ones I mentioned have advanced this field. If there are any must reads in this area, do share - I’d love to learn what people have done in this space.

Thoughts / feedback, let me know.

References:

[1] Region documentation on #LetsData - https://www.letsdata.io/docs#regions

[2] Google Map Reduce - https://storage.googleapis.com/pub-tools-public-publication-data/pdf/16cb30b4b92fd4989b8619a61752a2387c6dd474.pdf

[3] Apache Hadoop - https://hadoop.apache.org/docs/r1.2.1/hdfs_design.pdf

[4] EC2 Instance Type Pricing - https://aws.amazon.com/ec2/pricing/on-demand/

[5] #LetsData Case Study: Big Data: Building a Document Index From Web Crawl Archives - https://www.letsdata.io/#casestudies - PDF

[6] AWS Regions and Availability Zones - https://docs.aws.amazon.com/whitepapers/latest/get-started-documentdb/aws-regions-and-availability-zones.html

[7] Regions and Zones - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions-availability-zones

[8] AWS Latency Monitoring - https://www.cloudping.co/grid/p_99/timeframe/1M

[9] How to Calculate TCP throughput for long distance WAN links - https://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/

[10] Improving Performance and Reducing Cost Using Availability Zone Affinity - https://aws.amazon.com/blogs/architecture/improving-performance-and-reducing-cost-using-availability-zone-affinity/

[11] Are networks now faster than disks - https://serverfault.com/questions/238417/are-networks-now-faster-than-disks

Metric Definition

Metrics, Dashboards and Instrumentation results give a view into the running of the service, whats going as expected, what is an anomaly and how the overall system is performing. While this would be a great explanation of a service’s operational dashboard, the science around dashboards goes much deeper. One classification that is simple and that I liked were classifying a metric / dashboard as either exploratory or explanatory. Here is a visual from Brent’s Linked In post that talks about this [1]:

While we weren’t thinking this deeply when we implemented our metrics but in hindsight, it makes sense to apply this theory to our metrics. We essentially wanted to answer these primary questions:

• What do we intend to communicate with the metric / visualization? (Goal in Brent’s post)

• What will the viewer view this visualization for (or the action that they would take)? (Outcome in Brent’s post)

Some of the other dimensions he mentions were implicitly assumed during our work:

• audience is a LetsData user (audience)

• the LetsData user is familiar with the task and data domain (data familiarity)

• the LetsData user would want to know about the system’s performance (visualization focus)

• the user knows the narrative as it is specific to each metric - throughput from a read destination, scaling of a write destination etc (narrative)

What Should I Instrument?

So what do I instrument? Well, it depends on what are you intending to learn. For example, for my earlier startup, we measured the customer acquisition funnel and how it was changing over time with the experiments that we were running [2]. These were essentially business KPIs that told us how healthy the business was.

With LetsData, we are a developer focused offering and a lot of focus is on making sure that we provide the developers with visibility into the different parts of the system that they have delegated to LetsData. This is essentially:

• a service’s operational dashboard

• operators can use these to tune or correct the service

• operators can monitor the different components of the service and reason about the overall progress

A large part of what to instrument is intuitive and comes with experience developing, operating and monitoring services. When you are developing code, it becomes second nature to add metrics for long running cpu work, network calls and other such tasks. Also, not all these instrumentation need to be in the final service dashboards.

What we decide to display on the Service Level Dashboard needs to answer some need that user may have. One way to achieve this is to draw your system’s architecture diagram and reason about what components users would want to know more about. This approach seems to be consistent with different system analysis tasks - Threat Model Diagrams in Security, Data Flow Diagrams in Software Design etc. Generally, when unsure about some system aspects, drawing a diagram and breaking it into components helps as a divide and conquer approach.

Let’s look at the #LetsData Task architecture diagram and see how we’ve reasoned about a handful of top level metrics for the dashboard.

The first step in task lifecycle is the task start or task processing. As a #LetsData Task processes, the user would want to know how many tasks succeeded / failed (Task Success Rate). They would also want to know the time taken by a task (Task Latency).

• What we intend to communicate: Overall task progress, whether system is succeeding or erroring.

• What will the viewer view this visualization for (or the action that they would take)? Alerts the user to any latency issues or task failures and they investigate by looking at additional metrics, logs etc.

  ---

The task is responsible for reading data from the read destination - metrics for read throughput (Task Bytes Read) and read latency (Message Read Latency) would help diagnose any integration issues with the read destination.

• What we intend to communicate: The performance statistics of the task’s reader

• What will the viewer view this visualization for (or the action that they would take)? Alert the user to performance issues when reading from read destination and they investigate by looking at read destination metrics, logs etc.

Describing each box in the diagram for what metrics are needed would become rather tedious and boring. These should give an idea on how to define the metrics needed for the service’s dashboard. Correlating with the diagram, here is a list of the metrics that are on our dashboards:

• Task Success & Volume (Box A in the diagram)

• Task Latency (Box A)

• Task Checkpoint Success & Latency (Box H)

• Task Number of Records Processed / Errored / Skipped (Box F & G)

• Task Write Connector Success & Volume (Box F)

• Task Write Connector Put Retry % (Box F)

• Task Record Latencies & Volume (Iteration Latency - Box B to J)

• Task Bytes Read & Bytes Written (Box A and Box F)

Here is an actual dashboard from the case study “Big Data: Building a Document Index From Web Crawl Archives” - the metrics are available and browsable at: https://www.letsdata.io/#casestudies

The metrics we’ve defined are mostly explanatory - there is very little exploration. However, they may lead to some curious exploration to understand the system. Our metrics infrastructure currently is not setup for exploration - specifically the speed and flexibility needed for exploration is missing. Enabling the users to filter, slice and dice the data and query metrics would be needed for exploration.

Some advice from the metrics dashboard development experience:

• metrics deluge on dashboards can be distracting so try limiting to one or two metrics per component area

• know what type of visualization / chart you need - our metrics are mostly time series, so we’ve used line charts. An excellent resource on choosing different types of visualization is at [3]

• one doesn’t know what screen size the graph would render on, and how much data / data range would the chart have. Would a larger range or limited screen size make the chart unusable? See if a zoom can be implemented that draws the chart in a modal maximizing the screen space (this was built from experience - one of the charts were rending such that the data wasn’t legible - we implemented a zoom control)

• since you are emitting the metrics, you know what they mean. You know the idiosyncrasies of your data gathering library. Document how the metric should be interpreted. We’ve built an inline help control in our chart widget that we are extremely proud of. Help is only a click away. This IMHO should be a standard in all metrics and charting controls.

  

  I’ll also link advice from the charting authorities [4] [5] that should make charts and visualizations crisper. We need to implement some of these in our charts as well.

How do I Instrument?

This section is about the some technical aspects around how to emit metrics around AWS Cloudwatch - feel free to skip ahead if not interested.

We use AWS Cloudwatch Metrics throughout #LetsData. We love AWS Cloudwatch Metrics:

• Simple API: the SDK is super simple to use, comes with single and batched put metric APIs [6]. At high scale, the single datapoint API can quickly get throttled (example, 10 tasks, each processing 30 messages per second, each message emitting 30 metrics ~ 9000 metrics per second!). We almost exclusively use the statistics set and they work great - no throttling issues since we moved to these!

• Embedded Metrics Format: Since we’ve built our statistics set metrics infrastructure, Cloudwatch now also supports an embedded metric format [7] where you log your metrics to Cloudwatch logs and they automatically get recorded. We are yet to try this but this is the next level simplification to a very scale intensive problem. Recommend newer integrations to Cloudwatch to give this a try before the put-metric-data API.

• Data Lag & Granularities: The data is available almost realtime and the per minute granularities that make it extremely useful (they also support 1-sec metrics). We emit per minute metrics and have not had any issues during investigations.

• Query: Rich querying capabilities allow for data querying for large number of use-cases [8]. The API allows for multiple query results in a single call - we get our entire dashboard data in a single call. In addition, you can specify what statistics you need so the server automatically does that computation [9].

• Console Metrics & Dashboards: Cloudwatch Metrics console visualizations and dashboards are great - you can construct different types of visualizations and create detailed dashboards.

• Metric Streams: Metric streams are also available [10] which allow for a interesting range of new scenarios that can be built around metrics. For example, a write destination scaling / descaling service that listens to the LetData PutItem Retries metrics could automatically scale up the write destination if the retries are greater than a threshold.

What data visualizations should I create?

Our metrics are mostly time series, so we’ve used line charts. However, choosing the right data visualization for the data is critical.

[3] “Data Visualization Cheat Sheet” and [4] “20 ideas for better data visualization” do an excellent job in how to choose the right visualization and the DOs and DONTs of each visualization type. [5] “What to consider when using text in data visualizations” has excellent advice on how to annotate charts. Even if you don’t have any need for charting, you should at-least read and bookmark these three links as a great “Getting Started To Charting” links whenever you do have some charting needs. From the data visualization cheatsheet:

How do I create data visualizations?

While AWS Cloudwatch Metrics gives us the raw metric datapoints, we use Chart.js (https://www.chartjs.org/) on our website to create the data charts that we’ve shared earlier. We love Chart.js - a simple api, performant and feature rich.

We did have to write a server data translation later that converts AWS Cloudwatch Metrics datapoints to the chart.js format - essentially creating all the chart data and labels etc on the server, sending packaged data to the browser that can use it to display charts without any additional processing.

However, being such big fans of AWS Cloudwatch Metrics, we wish we didn’t have to use charts.js and the metrics data translation layer that we’ve currently written. AWS Metrics and Dashboards should natively support displaying charts and dashboards on external sites. Ideally in our multi tenant case, we would have a dashboard template that we would apply to each dataset which would generate a dataset dashboard. This dashboard can be displayed to the users on LetsData website in a frictionless manner.

Signing off

That concludes today’s presentation. Hope you’ve enjoyed reading this.

We’d like to know:

• what stack are you using for metrics, data visualizations and dashboards

• any additional items that should have been included when talking about data visualizations.

• how we can improve

Resources:

[1] Brent Dykes’ post on differences between exploratory and explanatory data visualizations: https://www.linkedin.com/posts/brentdykes_datastorytelling-datavisualization-datavisualisation-activity-7097629835941875714-iL9b

[2] LetsResonate blog post “Growth Hack - The Case For Device Push Notifications” - https://blog.letsresonate.net/post/612587620515627008/growth-hack-the-case-for-device-push

[3] Richie Cotton - “Data Visualization Cheat Sheet” https://www.datacamp.com/cheat-sheet/data-viz-cheat-sheet

[4] Taras Bakusevych - “20 ideas for better data visualization” - https://uxdesign.cc/20-ideas-for-better-data-visualization-73f7e3c2782d

[5] Lisa Charlotte Muth - “What to consider when using text in data visualizations” - https://blog.datawrapper.de/text-in-data-visualizations/

[6] AWS Cloudwatch Metrics - “Publish custom metrics” - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html

[7] AWS Cloudwatch Metrics - “Specification: Embedded metric format” -https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html

[8] AWS Cloudwatch Metrics - “GetMetricData” - https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html

[9] AWS Cloudwatch Metrics - “Get statistics for a metric” - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/getting-metric-statistics.html

[10] AWS Cloudwatch Metrics - “Use metric streams” - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html

#LetsData datasets are Read-Compute-Write and different components often use multi threaded concurrency to achieve efficiency and performance. Most of the time, this internal parallelism does not need to be shared with the customers, but in some scenarios, the system’s internal details are needed so that customers can make an informed scaling decisions.

Sagemaker Compute Engine endpoints (fleets) need to be scaled in accordance with the number of tasks, internal concurrency threads and additional parameters.

This post looks at how concurrency in general is architected for #LetsData and how it can be used to set concurrency configuration that results in adequate performance. For details around the Sagemaker Compute Engine and its design, look at the Sagemaker Compute Engine Launch Announcement[1], Developer docs[2] and Step by Step Examples[3].

Design Assumptions

Lets look at a few assumptions and design issues and then understand the Sagemaker Compute Engine issue.

The Random Distribution Assumption

The concurrent reads, computes and writes are sufficiently randomly distributed so as not to run into pathological issues such as a larger number of requests go to some single server or a subset of servers etc.

The Faster Reads Assumption

Assumption that reads are generally going to be faster than writes - so we will have more data that waiting to be written. The standard pattern for such reader-writer latency mismatch is to use multiple concurrent writers (if possible) for each reader.

Batch Writes and Concurrent Writes

Multiple concurrent writes aren’t possible where records are ordered by some write key (these records need to be written serially) - but in most cases, the random distribution of the records being read leads to performance gains when using concurrent writes.

For example, our write connectors collect documents in buffers and then either:

• Use a batch API (Kinesis, SQS)

• or do a multi threaded client side batch if a batch API doesn’t exist or isn’t tenable (DynamoDB batch write API doesn’t support conditional checks, SQS batch call limits to 10 messages etc)

Write Performance for AWS Services

While the writes are slower than reads, they are still blazingly fast to cause any perceptible issues in performance.

For example, for services such as DynamoDB, a write takes around 15-20ms on average. Kinesis batched put latency is ~150 ms on average for ~500 records. They still benefit from multi threading in some cases (DynamoDB and SQS for example). In these cases, we allow max 10 threads per writer to write concurrently to the write destinations.

(We assume write destination is adequately scaled.)

The Writer Fanout

The #LetsData dataset’s work is divided into tasks at the time of initialization - where each task could be:

• a single file - S3 readers create 1 task for each file

• a single shard - Kinesis readers create 1 task per shard

• or some fixed slice of the overall work - SQS readers divide the work according to concurrency configuration

Assume we are processing with 10 tasks in a dataset and lambda concurrency is >= 10. This essentially means that we will have 10 tasks reading and writing concurrently to the read and write destination.

If a write destination is DynamoDB or SQS, each of these tasks is then writing with 10 threads concurrently. So your write destination (SQS / DynamoDB) can expect 10 tasks x 10 threads concurrent write requests at any instant in time.

At #LetsData, we’ve mostly taken an experiential approach to deciding the concurrency and threading parameters for different concurrency issues such as the number of writer threads.

We’d initially set a value from what we believe is reasonable, and increase / decrease it based on the results and these rudimentary tests became our informed capacity tests. We didn’t find the need to optimize these with scientifically (mathematically) designed tests since these experience based parameters were mostly working well and further perf optimizations could be deferred to when they would start becoming a perf issue.

This was great until a few weeks ago, when we integrated with Sagemaker Compute Engine and saw latencies that were starkly different from what we had seen.

Compute Workloads

When we integrated with Sagemaker Compute Engine, we were seeing a batch of 5 records latency of 2-3 secs and single call latencies of 500 ms on average.

Compute workloads are different from the traditional read & parse workloads, and this is probably why you see APIs such as InvokeEndpointAsync [4] [5] (for large inputs, inputs and outputs stored in S3) & Batch Transform APIs (asynchronously transform data stored in S3) in the Sagemaker API set. So the on-demand inference API that we are using can be expected to have higher latencies than AWS Service read / write / simple compute latencies.

The Sagemaker InvokeEndpoint (on-demand) API is not batched - so we had to use multi threaded parallel calls to generate the vector embeddings. Since we configure the Sagemaker fleet size, we need to set the number of Sagemaker compute threads accordingly so that they don’t slowdown the Sagemaker fleet (Or conversely, we need to set the Sagemaker fleet size so that the multiple compute threads aren’t causing a performance issue).

Sagemaker Compute Fanout

The Lambda Task Concurrency decides the number of concurrent dataset tasks. For each of these lambda data tasks invocations, we allocate 5 threads (fixed internally in the system) to concurrently call Sagemaker endpoints. Each one of these threads processes a single document that might have multiple elements that need vectorization. If there are two elements that need vectorization (example, vectors for content and vectors for metadata), then we make these two calls in parallel to Sagemaker.

This becomes a fanout as follows:

    # of parallel Sagemaker calls = Lambda Task Concurrency x 5 Fixed Internal Threads x min(3, # of vectors per document)   

Sagemaker Design Issues

We need to be able to answer a few different questions:

• How do we decide the Lambda Task Concurrency and the Sagemaker Fleet Size in light of this scaling factor?

• Our customers would be setting lambda concurrency and Sagemaker fleet sizes as well, so how do we provide documentation and framework that they can calculate and set these using our guidelines?

• This does feel like a possible operation hot area - latency questions and issues will come up again and again around this space. How can we enable our customers so that they have the information that they need to investigate?

We do the following:

1. Run latency tests for the base unit of work to get a baseline latency profile

2. Revisit our Queuing System concepts to see how we can use these latency numbers to put reason around what the fleet sizes should be - we make this into workable instructions

3. We instrument the record as it flows through the system for latency - we capture different latencies and make them available to the customers. We also surface AWS Sagemaker metrics so that customers can self serve investigations.

Sagemaker Latency Profile

To get the baseline latency profile for our Sagemaker Compute Dataset, we run our datasets with Lambda Concurrency 1 (one task is running), Sagemaker Serverless with concurrency of 1 and Sagemaker Provisioned with instance count 1.

Sagemaker Serverless

Our Sagemaker Compute number of threads is still set to 5 - so we’d be sending 5 requests over at any time.

Sagemaker Serverless will process them serially. AWS metrics [6][7] tell us that each request takes around 250ms for model latency and 170 ms for model overhead. We process ~75 requests every minute.

On the client side, we measure the latency of the batch as approximately (model overhead + model latency) x batch_size => (170+250) x 5 ~ 3 secs

What we’ve seen is that the model overhead latency remains constant with Sagemaker Serverless and almost seems like a fixed cost - we tried this setup with 5 Sagemaker Serverless Consistency and 20 Sagemaker Serverless latency - we do see some slight variation in model latency but model overhead remains similar ~ 170ms. We don’t see this overhead with Sagemaker Provisioned, so this seems like an Sagemaker Serverless artifact.

Sagemaker Provisioned

Sagemaker Provisioned instance count is 1 - our Sagemaker Compute number of threads is still set to 5 - so we’d be sending 5 requests over at any time.

Sagemaker Provisioned, since its an EC2 instance processes these in parallel. AWS metrics tell us that each request takes around 280ms for model latency and 5 ms for model overhead. We process ~800 requests every minute.

On the client side, we measure the latency of the batch as approximately (model overhead + model latency) => (5+280) ~ 400 ms

Here are the total times and throughputs for the baseline latency profiles.

Queuing Systems Theory

The Queuing Systems chapter in the book Probability and Statistics for Computer Scientists (2nd Edition) by Michael Baron [8] provides a good refresher on continuous-time queuing process and our current problem can essentially be modeled as one (Book is a recommended read for computer engineers).

We are an essentially M/M/1 queuing process - where the first M is distribution of queue’s inter arrival times, second M is the distribution of the queue’s service times and 1 is the number of servers. M denotes an exponential distribution because it is memoryless and is a Markov process (i.e. present state decides the result, we get no information from the past that can be used to predict the future).

Here is an M/M/1 queueing process cheatsheet and the utilization results for the Sagemaker Serverless and Sagemaker Provisioned setup.

We’ve made some assumptions (model overhead is equal to wait time) and tried to apply this theory to our process. If we go by these numbers, and aim for a target utilization of 70-80%:

• 1 Sagemaker Serverless Concurrency can support two Dataset tasks (2 lambda concurrency)

• 1 Sagemaker Provisioned instance can support possibly 4-5 Dataset tasks (maybe more)

With these informed numbers, I’d rerun Sagemaker Serverless with two dataset tasks (lambda concurrency 2) and Sagemaker Provisioned with 5 Dataset tasks (lambda concurrency 5) and recalculate the utilizations and adjust the Lambda Tasks Per Sagemaker Single Unit. I’d then use this Lambda Tasks Per Sagemaker Single Unit metric to decide the Sagemaker fleet size.

For example, if Lambda Tasks Per Sagemaker Single Unit for Sagemaker Serverless is 2 (i.e. 2 Tasks stress the Sagemaker Serverless instance by 80% utilization), and my dataset needs to process 100 tasks, then for 10 Lambda concurrency, maybe 5 Sagemaker concurrency is required. (I’d set this higher though, say to 20, to give some headroom given that the costs might be reasonable).

Latency Metrics Dashboard

We’ve added a Task Details Dashboard which measures the latencies of the record as it moves through different subsystems and queues. We’ve gathered metrics such as read time, queue wait times, compute execution times, write times and doc sizes to help with any latency investigations that you may want to do.

Conclusions

We were surprised by the latency issue when we integrated with Sagemaker and that led us into a formal deep dive into how to reason about the queuing systems and decide on the different concurrency configurations. We also instrumented our implementation to allow detailed metrics around record processing.

Approaches that we would like to investigate but did not for this investigation:

• benchmarking our model performance and seeing if the ~250-280 ms latency that we are seeing from model execution is reasonable or not

• comparing the model execution latencies when its executed in non Sagemaker environments with Sagemaker execution latencies

• testing with different models that may be differently computationally intensive and see if the results degrade as expected - for example, running a model that takes 700 ms or larger doc sizes - would the overall system be negatively impacted by such changes?

Thoughts / Comments?

Resources

• [1] Launch: AWS Sagemaker available as a Compute Engine on #LetsData

  https://blog.letsdata.io/p/launch-aws-sagemaker-available-as

• [2] #LetsData Sagemaker Compute Engine Docs: https://www.letsdata.io/docs#computeengine

• [3] #LetsData Example - Generate Vector Embeddings Using Lambda and Sagemaker Compute Engine: https://www.letsdata.io/docs#examples

• [4] Sagemaker - Different Inference APIs

  • Real Time Inference: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html

  • Async Inference: https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html

  • Batch Transforms: https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html

• [5] Sagemaker Async Inference Notebook: https://github.com/aws/amazon-sagemaker-examples/blob/main/async-inference/Async-Inference-Walkthrough.ipynb

• [6] How do I troubleshoot latency with my Amazon SageMaker endpoint https://repost.aws/knowledge-center/sagemaker-endpoint-latency

• [7] Sagemaker Metrics Docs: https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html

• [8] The Queuing Systems chapter in the book Probability and Statistics for Computer Scientists (2nd Edition) by Michael Baron

Architecture

Here is an architecture diagram that shows how the Sagemaker compute engine has been integrated with #LetsData pipelines.

The Sagemaker compute engine has two major components - a Lambda compute component and a Sagemaker compute component. Here is how the pipeline works:

• Read and Parse Feature Doc: The Lambda compute component is responsible for reading the read destination, parsing the data using the user’ data handler interface implementations and creating a feature document as before. (Steps 1-4).

• Extract Doc Elements For Vectorization: The Lambda feature document previously would have been written to the write destination. However, with the Sagemaker compute engine, the feature document is vectorized. Step 5 extracts the feature doc elements that require vectorization. (This is a user’s implementation of #LetsData interface.)

• Generate Vector Embeddings using Sagemaker: The extracted elements are then sent to a Sagemaker Endpoint that generates vector embeddings (Step 6).

• Construct Output Vector Doc: The output vector doc is constructed from these vectors (Step 7) .

• Write Vector Doc: The rest of the pipeline is similar to earlier - the output vector document is written to the write destination and any errors are recorded in the error destination (Step 8-10).

Lets look at:

• how this architecture can be used in the emerging LLM app stacks

• the new #LetsData Sagemaker Vectors Interface

• the AI / ML models and how they can be used with #LetsData Sagemaker compute engine

• the details around setting up Sagemaker Endpoints that can be invoked to generate vector embeddings at scale.

• the overall Sagemaker configuration

#LetsData Sagemaker Compute Pipelines For LLM Apps

I had shared some thoughts in an earlier post around how #LetsData might be useful in the AI / ML apps and promised a deep dive around this. The LLM App Architecture from the Emerging Architectures for LLM Applications (an Andreessen Horowitz blog) should help us understand how #LetsData can add value to the AI / ML ecosystem.

#LetsData Sagemaker pipelines can be used to implement Data Pipelines, Embedding Model, Orchestration, API / App Hosting and Queries.

#LetsData’s Sagemaker Vector Interface

We’ve defined a simple Sagemaker interface(GitHub) that users can implement to:

1. Extract Document for Vectorization from the Feature Doc

2. Construct a Vector Doc from the Feature Doc and the generated Vector Embeddings

Here is the interface definition:

Here is a sample implementation for the Common Crawl Web Archives (GitHub) documents.

The complete example and step by step instructions are also available on our website (Generate Vector Embeddings Using Lambda and Sagemaker Compute Engine).

Using AI/ ML Models with #LetsData Sagemaker

The #LetsData Sagemaker Compute Engine is automation built around the Sagemaker Inference models and endpoints - this essentially means that any AI / ML model that can be used with AWS Sagemaker can be used with #LetsData - the model code is packaged as a zip file and uploaded to S3 and imported as AWS Sagemaker model that is used by #LetsData.

#LetsData supports AI/ ML models for Sagemaker in these following configurations:

• Reuse Existing LetsData Model: You created a model for a dataset on LetsData and would like to reuse it. You can specify the model Arn and LetsData will use that model for Sagemaker.

• Create New LetsData Model: You have the model code packaged as a zip file in S3. You'll specify the S3 Arn and LetsData will create a model for Sagemaker.

We’ve tested with the Hugging Face’s Sentence Transformer models in our implementations and have detailed examples on how to get the AI models working with #LetsData and the different customizations that are offered. Here are some quick highlights:

• Model Container Images: We support the entire gamut of ECR Sagemaker model container images - HuggingFace, Inferentia, Pytorch, Scikit to name a few. The complete support list is at our website and at the AWS ECR Sagemaker Image List

• Model Environment Variables: With #LetsData Sagemaker, you can customize your model environment to specify 1./ model runtime configuration and 2./ your model implementation details. For example, our HuggingFace SentenceTransformer model implementation uses the following env variables, essentially informing the model to use the vector generations for question-answer format and that our model code is in inference.py file and model/ directory has the custom code.

      "HF_TASK": "question-answering",
      "SAGEMAKER_PROGRAM": "inference.py",
      "SAGEMAKER_SUBMIT_DIRECTORY": "model/"

• Request and Response Customizations: While we’ve defined a fix format for request and response to the Sagemaker endpoints, your model code can add customizations as needed.

  Request
  -------    
  def input_fn(request_body, request_content_type):
      """
          Args:
          request_body: The body of the request sent to the model.
          request_content_type: (string) the content type
          Returns:
          Input data in json format.
      """
      if request_content_type == 'text/plain':
          inp_var = request_body
          return inp_var.decode("utf-8")
      else:
          raise ValueError("This model only supports text/plain input")     
  
  Response
  --------
  def predict_fn(data, model_and_tokenizer):
      """
          Args:
          input_data: Returned input data from input_fn
          model: Returned model from model_fn
          Returns:
          The predictions
      """
      model, tokenizer = model_and_tokenizer
  
      ...
          
      return vector_embeddings[0].tolist()

Our Compute Engine Documentation and Step By Step Example has more details around integrating models with #LetsData.

Sagemaker Endpoints to Generate Vector Embeddings

The Sagemaker Endpoint hosts the container image and the model, is called with the documents and it returns the vector result.

Sagemaker endpoints can be fine-tuned for concurrency, hardware, memory etc. Sagemaker supports two types of endpoints:

• Serverless: Sagemaker automatically hosts the model and containers and scales it to your desired concurrency and memory

• Provisioned: Sagemaker provisions the requested hardware and automatically hosts the model and containers

#LetsData supports both Serverless and Provisioned Endpoints for Sagemaker in these following configurations:

• Bring Your Own Endpoint: You have an existing Sagemaker endpoint in an AWS account, you can specify the endpoint Arn and endpoint config Arn and LetsData will use that endpoint for Sagemaker.

• Reuse Existing LetsData Endpoint: You created an endpoint for a dataset on LetsData and would like to reuse it. You can specify the endpoint Arn and endpoint config Arn and LetsData will use that endpoint for Sagemaker.

• Create New LetsData Endpoint: You'd like a new Endpoint created for the dataset. You'll specify the endpoint type (Serverless/Provisioned) and its endpoint configuration and LetsData will create a Sagemaker endpoint for dataset execution.

We’ve run our tests with Serverless and Provisioned endpoints and have seen the benefits of GPU hardware acceleration and the ml.inf.* EC2 instance types. We’ve been impressed with the overall AI / ML inference infrastructure that AWS supports and how a diverse set of models and technology can all be integrated with the service. Our integration with Sagemaker further simplifies the end to end AI / ML use-case data integration.

Sagemaker Configuration

The overall sagemaker configuration / schema is as follows:

Schema

Example - Create New Model & Endpoints

Example - Bring Your Own Model & Endpoints

Our Compute Engine Documentation and Step By Step Example has the complete details around configuration and examples for different Sagemaker configurations.

Future Work

• Integration with Vector Database: The write destination should be some Vector Database / Index instead of Kinesis Stream - we need to implement a Vector write destination. We are looking into different options and will work on having some option natively available in #LetsData. This is done, we are now integrated with Momento Vector Indexes (https://www.letsdata.io/docs/write-connectors?tab=momentovectorindexes)

• Validate a Customer Journey: While we’ve tested and built the ML / AI pipelines, we’ve not constructed an end user example / customer journey yet. In large part because we aren’t putting these vector embeddings in a vector database / queryable source. We should do this to further improve what we might have missed and validate the user scenario. We did validations with Web Crawl Archives vector index and web search. See the search section for results (https://www.letsdata.io/docs/write-connectors?tab=momentovectorindexes#momento-vector-index-write-connector-implementation)

• Enabling Learning Scenarios: Current Sagemaker Support is Inference only - we need to look at enabling learning / training scenarios as well.

• Growing our system from individual datasets to a pipeline of connected datasets: Today, our datasets read from a read destination, perform compute and then write to a write destination. If the write destination is an intermediate destination such as a Kinesis stream, we need another dataset to read from stream and write it to some durable location such as Vector Database. We need to natively support this. Example:

  {
      "pipelineName": "VectorIndexPipeline",
      "artifact": {
          ...
      },
      "errorConnector": {
          ...
      },
      "datasets": [
          {
              // dataset 1 
              //	- read from s3
              //  - run sagemaker compute 
              //  - write to kinesis
          },
          {
              // dataset 2 
              //  - read kinesis stream in dataset 1,
              //  - run lambda compute 
              //  - write to database
          }
      ]
  }

Resources

• #LetsData Sagemaker Compute Engine Docs: https://www.letsdata.io/docs#computeengine

• #LetsData Example - Generate Vector Embeddings Using Lambda and Sagemaker Compute Engine: https://www.letsdata.io/docs#examples

• Here are some references on customizing models with Sagemaker.

  • Hugging Face Sagemaker Docs: User defined code and modules

  • Hugging Face Sagemaker Custom Inference Notebook: Sentence Embeddings with Hugging Face Transformers

  • Blog at medium.com: Leveraging AWS SageMaker Serverless Inference for Customized Model Serving

• Some foundational reading on AI / ML / LLM architectures:

  • Emerging Architectures for LLM Applications (an Andreessen Horowitz blog): https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/

  • How OpenAI trained ChatGPT (an excellent summary of the MS Build talk): https://blog.quastor.org/p/openai-trained-chatgpt

We had been using LinkedIn articles and our LinkedIn page for our blogging needs. However, we like the conventional feel of having a permanent home on the website / linked to the website, having blogging features such as archiving, cross linking etc and the newsletter features such as delivery to subscribers inboxes. We’re still huge fans of LinkedIn and our page - we’d be sharing our progress on there - just that the content would be on this newer blog.

We’ve started a newsletter / blog at substack. For now, we’ve backfilled the blog with the content from our LinkedIn articles and page posts. I have been hearing great things about substack - though building a size-able subscriber base would be challenging (hint: subscribe to the newsletter if interested).

We have some interesting posts scheduled for the next few weeks - including our much anticipated Launch announcement tomorrow. Visit our blog (blog.letsdata.io) and subscribe to our newsletter so that you get the latest in your inbox as soon as we launch!

• Blog: blog.letsdata.io

• LinkedIn Page: www.linkedin.com/company/letsbigdata

• Website: www.letsdata.io

Woke up today to a couple of overnight alarms for #LetsData - the website's main page had thrown an error in the wee hours of the night. Also, a dataset's initialization was failing.

Initial thoughts, great that we've started running into issues like these - people are looking at the service and trying to use it!

Okay, enough gloating, its time to see how bad the failure situation is.

Website timeout

Look into the logs and find that the website's main page had timed out. This can happen from time to time - on initialization, we read a bunch of data from network and populate the caches - so the initial call is heavy, after that everything's cached.

The other issue is that we've implemented the server API in Lambda and when Lambda is inactive, the function is reclaimed. So, after 15-30 mins of inactivity, a request to the website would re-initialize the function and the network reads. 

A couple of quick fixes:

• Parallelize the network reads and do only the reads that are required for the page - 15 sec timeout reduced to 3 secs lambda response time

• Run a cron that retrieves the index, docs and home page every 10 mins so that atleast 1 Lambda function remains initialized (and customers do not get the initial delay - this is a TODO as of now)

Dataset initialization

When a user creates a dataset, we do a number of initializations, initialize queues, task databases, lambda functions etc. ("$ > letsdata datasets view help" for details)

In this case, our continuous tests had created a dataset and the build for the dataset had failed. The initialization didn't know what to do in this case.

A couple of things:

• About our continuous tests: The MVP that we've built has 2 different services that we can read from and the reads can be 5 different configurations. We have 6 destinations that we can write to, and 3-4 different ways we can specify what work needs to be done. So in total, we did some data generation and have 33 different combinations of read, write and work specifications that can be done. We wrote a test suite that creates a dataset from these 33 configurations every 20 mins, waits for it to complete initialization, start processing and then deletes the dataset. This makes sure that our customers don't run into unknown issues.

• About the build failure: Looks like the build had some transient failure where one of the maven dependencies was not found. A simple manual retry of the build fixed the issue.

  This is a new issue and has happened twice in ~400 odd dataset runs thus far. Error signature copied to the code and a TODO added. As we see more of this, we'll either add an automated retry or fix the issue why it failed. 

So that is a founder's start to the day - a good coding workout before some of the other tasks!

Challenges and Complexities

Developing the #LetsData Kafka write connector has been a challenging undertaking. Kafka is a mature product and has a rich ecosystem. Some challenges:

• AWS implementations come in a couple of flavors - a Serverless option and a Provisioned option.

• Different Kafka versions support additional features such as tiered storage.

• Unlike its competitors such as Kinesis, AWS Kafka implementations require separate VPC and networking setup

• Complexities around VPC connectivity, compounded by cross account access

  

  We've resolved a bunch of these ambiguities in our implementation and automated the Kafka cluster setup and networking. With #LetsData Kafka Write Connector, you can create a working Kafka Cluster and a secure VPC within a few minutes. The connector comes with #LetsData trademark write performance, simplicity and operations. 

  You can read about the Kafka write connector in our docs at www.letsdata.io/docs#writeconnectors

  VPC Networking

  To manage the Kafka Write Connector Vpc and its connectivity, we’ve built an extensive networking subsystem and we are launching our Vpc & Vpc connectivity APIs today in our CLI. At a high level, we've implemented:

  • IP Address Management for the multi-tenant system with disparate IP Address allocations and reclamations for each dataset. We've used Vpc's IPAM (IP Address Manager) feature that works really well.

  • Setup a secure, isolated Vpc for each Kafka Write Connector, setup for outbound connectivity only. Here is the architecture at a glance:

  

  • Connectivity to the resources in the Vpc via Vpc Peering - we've added self service APIs to establish connectivity.

  ---

  # run this command with customerAccountForAccess's aws credentials
   
  $ > aws ec2 create-vpc-peering-connection --vpc-id 'clientVpcId' --peer-vpc-id 'letsdataVpcId' --peer-owner-id 'letsdataVpcOwnerId' --peer-region 'letsdataVpcRegion' 

  ---

  # accept the vpc peering connection on behalf of #Let's Data by using the following #Let's Data CLI command  
  
  $ > letsdata vpcs vpcPeeringConnections accept --datasetName 'datasetName' --vpcPeeringConnectionId 'vpcPeeringConnectionId' --requesterVpcId 'clientVpcId' --accepterVpcId 'letsdataVpcId' --prettyPrint  

  ---

  # list the vpc peering connections for a #LetsData vpc 
  
  $ > letsdata vpcs vpcPeeringConnection list --datasetName 'datasetName' --vpcId 'letsdataVpcId' [--userId 'userId'] 

  ---

  # delete a vpc peering connection for a #LetsData vpc 
  
  $ > letsdata vpcs vpcPeeringConnections delete --datasetName 'datasetName' --letsdataVpcId 'letsdataVpcId' --customerVpcId 'customerVpcId' --vpcPeeringConnectionId 'vpcPeeringConnectionId' [--userId 'userId']

  ---

  You can read the technical details around our Vpc subsystem at www.letsdata.io/docs#vpcs

  Support Matrix

  We support Kafka Serverless and Kafka Provisioned in the #LetsData account (resourceLocation: LetsData). We also support Kafka Provisioned as BYOC (Bring Your Own Cluster) when the Kafka cluster is located in an external AWS account. (ResourceLocation: Customer).

  

  

  That is quite a feature and test matrix for a write connector destination! What we seemed to have built is either at the bleeding edge for these technologies (or a small chance that we are doing it wrong and no one does it this way!) Why? Well, we ran into a couple of issues that cut short this matrix:

  • The AWS MSK IAM Auth externalId issue: AWS MSK IAM Auth is a seamless authentication library that solves the authentication in such a simple way (kudos AWS IAM folks!). However, it did seem like we had to build a custom version for our use case. GitHub Issue

  • Lambda Functions connecting to Vpc in a different account - Lambda functions work with Virtual Private Clouds by establishing elastic network interfaces to the Vpc subnets and accessing the resources in the subnets. However, this seems to be limited to Vpc's in the same account (So lambda cannot connect to Kafka cluster in customer account). Some creative networking (with Vpc Peering) is probably required, but as of now, it isn't supported out of the box. This is why the Kafka Cluster in the Customer AWS Account has only 1 supported case - when the cluster is publicly available (and accessible to Lambda).

  Example: Create a Dataset

  The examples that have the step by step instructions have been updated for a Kafka Serverless write connector. (www.letsdata.io/docs#examples)

  Here is an example dataset configuration and commands to create the dataset.

  {
      "datasetName": "ExtractTargetUriDemoTest",
      "accessGrantRoleArn": "arn:aws:iam::308240606591:role/Extractor",
      "customerAccountForAccess": "308240606591",
      "readConnector": {
          "connectorDestination": "S3",
          "bucketName": "commoncrawl",
          "bucketResourceLocation": "Customer",
          "readerType": "Single File Reader",
          "singleFileParserImplementationClassName": "com.letsdata.example.TargetUriExtractor"
          "artifactImplementationLanguage": "Java",
          "artifactFileS3Link": "s3://targeturiextractorjar-demotest/target-uri-extractor-1.0-SNAPSHOT-jar-with-dependencies.jar",
          "artifactFileS3LinkResourceLocation": "Customer"
      },
      "writeConnector": {
          "connectorDestination": "KAFKA",
          "resourceLocation": "LetsData",
          "kafkaClusterType": "Serverless",
          "kafkaTopicName": "commoncrawl",
          "kafkaTopicPartitions": 5,
          "kafkaTopicReplicationFactor": 3,
          "kafkaClusterSize": "small"
      },
      "errorConnector": {
          "connectorDestination": "S3",
          "resourceLocation": "letsdata"
      },
      "computeEngine": {
          "computeEngineType": "Lambda",
          "concurrency": 15,
          "memoryLimitInMegabytes": 10240,
          "timeoutInSeconds": 900,
          "logLevel": "DEBUG"
      },
      "manifestFile": {
          "manifestType": "S3ReaderTextManifestFile",
          "readerType": "SINGLEFILEREADER",
          "fileContents": "crawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00000.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00001.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00002.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00003.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00004.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00005.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00006.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00007.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00008.warc.gz\r\ncrawl-data/CC-MAIN-2022-49/segments/1669446706285.92/warc/CC-MAIN-20221126080725-20221126110725-00009.warc.gz"
      }
  }

  ---

# create the dataset on #Let's Data using the CLI. 
$ > ./letsdata datasets create --configFile datasetConfiguration.json --prettyPrint 


# view the dataset on #Let's Data using the CLI. Once the dataset is created, it takes ~3 mins to initialize the resources (dataset is in INITIALIZING state, no tasks have been created yet). 
$ > ./letsdata datasets view --datasetName ExtractTargetUriDemoTest --prettyPrint 


# list the dataset tasks on #Let's Data using the CLI 
$ > ./letsdata tasks list --datasetName ExtractTargetUriDemoTest --prettyPrint

Example: SDK Sample

We’ve also updated our SDK samples on how to read data from #LetsData Kafka clusters. (https://github.com/lets-data/letsdata-writeconnector-reader and www.letsdata.io/docs#customeraccountforaccess)

Here is the sample CLI driver application that can be used to read from the Kafka cluster. This assumes connectivity to the VPC has been established. (Instructions on how to establish connectivity can be found at: www.letsdata.io/docs#vpcs)

# cd into the bin directory
$ > cd src/bin

# awsAccessKeyId and awsSecretKey are the security credentials of an IAM User in the customer AWS account. This is the customer AWS account that was granted access. In case this is a root account, you can create an IAM user. See the "IAM User With AdministratorAccess" section above.

# Connect a Kafka Consumer to the Kafka Cluster using aws-msk-iam-auth library
$ > kafka_reader --clusterArn 'clusterArn' --customerAccessRoleArn 'customerAccessRoleArn' --externalId 'externalId' --awsRegion 'awsRegion' --awsAccessKeyId 'awsAccessKeyId' --awsSecretKey 'awsSecretKey' --topicName 'topicName'

> Enter the kafka consumer method to invoke. ["listTopics", "listSubscriptions", "subscribeTopic", "pollTopic", "commitPolledRecords", "topicPartitionPositions","assignTopicPartitions", "listAssignments","quit"]
listTopics
{commoncrawl1}

> Enter the kafka consumer method to invoke. ["listTopics", "listSubscriptions", "subscribeTopic", "pollTopic", "commitPolledRecords", "topicPartitionPositions","assignTopicPartitions", "listAssignments","quit"]
assignTopicPartitions

> Enter the kafka consumer method to invoke. ["listTopics", "listSubscriptions", "subscribeTopic", "pollTopic", "commitPolledRecords", "topicPartitionPositions","assignTopicPartitions", "listAssignments","quit"]
topicPartitionPositions
{commoncrawl1={0=0, 1=0, 2=0, 3=0, 4=0}}

> Enter the kafka consumer method to invoke. ["listTopics", "listSubscriptions", "subscribeTopic", "pollTopic", "commitPolledRecords", "topicPartitionPositions","assignTopicPartitions", "listAssignments","quit"]
pollTopic
...
...
...

> Enter the kafka consumer method to invoke. ["listTopics", "listSubscriptions", "subscribeTopic", "pollTopic", "commitPolledRecords", "topicPartitionPositions","assignTopicPartitions", "listAssignments","quit"]
commitPolledRecords

> Enter the kafka consumer method to invoke. ["listTopics", "listSubscriptions", "subscribeTopic", "pollTopic", "commitPolledRecords", "topicPartitionPositions","assignTopicPartitions", "listAssignments","quit"]
topicPartitionPositions
{commoncrawl1={0=179, 1=424, 2=249, 3=185, 4=233}}

> Enter the kafka consumer method to invoke. ["listTopics", "listSubscriptions", "subscribeTopic", "pollTopic", "commitPolledRecords", "topicPartitionPositions","assignTopicPartitions", "listAssignments","quit"]
quit

Our docs, CLI and web have all been updated to work with Kafka Write Connector.

Conclusion

With these new Vpc & Clustering subsystems, we’ll now be looking to add additional clustering data destinations to LetsData. We also have plans to expand the Read Connectors to Kinesis, DynamoDB and Kafka.

We hope you’d like working with the newer features and would love to hear any feedback on what works well / doesn’t work well.

1. Prospecting customers

2. Co-Founder - Sales hiring (lots of great conversations but nothing final, lets see how this goes)

3. Hiring Web Developer (Woot! we hired an engineer who started this week and already has some templates going!)

4. Building additional product features (Stay tuned, we have additional connector announcements soon)

5. Finding how we fit into the overall eco-system (Product-Market Fit)

Between all this, regular posting on LinkedIn kind of automatically got de-prioritized. Anyways, on to the main topic for today. 

We've been trying to find creative ways for prospecting customers(#1) and for any customer connects, it is critical that we enunciate a crisp narrative to our Product-Market Fit (#5). For these two critical reasons, I attended the AWS Summit in New York(www) yesterday. Here is my debrief of the event and its applicability to #LetsData.

Preparation

Leading up to the event, I updated our Marketing Solutions Brief (pdf) and our Executive Summary (pdf) to create a Customer Package for any customer connects. Printed 30 odd copies at our local Fedex office, sorted, bound, laminated and boxed ready to be handed out to customers. I had registered as an attendee (not as a sponsor), so had no booths assigned in the Expo / Exhibition Hall. The idea here was to squat on some empty table if such an opportunity arose and chat with interested folks. This was the prep work for prospecting customers (#1).

For Product-Market Fit (#5), I went through the session catalog and identified the sessions that I'd want to attend. The selection criteria was: 

• anything related to data / compute space in general 

• data / compute AWS techs that I hadn't worked with 

• customer stories on how customers were actually using data techs in the industry

The idea here was that as long as I was consuming knowledge about different techs, customer stories, they'd connect to define a crisper strategy narrative and clarity for Product Market Fit.

Here are some sessions that I selected (now attending all these would be a challenge, the ones I attended are highlighted):

• Not Applicable, The Keynote

• 200 - INTERMEDIATE, Self-paced labs: Come and go as you please

• 200 - INTERMEDIATE, Adobe’s journey toward building an internal developer platform (IDP)

• 200 - INTERMEDIATE, Chart your Kubernetes course with Amazon EKS

• 200 - INTERMEDIATE, Fidelity’s observability platform for telemetry

• 200 - INTERMEDIATE, Create a CI/CD pipeline to deploy your application to AWS ECS

• 200 - INTERMEDIATE, Serverless SaaS: Building for multi-tenancy

• 200 - INTERMEDIATE, AWS networking fundamentals: Setting up your global network

• 200 - INTERMEDIATE, How modern data management can fuel your success on AWS

• 300 - ADVANCED, Faster insight with Amazon Redshift: Zero-ETL integration & sharing

• 300 - ADVANCED, Build a data governance framework to balance data control and access

• 300 - ADVANCED, Architecting for low latency and performance in financial services

• 300 - ADVANCED, Best practices for microservices deployed on Amazon ECS

• 300 - ADVANCED, Making a modern data architecture a reality

• 300 - ADVANCED, Managing resources with the new AWS Cloud Control Terraform provider

AWS Summit, NY, 2023

Boarded a 9 PM (Seattle) flight and arrived at Newark at 5AM - a 30 min Uber ride and I was at the Jarvis Center in West Manhattan at 6 AM. Fueled by a Coffee and a Cream Cheese Bagel, I was ready to attend some sessions and talk to some customers. 

Some observations about the event, the venue and the overall ambience at the AWS Summit NYC. The venue, Javits Center, on the banks of the Hudson and a short walk from the 9th Ave skyscrapers and attractions such as Vessel, Madison Square Garden, was impressive. The AWS event itself was Grand, a choreographed AWS show that demonstrates their success as the de-facto cloud leader. Large number of attendees and the queuing / sell out at lectures was a gauge of the interest and applicability of what they are doing. Every tiny little detail had been thought about and meticulously planned - so my plans of squatting a table on the Expo Hall were impossible, I'd have to improvise. 

Create a CI/CD pipeline to deploy your application to AWS ECS

After getting my bearings around the event, I attended the Lab about ECS deployments - "Create a CI/CD pipeline to deploy your application to AWS ECS". This was a relatively large hall that was setup as a Computer Lab - every attendee got a desktop with couple of monitors, connected to the internet. The instructor led the session in a Peloton style (Peloton reference since I had seen their store in the neighborhood earlier that day) - telling the audience about the lab and getting confirmation that audiences were following along. 

The software and infrastructure that AWS has built around these Labs, Workshops, Skill building and Training was pretty impressive - simple login that gets you an AWS Cloud environment that is fully functional to work through these use cases. For example, the test (sandbox) and prod environments that I use during my day to day development are actual AWS accounts that I have (nothing sandbox about my test account) - so having such a scoped down environment and sandbox resources would be a huge productivity boost in my day to day development as well. (The scoped down env I know can be created via IAM roles and policies, but I don't know how to do sandbox on AWS). Overall, the ease with which attendees could get environments up and running was quite impressive IMHO. 

The ECS CI/CD lab walked us through creating an ECS deployment and staged updates. It talked about CodeCommit, Code Pipelines, Code Builds and Lambda. We use these constructs quite heavily in #LetsData, so nothing new here. The container side of things, while I know what they are, and have done some experiments with these. These (ECS, Kubernetes) are ideas for compute for containerized applications which can be used instead of the default Lambda compute that we currently have in #LetsData. Walkthroughs of the container registry and deploying containerized applications. Good stuff!

Faster insight with Amazon Redshift: Zero-ETL integration & sharing

Next, I attended the Chalktalk "Faster insight with Amazon Redshift: Zero-ETL integration & sharing" - where the presenters whiteboarded how their newer Zero ETL features simplified the data ingesting to AWS Redshift. They showed different integrations from AWS Aurora, AWS S3 and AWS Kinesis directly into AWS Redshift. The integrations seemed easy to setup, simplified the existing complex pipelines, no cost in most cases, automatic integrations and came with the Amazon's operability goodies (look at batches, progress in different tables).

No Code vs Low Code vs Your Code

This talk seemed highly relevant to #LetsData and on the surface, brings forth an existential questions for the company:

• If there are zero touch and no cost ETL integrations natively available in AWS, would #LetsData be able to do better? 

• Or in terms of a Product-Market-Fit (#5) question, How does #LetsData fit in when there are existing zero ETL solutions natively available? 

• A few people I had talked to had asked me about No Code options in #LetsData and I had told them its an area we need to explore. What are our plans for no code?

Our distinguishing feature, flagship capability and the brightest feature in our cap is the managed service we've built around AWS Lambda as a Compute Engine - a true serverless and infinitely elastically scalable compute offering. Connectors are connections on reading and writing to / from different destinations. 

With #LetsData Lambda Compute Engine, you do not have to create clusters, manage machines, run orchestration and management software such as Kubernetes etc.. Clusters, machine management and Kubernetes are extremely smart and feature rich on management of a shared pool of resources (machines, Kubernetes CPU resources etc) - however, this means:

• you are still responsible for the operability and scale

• infinite elastic scaling is not an option without additional provisioning

The #LetsData infinite elastic scale was actually demonstrated in our Web Crawl Archives Big Data Case Study - where with just a config option, we created 100 concurrent Lambda functions without having to provision any hardware, clusters or machines! (and the reason why we saw the system process 3 billion JSON docs in 48 hours! I like to compare it with Google's 8 billion searches per day - we aren't nearly as complex but 3 billion isn't your pocket change.)   

Additionally, with our #LetsData Lambda Compute Engine, we've not only packaged AWS Lambda for ease of data compute, but we've also extended the service with many little things that will delight you once you start using the service. For example, Lambda function runtime today has an upper limit of 15 mins. In case your task takes longer (and data intensive tasks that read 10s of gigabytes of data files can take longer), we've built in automated task rerun capabilities. This essentially means your tasks relinquishes control and reschedules for a rerun. 

I liken this to this to thread scheduling, premption and execution that the OSes do. For example, In an experiment that I ran, I fixed the task timeout to 60 seconds (config) and saw that the tasks were getting run for 60s, then giving the compute to other tasks and then getting rescheduled again. Very neat implementation and how it manifests in use, IMHO!

For example, reading from a partitioned stream as tasks for each partition and the partition doesn't have data available yet, this ensures there isn't starvation in a partition (similar to IO schedulers in low level OS implementations).

(I mentioned config a couple of times - this isn't traditional config, its actually the dataset configuration JSON that you create on defining your read, write, error and compute - we are not a config heavy system IMO, I believe our design philosophy follows this as a principle and that leads to simplicity. However, I acknowledge that we are just an MVP that does not have a gamut of features so becoming config heavy maybe the only option as we grow (i doubt that somehow), but as of now, I do believe this offers an ease of use. Also, I may have an owner's bias and some user might find our config overwhelming.)

I believe that this distinguishing feature solidly puts us in the "Your Code" (and maybe "Low Code") category instead of "No Code". Also, if I may bring another term into the mix, we are "No Provisioning" (true serverless). Consider "No Provisioning" design parameter when you design your infrastructure and realize the infrastructure and operability simplifications that this offers.

However, "No Code" / Zero ETL features are highly relevant and important - when the compute is simple copy to data into destination, they work really well (and I was impressed by how AWS has provided these with such ease of use). 

For example, built-in automation, turn on once and any new files / data would keep on syncing automatically. Debugability and batching etc. We might just use these Zero ETL features as is and package them with the #LetsData resources such as logging, metrics, tasks etc to be the "No Code" pieces in our data pipelines. This would "build this in code" for better manageability instead of rogue actions such as enabling in studio or on the console which can be source of operator errors. We can integrate with their API so that you dont have to!

So these are my thoughts about #LetsData positioning with respect to the "No Code" and "Zero ETL" offerings. We can integrate with them for the basic copy tasks and offer a compute that can do much much more!

Expo Hall

Next, I headed to the expo hall and started chatting with a number of different companies and understanding what they do. Lots of companies in the data pipelines, observability and monitoring space. Almost all the open source SQL and NoSQL variants had a presence. Companies that increase developer agility, decrease costs, secure software development end to end and similar are all on the display. 

From the expo hall, I made my way to the "Architecting for low latency and performance in financial services" session when it was the session time. There was a large queue and on the session time, the hall filled quicker and a number of attendees had to be turned away. Those early birds had beaten me to the punch :) Nevermind, I'll catch up on this session via the session recording on youtube when they are available. Back to the Expo hall I went. There was some time before the Keynote so spent some time in the Expo Hall talking to some folks. Some additional conversations around data pipelines and visiting card swaps. 

Keynote

Realizing that queuing and limited capacity may be facts of the AWS Event, I turned on my theme park mode from my younger years to optimize ride wait times etc. Actually kidding, the queuing and limited capacity weren't that big of an issue, I just gave myself ample time for the Keynote and any events I wanted to attend instead of showing up on the event time. 

The key takeaway from the Keynote was that AWS was making big investments in the AI and ML spaces - lots of new features, simplifications, cost reductions and ease of use for AI / ML.

During the course of my startup, a few people had asked me around how we can possibly benefit from the AI / ML wave that is sweeping the industry. I had read a few articles, some architecture docs and these two do a really good job at explaining these:

• Emerging Architectures for LLM Applications (an Andreessen Horowitz blog): https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/

• How OpenAI trained ChatGPT (an excellent summary of the MS Build talk):

https://blog.quastor.org/p/openai-trained-chatgpt

We definitely fit well in the "Data Preprocessing / Embedding" component of the architecture in the Andreessen Horowitz blog. In addition, from the MS Build talk summary (emphasis mine)

"The Data Mixture specifies what datasets are used in training. OpenAI didn’t reveal what they used for GPT-4, but Meta published the data mixture they used for LLaMA (a 65 billion parameter language model that you can download and run on your own machine).

...

From the image above, you can see that the majority of the data comes from Common Crawl, a web scrape of all the web pages on the internet"

Considering that we have demonstrated scale and performance in reading and processing Common Crawl docs "Case Study: Big Data: Building a Document Index From Web Crawl Archives", I believe we'd be very highly relevant here as well.

In addition, maybe the Compute can be leveraged for some training as well, but this seems highly specialized. I need to spend some time and work through a few example usecases and see how everything is working to internalize how #LetsData can be used in these scenarios. I'll say more about it once I myself internalize this a little better. (Task for self, deep dive into the OpenAI's / Meta's technology).

Headed for lunch after the keynote, grabbed a Tuna Sandwich on Ciabatta and a Diet Coke, headed out to the benches to enjoy a meal in the Sun. Hadn't done this in ages, for no specific reason, the warmth of the summer Sun made one of my liked rituals rather enjoyable. 

Making a modern data architecture a reality

Headed over to "Making a modern data architecture a reality" and learned about lake formation in AWS S3 via AWS Glue.

• Glue Crawlers infer data schema and add it as schema in the Glue Data Catalog

• Glue Transforms by doing an SQL join of two different S3 data sources and writing it back to S3 via the Glue Studio's drag and drop designer. The designer auto generated a python script that gets run (seemed like Spark code)

• Again, a Glue Crawler to infer the schema from the joined data

• Athena to query over the S3 data using the schema from the Glue Catalog

• Add a few neat Quicksight visualizations Heatmap and Treemap to round out the exercise

Data Cataloging is important for data governance and the built in Glue transformations and crawl infrastructure seems really interesting as well. Having a data catalog and Athena querying built in seems like a useful addition to #LetsData. We need to experiment with these a little and see what would be most beneficial for the customers and build something that the customers would use.   

Unlike the earlier lab, this one was a little different in that this was BYOL (bring your own laptop) - but the software infra of simple login with AWS cloud resources provisioned and available was VERY impressive. (pat on the back for folks who made this possible)

Expo Hall

Back in the Expo Hall one more time. A fun number of activities were also in the Expo Hall, professional executive headshots, games such as ping pong etc. The line for the professional headshots was a little thinner, and in between my conversations, I got these done. Not very flattering though, I guess, 20 hours of travel and tiredness were showing through. Ah well, I like my current LinkedIn pic quite well. 

AWS Startup Loft had a large presence on the Expo floor. I met with a few different Solution Architects and discussed the startup and got some tips around execution and engagement and the customary exchange of visiting cards. Next stop a few additional companies, asking as to what possible integrations and synergies might exist with #LetsData. 

At this time, I went to the large lobby area and chilled out on some benches, seeing folks and the event pass by. I was a little tired and thought I'd stretch my legs and wait for my next talk.  

Serverless SaaS: Building for multi-tenancy

AWS Community section on the Expo Hall were talking about Serverless SaaS and building for multi tenancy. Most of this was what I already knew from the AWS Well Architected Framework SaaS Lens (www). I had read this cover to cover and more for #LetsData and solved some really tricky challenges along the way but hearing this reiterated was fun. 

Almost 5PM, my flight's at 7:30, no time for the after event socials, at-least not this time. Uber back to the airport, checkin, wait, board, disembark drive back home.

1AM at home but there are people waiting for me to come back safely.

Alhamdulilah (Thank God), Life's Good!

#LetsData is now a trusted APN (AWS Partner Network) Partner and we are now listed in the AWS Partner Solution Finder listing.

We've also earned this shiny new AWS Partner badge!

The AWS Partner Network membership has many advantages and the validation criteria is thorough. A company has to follow the AWS best practices and undergo a Foundational Technical Review (FTR), which is designed to ensure the company follows best practices across categories such as auditing, logging, access controls, security etc. 

While we've built #LetsData following software best practices and have taken care that anything that customers / developers find nagging is eliminated, the FTR uncovered a range of things that takes our offering from a solid, startup service offering to an Enterprise grade offering. 

The value add of the simple FTR process is immense IMHO. In addition, the process emphasizes usage of tools and services (Management, Governance, Security, Identity and Compliance ) that are not mainly used when developing a product but are lifesavers when things go wrong. The FTR becomes a great ramp-up on the operational side of things. 

Let's look at the tools and services that we discovered as part of the FTR process and are now using continuously to shore up our compliance and governance infrastructure. 

Security Hub

AWS Security Hub automate AWS security checks and centralize security alerts. Its fairly easy to get started and use. It comes with a few different default benchmarks that once can enable. It finds all sorts of violations and deviations from best practices that are nice fixes to the product. An extremely useful service IMHO! 

AWS Config

AWS Config assess, audit, and evaluate configurations of the AWS resources. The way it works is that you enable the AWS Config recorder which starts recording the resource configurations and any changes to these. For example, an S3 bucket ACL change etc. 

Interestingly, it allows the users (and services such as security hub) define configuration rules that they are interested in. These rules are continuously evaluated and the results are forwarded to the default bus on AWS Event Bridge. Event Bridge allows you to build custom processing logic on each rule and then forward them to the whole gamut of AWS foundational services(SNS, Lambda, Cloudwatch etc).   

This is a powerful construct, because now this service becomes a foundational building block - you can detect any configuration change that you are interested in and invoke custom actions / pipeline. I was really amazed at the end to end around this - a large part of this is the richness of Event Bridge - but the overall end to end is a very complete experience. A job really well done guys!   

AWS Trusted Advisor

AWS Trusted Advisor provides recommendations that help you follow AWS best practices, reduce costs, improve performance, improve security. This service looks at the more practical side of the operations, things such as Cost optimization, Performance, Fault tolerance, Service limits and Security as well. 

We didn't find many issues when we ran this, some of this could be because:

• Trusted Advisor has tiers that are tied to the AWS Account's customer support plans. We were in the basic plan, so we didnt get to run the full gamut of checks.  

• we dont have real usecases running at this time, so cost, performance and service limits issues probably wouldn't surface at this time

• there are comprehensive checks for AWS services such as EC2, EBS, RDS etc that we are not currently using.

But overall looks like a great service to have continuously running and once we have some customer traction, I might just upgrade to premium support so that I can get the full gamut of Trusted Advisor checks!

AWS Guard Duty

Intelligent Threat Detection to Protect Your AWS Accounts and Workloads. This service looks at a bunch of different logs to flag issues that could be possibly security threats. In our case, it flagged for iam user root credential usage, bucket policy changes and changes in the cloudtrail logging configuration. Additional controls for EKS, RDS and Malware probably didn't run (since we are no in those services yet). We're keeping this running, let see what threats we discover over time. 

Amazon CloudTrail

Track user activity and API usage. This again is a foundational service that integrates with every other service in AWS and audits whatever actions are being done and by whom. I've used this to debug issues or two ever once in a while. However, the FTR's emphasis on cloudtrail enablement, cloudtrail logs security and paranoia over having these logs delivered to a different account underscores the importance of the data that this service logs. Its your complete AWS central audit system that can help trace every action that took place in the account. Here are some notable observations: 

• the data cloudtrail captures is very important and unlocks newer use cases of compliance, security and more. 

• cloudtrail auto generates insights from this data, such as the Cloudwatch Log service is seeing 465% increase in Create Log Stream API calls at this time etc. 

• integrates with data lakes, event data stores and allows sql query semantics over this data. 

  ---

AWS Event Bridge

Although not in the FTR services toolset, but I did want to mention this service since I was amazed when I used it in #LetsData use cases. The concept of event buses and custom rules trigged via events on the event buses or by cron is very powerful. 

The filtering mechanisms are simple to implement and the complexity from the large number and different types of events is very beautifully abstracted in the console by having all the sample events available to test in the console. 

And the auto delivery integrations to SNS, Lambda and Cloudwatch Events complete the end to end for this really well.

And the fact that systems can publish automatically on the default event bus without you having to configure different services, grant permission and other similar actions makes it simple to use. 

For example, to setup the "Alarm when S3 bucket becomes public", all one had to do was to copy the name of the Security Hub config rule for s3 bucket becomes public and create an event filter on the default bus to trigger when rule matches and is in alarm. Delivering this to cloudwatch logs automatically and configuring a cloudwatch alarm on these events completed the end to end without much fuss. Great vision folks!

AWS Backup

AWS Backup is a cost-effective, fully managed, policy-based service that simplifies data protection at scale and centralizes backup and restore. We enabled backups for all S3 buckets in the account. This included buckets that had customer data as well as #LetsData buckets. Great centralized backup service - did require some finagling to get it working - ideally I'd want this to be as easy and facile as DynamoDB Point in Time Restore (that feature works like a charm out of the box!)

What we fixed

Okay, now that we know what all was run, here are the issues that we found and fixed:

Cloudfront and S3 Bucket interactions

Cloudfront and S3 integrations are easy to configure and developers can get started in a few clicks. However, this also easily leaves secrity bugs if not configured properly. The Security Hub disabled public access on our S3 buckets (which we had behind the cloudfront, this was a remnant from when we didn't have cloudfront and public objects were used), configured default objects and configured controls that would disallow discovering unintended content in the S3 bucket using path based attacks. Benign but important. Great set of checks!

S3 Buckets

S3 being central to almost every usecase at AWS, gets a larger number of security checks to make sure the bucket configurations are correct. Here is what we enforced:

• we locked our S3 buckets to use SSL transport only (no man in the middle attacks)

• enabled block public access on all buckets so that even accidentally nothing leaks 

• enabled versioning on all our buckets

• we even added alerts that alarm when a bucket becomes public! And interestingly, the way alert was done uncovered a very powerful foundational pattern on the AWS IMHO (see AWS Event Bridge above) 

Again, great set of standardizations!

Identity and Access Management:

Security Hub flags for issues in the AWS account's identity, access and credentials usage. Things such as:

• disabling root security credentials - I had resisted this for the longest time, but if the system is built using the security best practices such as IAM Roles granting temporary access credentials, disabling the root security credentials is a validation that your system is secure!

• add hardware MFA to root account, software MFA to IAM accounts with console access, enforcing strong passwords and so on. These controls do hamper mobility, I believe when it comes to running secure enterprise services, such controls are a must. Also, they prevent operator errors that can occur from time to time since we are only human. Recommendations on rotation of keys and credentials every 90 days, audit access frequently and disable employee access on leaving company instills you to make sure that you have processes around these.

• standardize cross-account access via IAM Roles (we were already doing this) and a randomized / not guessable externalId when granting temporary credentials for an additional security layer as a solution to the confused deputy problem https://aws.amazon.com/blogs/apn/securely-accessing-customer-aws-accounts-with-cross-account-iam-roles (we added this)

• secret and credential storage in Secrets Manager (we were already doing this)

• we created an incident response runbook following the whitepaper: https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/runbooks.html - we normally know what to do when an incident occurs and these processes do formalize over time IMO as the issue collateral builds up. However, having a process defined at this time was useful in making sure we had everything in place to respond to any incidents.

We are using IAM according to the best practices! Great validation to have!

Data Security

We had already classified the different types of data in #LetsData and published it as part of the privacy policy and data at rest and data in transit was being encrypted. We audited this again as part of the FTR and found no violations. We are encrypted at rest and in transit! 

Cloudtrail

We audited what events we had had enabled for cloudtrail. Additionally, we enabled log security and have them delivered to a different AWS account. This enabled us to have a complete auditing solution in place in case we need to trace any issues! 

Backup and Restore

The FTR asks you to do your Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO) calculations to define what the service's recovery objectives are (SLAs).

Since the durability of Dynamo DB and S3 have so many 9s, we had taken this for granted that data loss would probably be only an academic concern. However, an important component of data loss is functional bugs that we or the partner code might have.

For example, if someone pushes a code bug that updates the database rows to an incorrect value, your data durability is toast. Granted, its a bug and fault of the code writer, but finger pointing and blame accusations are pointless in this scenario. As a resilient service, if this happpens, what will we do about it?

This logic had us enabling AWS Backups on all the S3 buckets and DynamoDB point in time restore (PITR) on all the Dynamo DB tables. We defined backup schedules and retention duration and came up with approximate RPOs and RTOs. We also did resilency tests (simulate data loss and then restore from backups)! And because we've built this into the service, #LetsData customers get this for free and can be assured that we have their data covered according to enterprise data best practices. Thanks FTR - we had completely missed this one! 

So that is what we had been upto in the last week or so! We are a trusted AWS Partner and have been validated for following the AWS foundational best practices for software running on AWS! You can engage with us either directly or though our partner page, knowing that AWS vouches somewhat for what we have to offer!

Originally posted as LinkedIn Article on the #LetsData page - moved to this substack to backfill the blog