Rusoto
Rusoto is an AWS SDK for Rust.
Rusoto consists of one core crate, containing common functionality shared across services, and then a crate for each supported service.
Services are generated from the botocore project's API definitions.
Rusoto also provides a credential crate. The credential crate provides the necessary functionality for properly loading and handling AWS credentials.
Help
If you are experiencing problems with rusoto, or would like clarification regarding something, please don't hesitate to contact us. You can reach us by opening an issue on the GitHub page for the project with the relevant information (code snippets and error messages are very helpful!). Alternatively, you can join us on Discord.
Usage & Example
Rusoto provides a crate for each AWS service it supports, containing a client and all of the associated types for that service. A full list of these services can be found on the Supported AWS Services page.
It also provides a core crate called rusoto_core
, containing all shared
functionality across services, such as the list of regions, signed request senders,
and credential loading.
Consult the rustdoc documentatation by running cargo doc
or visiting the online
API documentation for the latest crates.io release.
An example of using Rusoto's DynamoDB API to list the names of all the tables in a database:
Cargo.toml:
[package]
name = "my-crate"
version = "0.1.0"
authors = ["My Name <my@email.com>"]
[dependencies]
rusoto_core = "0.42"
rusoto_dynamodb = "0.42"
Rust code:
extern crate rusoto_core; extern crate rusoto_dynamodb; use std::default::Default; use rusoto_core::Region; use rusoto_dynamodb::{DynamoDb, DynamoDbClient, ListTablesInput}; fn main() { let client = DynamoDbClient::new(Region::UsEast1); let list_tables_input: ListTablesInput = Default::default(); match client.list_tables(list_tables_input).sync() { Ok(output) => { match output.table_names { Some(table_name_list) => { println!("Tables in database:"); for table_name in table_name_list { println!("{}", table_name); } } None => println!("No tables in database!"), } } Err(error) => { println!("Error: {:?}", error); } } }
Crates that extend Rusoto
Rusoto's current focus is to support all AWS services. This means higher level functionality present in other AWS SDKs are not currently available in Rusoto. See https://github.com/rusoto/rusoto/issues/1098 for more information on where Rusoto efforts are currently focused.
While Rusoto doesn't support various higher level functionality or better ergonomics, there are third party crates available that use Rusoto and provide this behavior.
A non-exhaustive list of projects that build on Rusoto to make it more ergonomic or provide higher level functionality:
- dynomite:
make your rust types fit dynamodb and visa versa
- envy-store:
deserialize AWS Parameter Store values into typesafe structs
- S4:
Simpler Simple Storage Service for Rust
If you'd like to add an example to this list, please open an issue or pull request on the rusoto.org repo.
Lambdas
Lambdas are a great place to use Rusoto! The runtime performance keeps costs down while the Rust language prevents classes of bugs found in other common languages used with AWS Lambda.
Building for Lambda
Lambda environments are x86 64 bit Linux. Existing Rust code can be cross-compiled from a different platform for this target. For example, on a Macbook, one can compile a Linux executable for Lambda. See Creating my first AWS Lambda using Rust for more information on setup.
Rustls
Rustls is modern TLS library written in Rust
. It can replace OpenSSL in many places, including Rusoto. This makes it far easier to get a working lambda executable. No more The OpenSSL library reported an error
on lambda startup because an OpenSSL flag isn't set right!
The rustls
option must be set on the project's Cargo.toml file for both the rusoto_core
crate and any Rusoto service crates used. For example, to use rustls
with S3 and SQS:
rusoto_core = {version = "0.42.0", default_features = false, features=["rustls"]}
rusoto_s3 = {version = "0.42.0", default_features = false, features=["rustls"]}
rusoto_sqs = {version = "0.42.0", default_features = false, features=["rustls"]}
The default features must be disabled since that uses the hyper-tls
crate. These flags can be found in the rusoto_core Cargo.toml.
Regions
Rusoto supports all regions AWS supports. There is also a Default
implementation that reads from environment variables AWS_DEFAULT_REGION
or AWS_REGION
. For more information and to see the list of regions, see the Region API documentation.
Custom Regions
Custom Regions can be used for interacting with services that behave like AWS services, such as Ceph, Minio and local DynamoDB.
For a complete example of using local DynamoDB, see this sample project.
extern crate rusoto_core; extern crate rusoto_dynamodb; use rusoto_core::Region; use rusoto_dynamodb::{DynamoDb, DynamoDbClient, ListTablesInput}; fn main() { // Create custom Region let region = Region::Custom { name: "us-east-1".to_owned(), endpoint: "http://localhost:8000".to_owned(), }; let client = DynamoDbClient::new(region); let list_tables_request = ListTablesInput::default(); let tables = client.list_tables(list_tables_request).sync(); println!("Tables found: {:?}", tables); }
Using Rusoto Futures
Rusoto Futures are like other Futures in Rust. They implement the Future
trait:
Impl Future for RusotoFuture
Treating all calls to AWS services as asynchronous allows for different ways to handle network calls, potentially improving throughput and resource usage.
All API calls return a future
All Rusoto calls to AWS return a Future. This Future can be acted on immediately, saved for later use or chained/combined with other Futures before running.
.sync()
Rusoto supports returning Futures to be executed later as well as synchronous, blocking calls. The .sync() function on Rusoto Futures allows this behavior.
Under the hood it uses a tokio::runtime
to immediately run the Future..
Chaining futures together
The usual collection of Future combinators can be used on Rusoto Futures. To read more about combinators, see the tokio docs on Futures. Another resource to help sort out compilation errors with Futures is the Futures Cheatsheet.
The example below uses the .then()
combinator to chain multiple Futures together.
Examples
See Chaining Rusoto Futures blog post for more information and links to complete examples.
extern crate futures; extern crate rusoto_core; extern crate rusoto_dynamodb; extern crate tokio_core; use futures::future::Future; use rusoto_core::{Region, RusotoError}; use rusoto_dynamodb::{ AttributeDefinition, AttributeValue, CreateTableInput, CreateTableOutput, DynamoDb, DynamoDbClient, GetItemError, GetItemInput, GetItemOutput, KeySchemaElement, UpdateItemInput, UpdateItemOutput, }; use std::collections::HashMap; use tokio_core::reactor::Core; fn main() { let item = make_item(); let client = get_dynamodb_local_client(); let mut core = Core::new().unwrap(); let create_table_future = make_create_table_future(&client); let upsert_item_future = make_upsert_item_future(&client, &item); let item_from_dynamo_future = make_get_item_future(&client, &item); let chained_futures = create_table_future .then(|_| upsert_item_future) .then(|_| item_from_dynamo_future); let item_from_dynamo = match core.run(chained_futures) { Ok(item) => item, Err(e) => panic!("Error completing futures: {}", e), }; println!("item: {:?}", item_from_dynamo); } fn make_create_table_future(client: &DynamoDbClient) -> impl Future<Item = CreateTableOutput> { let attribute_def = AttributeDefinition { attribute_name: "foo_name".to_string(), attribute_type: "S".to_string(), }; let k_schema = KeySchemaElement { attribute_name: "foo_name".to_string(), key_type: "HASH".to_string(), // case sensitive }; let make_table_request = CreateTableInput { table_name: "a-testing-table".to_string(), attribute_definitions: vec![attribute_def], key_schema: vec![k_schema], ..Default::default() }; client.create_table(make_table_request) } fn make_upsert_item_future( client: &DynamoDbClient, item: &HashMap<String, AttributeValue>, ) -> impl Future<Item = UpdateItemOutput> { let add_item = UpdateItemInput { key: item.clone(), table_name: "a-testing-table".to_string(), ..Default::default() }; client.update_item(add_item) } fn make_get_item_future( client: &DynamoDbClient, item: &HashMap<String, AttributeValue>, ) -> impl Future<Item = GetItemOutput, Error = RusotoError<GetItemError>> { // future for getting the entry let get_item_request = GetItemInput { key: item.clone(), table_name: "a-testing-table".to_string(), ..Default::default() }; client.get_item(get_item_request) } fn make_item() -> HashMap<String, AttributeValue> { let item_key = "foo_name"; let mut item = HashMap::new(); item.insert( item_key.to_string(), AttributeValue { s: Some("baz".to_string()), ..Default::default() }, ); item } fn get_dynamodb_local_client() -> DynamoDbClient { // Create custom Region let region = Region::Custom { name: "us-east-1".to_owned(), endpoint: "http://localhost:8000".to_owned(), }; DynamoDbClient::new(region) }
Disabling SSL Certificate Check
⚠️ Danger ⚠️
Disabling certificate checking is not part of regular use of Rusoto. There are very few legitimate reasons to disable default behavior. Production use of Rusoto should not use the behavior documented on this page. Use of AWS services should not disable certificate checking.
Use this only if you know what you're doing.
Opting out of certificate checking
The process of disabling an SSL cert check requires a few steps before using the services in Rusoto. The general steps are as follows:
- Create a
TlsConnector
using the TlsConnectorBuilder. Ensure the certificate check has been disabled. This is the important part! - Create a HttpConnector, this is needed to implement the From trait on the HttpsConnector.
- Create a HttpsConnector
using
from
, passing in the HTTP and TLS connectors. - Create a Rusoto
HttpClient
using thefrom_connector
method. - Initialize a service using the
new_with
method. - Build Things!
extern crate hyper; extern crate hyper_tls; extern crate native_tls; extern crate rusoto_core; extern crate rusoto_s3; use hyper::client::HttpConnector; use hyper_tls::HttpsConnector; use native_tls::TlsConnector; use rusoto_core::{DefaultCredentialsProvider, HttpClient, Region}; use rusoto_s3::{S3Client, S3}; fn main() { // 1. create tls connector which accepts invalid certs let tls_connector: TlsConnector = TlsConnector::builder() .danger_accept_invalid_certs(true) .build() .expect("failed to build tls connector"); // 2. create http connector - make sure to not enforce http let mut http_connector = HttpConnector::new(4); http_connector.enforce_http(false); // 3. create https connector let https_connector = HttpsConnector::from((http_connector, tls_connector)); // 4. create rusoto http client to be used in rusoto services let http_client = HttpClient::from_connector(https_connector); let cred_provider = DefaultCredentialsProvider::new().expect("failed to create cred provider"); // custom region to talk to local service with a self-signed cert let local_region = Region::Custom { name: "us-east-1".to_owned(), endpoint: "https://localhost:8000".to_owned(), }; // 5. initialize service clients with new http client let s3_client = S3Client::new_with(http_client, cred_provider, local_region); // 6. build! s3_client .list_buckets() .sync() .expect("failed to list buckets"); }
⚠️ Danger ⚠️
Do not use this for communicating to AWS services. Use this only for AWS-like services such as Ceph or Minio and generating trusted certificates is not possible.
Supported AWS Services
Debugging
Rusoto uses the log logging facade. For tests, Rusoto uses
env_logger
.
In order to see logging output (e.g. the actual requests made to AWS),
env_logger
needs to be initialized:
extern crate rusoto_core; extern crate rusoto_s3; extern crate env_logger; use std::default::Default; use rusoto_core::Region; use rusoto_s3::{S3, S3Client, ListObjectsRequest}; fn main() { let _ = env_logger::try_init(); // This initializes the `env_logger` let bare_s3 = S3Client::new(Region::UsWest2); let mut list_request = ListObjectsRequest::default(); list_request.bucket = "rusototester".to_string(); let result = bare_s3.list_objects(list_request).sync().unwrap(); println!("result is {:?}", result); }
To see the output of logging from integration tests, the command needs to be run as follows:
RUST_LOG=rusoto,hyper=debug cargo test --features all
To get the logging output as well as the output of any println!
statements,
run:
RUST_LOG=rusoto,hyper=debug cargo test --features all -- --nocapture
If more debugging is required, all debug info including details from the compiler can be seen by setting RUST_LOG to debug
. This will be noisy but will give a lot of debug information. For example:
RUST_LOG=debug cargo test --features all
Performance
Currently, performance isn't benchmarked.
To get the best runtime performance in a project using Rusoto, compile with the --release
flag.
For example: cargo build --release
Increasing performance with large objects
As of Rusoto 0.36.0, we expose the http1_read_buf_exact_size
setting in the hyper
HTTP client. This can greatly improve performance dealing with payloads greater than 100MB. See https://github.com/rusoto/rusoto/pull/1227 for more information.
Example of increasing the buffer size from the default of 8KB to 2MB:
extern crate rusoto_core; extern crate rusoto_ecs; use rusoto_ecs::{Ecs, EcsClient, ListClustersRequest}; use rusoto_core::{Region, DefaultCredentialsProvider}; use rusoto_core::request::{HttpClient, HttpConfig}; fn main() { // EcsClient configuration demonstrates setting the hyper read_buf_size option // to 2MB: let cred_provider = DefaultCredentialsProvider::new().unwrap(); let mut http_config_with_bigger_buffer = HttpConfig::new(); http_config_with_bigger_buffer.read_buf_size(1024 * 1024 * 2); let http_provider = HttpClient::new_with_config(http_config_with_bigger_buffer).unwrap(); let ecs = EcsClient::new_with(http_provider, cred_provider, Region::UsEast1); match ecs.list_clusters(ListClustersRequest::default()).sync() { Ok(clusters) => { for arn in clusters.cluster_arns.unwrap_or(vec![]) { println!("arn -> {:?}", arn); } } Err(err) => { panic!("Error listing container instances {:#?}", err); } } }
Potential runtime performance improvements
Migrations
Rusoto Versions < 0.38.0 to Versions >= 0.38.0 Error Handling
Rusoto version 0.38.0 introduced the enum rusoto_core::RusotoError
. In versions
before 0.38.0 errors would be typed as themselves such as rusoto_s3::CreateBucketError
.
In versions >= 0.38.0 the service method errors would be wrapped in the new enum,
such as rusoto_core::RusotoError<rusoto_s3::CreateBucketError>
.
Rusoto < 0.38.0:
extern crate rusoto_core;
extern crate rusoto_s3;
use rusoto_core::Region;
use rusoto_s3::{CreateBucketError, CreateBucketOutput, CreateBucketRequest, S3Client, S3};
fn main() {
let s3_client = S3Client::new(Region::UsEast1);
let bucket_name = "itshabib-buckets".to_owned();
match create_bucket_sync(&s3_client, bucket_name.clone()) {
Ok(res) => println!("successfully created bucket! resp: {:#?}", res),
Err(err) => println!("Error creating bucket. err: {:#?}", err),
};
}
// error handling for versions < 0.38
fn create_bucket_sync(
s3_client: &S3Client,
bucket_name: String,
// errors are not wrapped by RusotoError in version < 0.38
) -> Result<CreateBucketOutput, CreateBucketError> {
let create_bucket_req = CreateBucketRequest {
bucket: bucket_name.clone(),
..Default::default()
};
s3_client.create_bucket(create_bucket_req).sync()
}
Rusoto >= 0.38.0:
extern crate rusoto_core; extern crate rusoto_s3; use rusoto_core::{Region, RusotoError}; use rusoto_s3::{CreateBucketError, CreateBucketOutput, CreateBucketRequest, S3Client, S3}; fn main() { let s3_client = S3Client::new(Region::UsEast1); let bucket_name = "itshabib-buckets".to_owned(); match create_bucket_sync(&s3_client, bucket_name.clone()) { Ok(res) => println!("successfully created bucket! resp: {:#?}", res), Err(err) => println!("Error creating bucket. err: {:#?}", err), }; } // error handling for versions >= 0.38 fn create_bucket_sync( s3_client: &S3Client, bucket_name: String, // service method error is wrapped in RusotoError enum ) -> Result<CreateBucketOutput, RusotoError<CreateBucketError>> { let create_bucket_req = CreateBucketRequest { bucket: bucket_name.clone(), ..Default::default() }; s3_client.create_bucket(create_bucket_req).sync() }
Rusoto 0.31.0 to 0.32.0
With async
support, the easiest migration is to use the new simple()
constructor on clients and add .sync()
to the function calls.
Rusoto 0.31.0:
extern crate rusoto_core;
extern crate rusoto_polly;
use rusoto_polly::{Polly, PollyClient, DescribeVoicesInput};
use rusoto_core::{DefaultCredentialsProvider, Region, default_tls_client};
...
let credentials = DefaultCredentialsProvider::new().unwrap();
let client = PollyClient::new(default_tls_client().unwrap(), credentials, Region::UsEast1);
let request = DescribeVoicesInput::default();
println!("{:?}", client.describe_voices(&request).unwrap());
Rusoto 0.32.0 with async:
extern crate rusoto_core;
extern crate rusoto_polly;
use rusoto_polly::{Polly, PollyClient, DescribeVoicesInput};
use rusoto_core::Region;
...
let client = PollyClient::simple(Region::UsEast1);
let request = DescribeVoicesInput::default();
println!("{:?}", client.describe_voices(&request).sync().unwrap());
Note:
DefaultCredentialsProvider
isn't required with the::simple()
constructordefault_tls_client
isn't required with the::simple()
constructor.sync()
is called on the result fromdescribe_voices
. This blocks until the call is finished.
Rusoto 0.24.0 to 0.25.0 or later
As of Rusoto 0.25.0, the Rusoto crate is now deprecated. This decision was made because the single crate implementing all AWS services was too large to compile, especially on TravisCI and Appveyor. The new main crate is rusoto_core
and all services now have their own crate.
To continue implementing new services and reducing compilation times, we've extracted a few core crates and every AWS service we support now has its own crate.
rusoto_core
now contains the core functionality of Rusoto: AWS signature handling, regions, requests to services and XML helpers.
rusoto_mock
was also extracted. Crate users shouldn't need this: it's for developing on Rusoto itself.
rusoto_credential
remains its own crate, providing AWS credential sourcing. If you're working on something that uses AWS and Rusoto doesn't support it, you can use that crate instead of rolling your own AWS credential providers.
The new service crates depend on rusoto_core
.
Required changes
Previously, to bring in a Rusoto implementation of an AWS service, you'd specify something like this in your Cargo.toml file:
rusoto = {version = "0.24", features = ["rds"]}
Now, you'd bring in services like this:
rusoto_core = {version = "0.25.0"}
rusoto_rds = {version = "0.25.0"}
Once the new crates have been brought in, use the new crates in your code. A sample before:
extern crate rusoto;
use rusoto::rds::{RdsClient, CreateDBInstanceMessage, DescribeDBInstancesMessage};
use rusoto::{DefaultCredentialsProvider, Region, default_tls_client};
And after:
extern crate rusoto_core;
extern crate rusoto_rds;
use rusoto_rds::{Rds, RdsClient, CreateDBInstanceMessage, DescribeDBInstancesMessage};
use rusoto_core::{DefaultCredentialsProvider, Region, default_tls_client};
Note there are now two crates required: rusoto_core
as well as the RDS crate, rusoto_rds
. There's also a new trait for each service. In this case it's Rds
and we bring that in. This is used to make calls to services easier to test and improve ergonomics of using Rusoto clients.
Credential timeouts
If AWS credentials aren't found locally, Rusoto attempts to query the ECS container role provider, then the EC2 Instance Profile provider. When running locally, such as in a command line tool, timing out waiting for those services can take 30 seconds or more.
To improve responsiveness of credential sourcing, timeouts can be provided to the credential provider. This example uses a 200ms timeout:
extern crate rusoto_core; extern crate rusoto_s3; use rusoto_core::credential::ChainProvider; use rusoto_core::request::HttpClient; use rusoto_core::Region; use rusoto_s3::{S3, S3Client}; use std::time::{Duration, Instant}; fn main() { let mut chain = ChainProvider::new(); chain.set_timeout(Duration::from_millis(200)); let s3client = S3Client::new_with( HttpClient::new().expect("failed to create request dispatcher"), chain, Region::UsEast1, ); let start = Instant::now(); println!("Starting up at {:?}", start); match s3client.list_buckets().sync() { Err(e) => println!("Error listing buckets: {}", e), Ok(buckets) => println!("Buckets found: {:?}", buckets), }; println!("Took {:?}", Instant::now().duration_since(start)); }
Now, if Rusoto can't find any credentials within 200ms, it will return an error like this:
Error listing buckets: Couldn't find AWS credentials in environment, credentials file, or IAM role.
Last updated for Rusoto 0.34.0 with rusoto_credential 0.13.0.