Category Archives: aws

DynamoDB access in Java(structure)

1. dynamodb vs dynamodb-enhanced 1). software.amazon.awssdk, dynamodb 2). software.amazon.awssdk, dynamodb-enhanced, a high-level library. Can map class to DynamoDb tables. Introducing enhanced DynamoDB client in the AWS SDK for Java v2 2. Client vs Table DynamoDbClient/DynamoDbEnhancedClient is like databases access. Think you know the username/password to the database. DynamoDbTable/DynamoDbAsyncTable is the table you will operate on.… Read More »

aws profile

➜ GettingStarted git:(master) ✗ aws configure –profile “sandbox” AWS Access Key ID [None]: AWS Secret Access Key [None]: Default region name [None]: us-west-2 Default output format [None]: ➜ GettingStarted git:(master) ✗ aws s3 ls s3://xxxx-bucket –profile sandbox

Category: aws

How Kinesis getShardIterator works

getShardIterator, by its name, it gets the iterator for the shard. get-shard-iterator –stream-name <value> –shard-id <value> –shard-iterator-type <value> [–starting-sequence-number <value>] Let’s say we use getShardIterator, we got iterator on shard2, seq2_0. Then we use this iterator to iterate message in Kinesis shard2 from seq0_0. After one iteration, it returns next iterator which points to seq2_1. If we want… Read More »

Category: aws

Aws cli, put/get kinesis record

AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=xxxx AWS_SESSION_TOKEN=xxxx \ aws kinesis put-record \ –region us-east-1 \ –stream-name ${streamName} \ –data “{\”pengliid\”: \”pengli\”,\”value\”: [\”1\”, \”2\”, \”3\”, \”4\”]}” \ –partition-key pengli-id In the output, remember the sequence number and shardId AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=xxxx AWS_SESSION_TOKEN=xxxx \ aws kinesis get-shard-iterator \ –region us-east-1 \ –stream-name ${streamName} \ –shard-id ${shard_id} \ –shard-iterator-type AT_SEQUENCE_NUMBER \ –starting-sequence-number ${sequence_number}… Read More »

Category: aws

Two ways to read aws access key/secret. Different CredentialProvider

instance can have instance role. When an application run in instance, it can InstanceProfileCredentialsProvider() to retrieve instance role and have the access. For example, instance_profile from EMR cluster is the role for EMR instance. StsAssumeRoleSessionCredentialsProvider AWSStaticCredentialsProvider AWSCredentialsProviderChain. It will test different credential one by one, until it finds one. Here is a code example

Category: aws

Policies setting: aws account is able to upload to s3

Assume Role Way create a aws user. This user doesn’t has any policy. 2. After created user account, it should tell you the ACCESS_ID and ACCESS_SECRET, copy that to somewhere. 3. create iam role. This role needs a policy, which has access to s3 bucket 4. This role should also have trust relationship with user… Read More »

Category: aws

IAM assume policy, IAM policy

1. Create an iam role. During creating the iam role, define IAM assume role policy. Assume policy tells who can assume this iam role. 2. Define IAM Policy. Attach IAM policy to this role. In aws UI, the assume policy will be shown as Trust relationships tab, The normal IAM policy is shown as in… Read More »

Category: aws