Daily Archives: May 21, 2021

Aws cli, put/get kinesis record

AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=xxxx AWS_SESSION_TOKEN=xxxx \
        aws kinesis put-record \
        --region us-east-1 \
        --stream-name ${streamName} \
        --data "{\"pengliid\": \"pengli\",\"value\": [\"1\", \"2\", \"3\", \"4\"]}" \
        --partition-key pengli-id

In the output, remember the sequence number and shardId

AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=xxxx AWS_SESSION_TOKEN=xxxx \
        aws kinesis get-shard-iterator \
        --region us-east-1 \
        --stream-name ${streamName} \
        --shard-id ${shard_id} \
        --shard-iterator-type AT_SEQUENCE_NUMBER \
        --starting-sequence-number ${sequence_number}

remember the shardIterator

AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=xxxx AWS_SESSION_TOKEN=xxxx \
        aws kinesis get-records \
        --region us-east-1 \
        --shard-iterator ${shardIterator}

Output is base64 format. Transform it.

kinesis_output

base64

Kinesis get-records shard-iterator

command1: aws kinesis get-records --shard-iterator xxxxxx
return:
  {
    records: [record1]
      nextiterator: yyyyyyy
  }

command2: aws kinesis get-records --shard-iterator yyyyyyy
return:
  {
    records: [record2]
      nextiterator: zzzzzzz
  }
Category: aws

Cache TTL consideration

  1. When there is no eviction mechanism, we can use ttl to invalidate. But this is based on that cache might have stale data.
  2. TTL doesn’t lead 100% cache correctness. If we want to guarantee cache correctness, we need evection mechanism, write-through etc strategies.
  3. In theory, TTL is just a way of data replacement. Others are LRU, LFU.