Presigned URLs in S3

Image by ArtTower from Pixabay

S3 is the Amazon service to store files in the cloud. It is reliable, very reliable, the expected time to lost a single file from a group of 10 million of them is 10000 years. Even other services on Amazon uses internally S3 to store its files. On the bad side, as it is one of the first services that Amazon created, it can be a headache to fine grain permissions form all its capabilites and evolutions, making it difficult to be sure that a file is not accesible for those that should not be allowed.

In S3 you can define what they call a bucket, which is like a directory in a filesystem. The name of the bucket must be unique, not only in your account but in the global namespace from all AWS accounts in the world. That means you have to be creative when picking a bucket name.

A bucket can be private or publicly accessible. In the public side, one of the special uses is to serve static content from as a web server, even html pages from your custom domain. But what if you want to allow users to download files, for example an image, and you don’t want the user to be able to make it public sharing the link to the image?

I’ve played today with a very useful feature for that case. It allows to have a private bucket that can temporary allow the access to a single file to GET or even PUT/POST for a limited amount of time. You’ll need to use AWS SDK of your favourite supported programming language or AWS CLI from command line, to query AWS API for a temporary authorized url. Let’s see how with an example from scratch, installing and using AWS CLI in a Debian based environment.

Make sure you have access to an AWS account (you already have one if you have an account) and generate a pair of AWS Access Key and AWS Secret Access Key from web console.

$> sudo apt instal awscli
$> aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: eu-west-1
Default output format [None]:

Create a local file called piticli with the content you prefer. Let’s create also a new S3 bucket using aws cli

# Create a convenience environment variable with a kind of random bucket name
$> BN="s3://thomarite-blog-test-$RANDOM"
# Let's actually create the bucket
$> aws s3 mb $BN 
make_bucket: thomarite-blog-test-1337
# Let's see it exists
$> aws s3 ls
2020-04-16 23:01:27 thomarite-blog-test-1337
# Now let's upload piticli into the new bucket
$> aws s3 cp piticli $BN
2020-04-17 23:01:45          26 piticli

Now let’s create a presigned url for piticli and store it in PRESIGNED_URL env var. As you can see, the temporary URL includes the bucket name, the file name and new AWS Access Key and signature, and a hint about the expiration date.

# Store the URL into a env var for future use
$> PRESIGNED_URL=$(aws s3 presign $BN/piticli)

That’s the public url and will be valid for 1h by default. You can set the expiration time in aws s3 presign command using the parameter --expires-in and set the seconds allowed until it expires.

Now you have a public url accessible by any browser. Let’s open it via curl:

$> curl -Ls $PRESIGNED_URL
piticli is now… sleeping

And finally to clean things up let’s remove all the files and the bucket in AWS

$> aws s3 rb --force $BN
delete: s3://thomarite-blog-test-1337/piticli
remove_bucket: thomarite-blog-test-1337


Today I’ve received a copy of Serious Cryptography and jumped straight ahead to Chapter 12 talking about Elliptic Curves. I’m more or less aware of how RSA works, but was intruigued for a few years now about how criptography was able to achieve same security capabilities with more efficiency while dealing with less information.

Basically that is a different kind of beast based on the hard it is to retrieve the exponent from a discrete logarithm, instead of rely on the factorization of a number composed from two large primes as it is with RSA. As the size of the numbers are much lower with the same hardness to attack, the keys involved are also way smaller.

Elliptic Curve comes after the properties of the equation that the field of work is based, an equation of the curve of the type y^2=x^3 + ax + b where a and b are pre-cooked parameters that must be carefully chosen to avoid security risks.

In fact, there are two main curves used nowadays. One are a family of curves that the NIST approved, the most famous known as P-256, being the most commonly used in the industry, but also had some critisism because those a and b params in the equation were defined by the NSA and the generation is not completely publicly known. Therefore it could have potentially unknown pre-cooked attacks, althought based on the book, the general consensus by the experts is that there is no problem.

The second one is the curve generated by Daniel J. Bernstein that have pretty much the security of the NIST one and arguabilly a bit more of performance speed. It is called Curve25519 because the (discrete) field it works on is based on the prime number 2^255-19, having a=486662 and b=x .

LVM 101 + Linux disk encryption

Once more post from Cloudflare. I think most Linux distributions already offer by default transparent disk encryption. As far as I can see in my Debian, I have encryption with LVM. I need to write a post about LVM as I have always to google most basic command. “Logic Volume Manager” (LVM) is an abstraction layer for managing storage (maybe too basic explanation but that is how I understand it). When I built my laptop, I had the option (I think it was by default) to choose LVM + encryption (dm_crypt module). So I took that.

So first, how I check my LVM? Well, df -h, will give the first clues

# df -hT
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 794M 2.7M 791M 1% /run
/dev/mapper/laptop--vg-root ext4 24G 17G 6.3G 73% /
tmpfs tmpfs 3.9G 414M 3.5G 11% /dev/shm
tmpfs tmpfs 5.0M 8.0K 5.0M 1% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda2 ext2 237M 155M 70M 69% /boot
/dev/sda1 vfat 496M 60M 437M 13% /boot/efi
/dev/mapper/laptop--vg-home ext4 20G 9.9G 8.7G 54% /home
tmpfs tmpfs 794M 24K 794M 1% /run/user/1000

You see thing with “/dev/mapper” and “vg” (volume group). So you have LVM running.

Some basic LVM notes:

# pvs –> it will show the physical disks, partitions, etc used in your LVM setup and the “vgs” they belong to. PVS stands for “physical volume system”. In my case only the partition sda3 from my physical disk is part of LVM. Physical volumes are used to create Volume groups.

# pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/sda3_crypt laptop-vg lvm2 a-- 237.73g <2.62g

# vgs –> it will show you the volumes in your system, the number of PV they are using and the number of LV they are providing. VGS stands for “volume group system”. In my case, I have just one VG, that is use 1 PV and is providing 4 LV.

# vgs
VG #PV #LV #SN Attr VSize VFree
laptop-vg 1 4 0 wz--n- 237.73g <2.62g

#lvs –> it will show the “logical volumes” you have created from a VG. In my case, I have four LV.

# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home laptop-vg -wi-ao---- 22.00g
root laptop-vg -wi-ao---- 24.31g
storage laptop-vg -wi-ao---- 182.00g
swap_1 laptop-vg -wi-ao---- 6.80g

BTW, how I can see all the partitions in my machine, “fdisk -l”

root@athens:/boot# fdisk -l
Disk /dev/sda: 238.49 GaiB, 256060514304 bytes, 500118192 sectors
Disk model: NISU SSD ALLI
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 1550335 499712 244M Linux filesystem
/dev/sda3 1550336 500117503 498567168 237.8G Linux filesystem

So based on our “pvs” we know “dev/sda3” is part of LVM. How the encryption is happening? The type of partition will tell us

# blkid /dev/sda3
/dev/sda3: UUID="f6263aee-3966-4c23-a4ef-b4d9916f1a07" TYPE="crypto_LUKS" PARTUUID="b224eb49-1e71-4570-8b62-fb38df801170"

So, “crypto_LUKS” is key. Our LVM is running over a partition that is encrypted.

So after this detour, lets go back to Cloudflare post about Linux disk encryption.

I really enjoyed the kind of forensic work trying to discover when and why the changes in the Linux kernel code (!) were happening and how affected the speed. BTW, I crashed my laptop when trying to run their tests!

Iptables Conntrack

I am subscribed to Cloudflare blog as they are in general really good. And definitely, you always learn something new (and want to cry because you have so much to learn from these guys).

This time was a dissection of conntrack in iptables to improve their firewall performance.

I never thought about the limits of the conntrack table and how important is to have in mind (or make a tattoo of) the iptables diagram:

SSH Keys

I already use RSA ssh keys to access my VPS but a friend of mine send me a link about ED25519 public-key algorithm. But why ssh-keys? Mainly to avoid to type your password every single time.

I will not explain the maths behind because I can’t (but I would love to understand) so wikipedia can do a better work (and in the main time, think of donating a few bucks 🙂

If you still want to generate RSA keys (you can have both), this is my go-to link:

Summary, just in case the links disappear:

# create your key RSA or Ed25519

$ ssh-keygen -t rsa -f ~/.ssh/id_rsa4096 -b 4096 -C "user@origin"


$ ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "user@origin"

# Add your priv key into your ssh-agent so it is used when connecting to the destination

$ ssh-add ~/.ssh/id_xxx

# Copy your PUBLIC!!! key to the remote server you want to login with that key (and so you dont need to type a password)

$ ssh-copy-id -i .ssh/ user@remove_server

# Test your new ssh-key

$ ssh -i ~/.ssh/id_xxx user@remove_server