Category: *unix

One more cloudflare blog that I had in the to-read list:

https://blog.cloudflare.com/when-bloom-filters-dont-bloom

I had never heard about Bloom filters so that was interesting and the actual uses of them:

https://en.wikipedia.org/wiki/Bloom_filter#Examples

I like his point to chose  ‘m’, number of bits in the bit array, to be a power of two (module operation becomes a bitwise AND):

https://stackoverflow.com/questions/41183935/why-does-gcc-use-multiplication-by-a-strange-number-in-implementing-integer-divi

But at the end, it is not all about the Bloom filters. It is understanding how things work under the hood and see if they are actually delivering, if not, you should change your approach. So the debugging section “A secret weapon – a profiler” is very good. Profiling is not one of my strengths so the tools used are the ones I need to understand and use more often:

strace -cf
perf stat -d
perf record
perf record | head -n 20
perf annotate process_line --source
google-perftools' with kcachegrind  

As well the reference to the performance numbers that are good to have in mind:

http://highscalability.com/blog/2011/1/26/google-pro-tip-use-back-of-the-envelope-calculations-to-choo.html

So I take a copy here:

• L1 cache reference 0.5 ns
• Branch mispredict 5 ns
• L2 cache reference 7 ns
• Mutex lock/unlock 100 ns
• Main memory reference 100 ns
• Compress 1K bytes with Zippy 10,000 ns
• Send 2K bytes over 1 Gbps network 20,000 ns
• Read 1 MB sequentially from memory 250,000 ns
• Round trip within same datacenter 500,000 ns
• Disk seek 10,000,000 ns
• Read 1 MB sequentially from network 10,000,000 ns
• Read 1 MB sequentially from disk 30,000,000 ns
• Send packet CA->Netherlands->CA 150,000,000 ns 

Things to take in mind:

• Notice the magnitude differences in the performance of different options.
• Datacenters are far away so it takes a long time to send anything between them.
• Memory is fast and disks are slow.
• By using a cheap compression algorithm a lot (by a factor of 2) of network bandwidth can be saved.
• Writes are 40 times more expensive than reads.
• Global shared data is expensive. This is a fundamental limitation of distributed systems. The lock contention in shared heavily written objects kills performance as transactions become serialized and slow.
• Architect for scaling writes.
• Optimize for low write contention.
• Optimize wide. Make writes as parallel as you can.

As well, “The lessons learned” is a great summary of his trip.

• Sequential memory access great / Random memory access costly -> cache prefetching
• Advanced data structures to fit L3: optimize for reduced number loads than the amount of memory used.
• CPU hits the memory wall

So another great post from Marek.

A couple of weeks ago, at work, sysadmin guys were working on some ZFS issues. They were talking about ZIL and ARC, and I had no idea what was that.

I always wanted to run ZFS, so I think early 2019 I configured my laptop to use ZFS, not in the root partition but in a different partition. I had to configure my Debian Testing to support ZFS (I dont remember if it was very difficult) and then backup some data to make room for my new ZFS partition.

For ZFS basics, you can follow the link below but there are many good tutorial searching in your favourite engine:

In my case, it is a laptop, so I just have one pool that is based on my LV “storage”. I think this was the command I used:

#zpool create -o mountpoint=/home/username storage /dev/mapper/laptop--vg-storage

# zpool status
  pool: storage
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 0 days 00:10:39 with 0 errors on Sun Jan 12 00:34:40 2020
config:

	NAME                  STATE     READ WRITE CKSUM
	storage               ONLINE       0     0     0
	  laptop--vg-storage  ONLINE       0     0     0

errors: No known data errors
# 

And that would be mounted where I requested

$ df -hT | grep zfs
storage        zfs       176G   73G  103G  42% /home/username/storage

This is too basic, in most cases your will want to have a kinf of RAID. But again, this is a simple laptop. As well, you can configure snapshots (useful if you want to have rollback a server upgrade that involves a huge amount of data) and other performance parameters (as per document below):

https://www.percona.com/live/17/sites/default/files/slides/pl17_ZFS_MySQL_Salesforce_0.pdf

So once you have your ZFS configured and mounted you can work with it as usual.

So back to the ZIL and ARC. Based on the links below:

• ZFS Intent Log, or ZIL, to buffer WRITE operations.
• ARC and L2ARC which are meant for READ operations.

In my laptop, I dont have any space left to play with this, so I can only check in my employer systems.

I have read the below article and I am going to give it a go in my laptop

https://www.simula.no/file/lj-219-jul-2012pdf/download

First, check the status of tcp thin:

# sysctl net.ipv4.tcp_thin_linear_timeouts
net.ipv4.tcp_thin_linear_timeouts = 0

I have realised that I dont have “/proc/sys/net/ipv4/tcp_thin_dupack” as the article mentions…

Ok, let update the value and be sure is still active after reboot.

Enable the value:

# echo "1" > /proc/sys/net/ipv4/tcp_thin_linear_timeouts
# sysctl net.ipv4.tcp_thin_linear_timeouts
net.ipv4.tcp_thin_linear_timeouts = 1

Make it permanent, edit /etc/sysctl.conf like this

# Based on https://www.simula.no/file/lj-219-jul-2012pdf/download
# enabling tcp thin-steam modifications for reducing latency in interactive apps
net.ipv4.tcp_thin_linear_timeouts = 1

Now it is time to test and see if you see any improvement or degradation!

This week attended a webinar from Alex Blewitt about CPU microarchiteture to increase application performance. The link was sent by a work colleague but you can get pdf and see the presentation from the below source:

https://www.infoq.com/presentations/microarchitecture-modern-cpu

The presentation is obviously very interesting. You need to know your CPU to get the most of it.

How can I see my CPU architecture? “lstopo” (graphic) or “lstopo-no-graphics” is your friend. This is from my laptop.

# lstopo-no-graphics
Machine (7934MB total)
Package L#0
NUMANode L#0 (P#0 7934MB)
L3 L#0 (4096KB)
L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#2)
L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
PU L#2 (P#1)
PU L#3 (P#3)
HostBridge
PCI 00:02.0 (VGA)
PCIBridge
PCI 02:00.0 (Network)
Net "wlp2s0"
PCI 00:1f.2 (SATA)
Block(Disk) "sda"
Misc(MemoryModule)
Misc(MemoryModule)
#

As you can see, my humble laptop just have one NUMA node, with two cores/processors, and two hyperthreads per core.

But in a server, very likely you will have more NUMA nodes, more cores and more processors so you want to be sure that is used properly.

I am not expert in CPU performance at all but there many important points like memory allocation, huge pages, pinning memory/threads (isolcpu, taskset, etc), compiler strategies and tools to test the performance. Some of them ring the bell and it is nice to know that exist. You never know when you will have to dive in this type of water.