Category: *unix

Something very basic but took me several hours to workout. I had a VM that I wanted to increase a VG as I wanted to create a new LV. I increased the partition in the host server so the PV of the VG had the extra space, but then I couldnt see the increase inside the VM:

[root@HOST]# lvs
  LV      VG  Attr  LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm_data vg_os -wi-ao---- 300.00g

[root@VM]# pvs
  PV         VG      Fmt  Attr PSize    PFree
  /dev/vdb   vg_data lvm2 a--  <200.00g 1020.00m

"fdisk" was telling me the disk was already 300G...

[root@VM ~]# fdisk /dev/vdb

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

The old LVM2_member signature will be removed by a write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xd46fa2fc.

Command (m for help): p
Disk /dev/vdb: 300 GiB, 322122547200 bytes, 629145600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd46fa2fc

I did a pvscan... and nothing. What I was missing? just "pvresize".... and then I can see my extra 100G in the PV and in the VG. So I can create the new LV I wanted...

[root@VM ~]# pvresize /dev/vdb
  Physical volume "/dev/vdb" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized
[root@VM ~]# pvs
  PV         VG      Fmt  Attr PSize    PFree   
  /dev/vdb   vg_data lvm2 a--  <300.00g <101.00g
[root@VM ~]# 
[root@VM ~]# vgs
  VG      #PV #LV #SN Attr   VSize    VFree   
  vg_data   1   1   0 wz--n- <300.00g <101.00g

I like git, I use it, but of course, I am not an expert. And everytime I want to do something outside my comfort zone, I have to serch for help. Will try to add expamples. Most of them will be obvius for most people.

• I want to see the differences between the files I have changed (before commit) and the last commit. Thanks to stackoverflow:

~ git diff

One more cloudflare blog that I had in the to-read list:

https://blog.cloudflare.com/when-bloom-filters-dont-bloom

I had never heard about Bloom filters so that was interesting and the actual uses of them:

https://en.wikipedia.org/wiki/Bloom_filter#Examples

I like his point to chose  ‘m’, number of bits in the bit array, to be a power of two (module operation becomes a bitwise AND):

https://stackoverflow.com/questions/41183935/why-does-gcc-use-multiplication-by-a-strange-number-in-implementing-integer-divi

But at the end, it is not all about the Bloom filters. It is understanding how things work under the hood and see if they are actually delivering, if not, you should change your approach. So the debugging section “A secret weapon – a profiler” is very good. Profiling is not one of my strengths so the tools used are the ones I need to understand and use more often:

strace -cf
perf stat -d
perf record
perf record | head -n 20
perf annotate process_line --source
google-perftools' with kcachegrind  

As well the reference to the performance numbers that are good to have in mind:

http://highscalability.com/blog/2011/1/26/google-pro-tip-use-back-of-the-envelope-calculations-to-choo.html

So I take a copy here:

• L1 cache reference 0.5 ns
• Branch mispredict 5 ns
• L2 cache reference 7 ns
• Mutex lock/unlock 100 ns
• Main memory reference 100 ns
• Compress 1K bytes with Zippy 10,000 ns
• Send 2K bytes over 1 Gbps network 20,000 ns
• Read 1 MB sequentially from memory 250,000 ns
• Round trip within same datacenter 500,000 ns
• Disk seek 10,000,000 ns
• Read 1 MB sequentially from network 10,000,000 ns
• Read 1 MB sequentially from disk 30,000,000 ns
• Send packet CA->Netherlands->CA 150,000,000 ns 

Things to take in mind:

• Notice the magnitude differences in the performance of different options.
• Datacenters are far away so it takes a long time to send anything between them.
• Memory is fast and disks are slow.
• By using a cheap compression algorithm a lot (by a factor of 2) of network bandwidth can be saved.
• Writes are 40 times more expensive than reads.
• Global shared data is expensive. This is a fundamental limitation of distributed systems. The lock contention in shared heavily written objects kills performance as transactions become serialized and slow.
• Architect for scaling writes.
• Optimize for low write contention.
• Optimize wide. Make writes as parallel as you can.

As well, “The lessons learned” is a great summary of his trip.

• Sequential memory access great / Random memory access costly -> cache prefetching
• Advanced data structures to fit L3: optimize for reduced number loads than the amount of memory used.
• CPU hits the memory wall

So another great post from Marek.

A couple of weeks ago, at work, sysadmin guys were working on some ZFS issues. They were talking about ZIL and ARC, and I had no idea what was that.

I always wanted to run ZFS, so I think early 2019 I configured my laptop to use ZFS, not in the root partition but in a different partition. I had to configure my Debian Testing to support ZFS (I dont remember if it was very difficult) and then backup some data to make room for my new ZFS partition.

For ZFS basics, you can follow the link below but there are many good tutorial searching in your favourite engine:

In my case, it is a laptop, so I just have one pool that is based on my LV “storage”. I think this was the command I used:

#zpool create -o mountpoint=/home/username storage /dev/mapper/laptop--vg-storage

# zpool status
  pool: storage
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 0 days 00:10:39 with 0 errors on Sun Jan 12 00:34:40 2020
config:

	NAME                  STATE     READ WRITE CKSUM
	storage               ONLINE       0     0     0
	  laptop--vg-storage  ONLINE       0     0     0

errors: No known data errors
# 

And that would be mounted where I requested

$ df -hT | grep zfs
storage        zfs       176G   73G  103G  42% /home/username/storage

This is too basic, in most cases your will want to have a kinf of RAID. But again, this is a simple laptop. As well, you can configure snapshots (useful if you want to have rollback a server upgrade that involves a huge amount of data) and other performance parameters (as per document below):

https://www.percona.com/live/17/sites/default/files/slides/pl17_ZFS_MySQL_Salesforce_0.pdf

So once you have your ZFS configured and mounted you can work with it as usual.

So back to the ZIL and ARC. Based on the links below:

• ZFS Intent Log, or ZIL, to buffer WRITE operations.
• ARC and L2ARC which are meant for READ operations.

In my laptop, I dont have any space left to play with this, so I can only check in my employer systems.

I have read the below article and I am going to give it a go in my laptop

https://www.simula.no/file/lj-219-jul-2012pdf/download

First, check the status of tcp thin:

# sysctl net.ipv4.tcp_thin_linear_timeouts
net.ipv4.tcp_thin_linear_timeouts = 0

I have realised that I dont have “/proc/sys/net/ipv4/tcp_thin_dupack” as the article mentions…

Ok, let update the value and be sure is still active after reboot.

Enable the value:

# echo "1" > /proc/sys/net/ipv4/tcp_thin_linear_timeouts
# sysctl net.ipv4.tcp_thin_linear_timeouts
net.ipv4.tcp_thin_linear_timeouts = 1

Make it permanent, edit /etc/sysctl.conf like this

# Based on https://www.simula.no/file/lj-219-jul-2012pdf/download
# enabling tcp thin-steam modifications for reducing latency in interactive apps
net.ipv4.tcp_thin_linear_timeouts = 1

Now it is time to test and see if you see any improvement or degradation!