{"id":1340,"date":"2023-06-04T22:38:35","date_gmt":"2023-06-04T21:38:35","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=1340"},"modified":"2023-06-04T22:38:35","modified_gmt":"2023-06-04T21:38:35","slug":"infiniband-essentials","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2023\/06\/04\/infiniband-essentials\/","title":{"rendered":"Infiniband Essentials"},"content":{"rendered":"\n<p>NVIDIA provides this <a href=\"https:\/\/academy.nvidia.com\/en\/training-search-wizard\/?mySearch=infiniband\">course<\/a> for free. Although I surprised that there is no much &#8220;free&#8221; documentation about this technology. I wish they follow the same path as most networking vendors where they want you to learn their technology without much barriers. And it is quite pathetic that you can&#8217;t really find books about it&#8230;<\/p>\n\n\n\n<p>The course is very very high level and very very short. So I didnt become an Infiniband CCIE&#8230;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Intro to IB<\/li>\n<\/ul>\n\n\n\n<p>&#8212; Elements of IB: IB switch, Subnet Manager (it is like a SDN controller), hosts (clients), adaptors (NICs), gateways (convert IB &lt;> Ethernet) and IB routers.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Key features<\/li>\n<\/ul>\n\n\n\n<p>&#8212; Simplify mgmt: because of the Subnet Manager<\/p>\n\n\n\n<p>&#8212; High bw: up to 400G)<\/p>\n\n\n\n<p>&#8212; Cpu offload: RDMA, bypass OS.<\/p>\n\n\n\n<p>&#8212; Ultra low latency: 1us host to host.<\/p>\n\n\n\n<p> &#8212; Network scale-out: 48k nodes in a single subnet. You can connect subnets using IB router.<\/p>\n\n\n\n<p>&#8212; QoS: achieve loss-less flows.<\/p>\n\n\n\n<p>&#8212; Fabric resilience: Fast-ReRouting at switch level takes 1ms compared with 5s using Traffic Manager => Self-Healing<\/p>\n\n\n\n<p>&#8212; Optimal load-balancing: using AR (adaptive routing). Rebalance packets and flows.<\/p>\n\n\n\n<p>&#8211;MPI super performance (SHARP &#8211; scalable hierarchical aggregation and reduction protocol): off-load operations from cpu\/gpu to switches -> decrease the retransmissions from end hosts -> less data sent. Dont really understand this.<\/p>\n\n\n\n<p>&#8212; Variety of supported topologies: fat-tree, dragonfly+, torus, hypercurve and hyperx.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Architecture:<\/li>\n<\/ul>\n\n\n\n<p>&#8212; Similar layers as OSI model: application, transport, network, link and physical.<\/p>\n\n\n\n<p>&#8212; In IB, applications connect to NIC, bypass OS.<\/p>\n\n\n\n<p>&#8212; Upper layer protocol:<\/p>\n\n\n\n<p>  &#8212; MPI: Message Passing Interface<\/p>\n\n\n\n<p>  &#8212; NCCL: NVIDIA Collective Communication Library<\/p>\n\n\n\n<p>  &#8212; iSEB: RDMA storage protocols.<\/p>\n\n\n\n<p>  &#8212; IPoIB: IP over IB<\/p>\n\n\n\n<p>&#8212; Transport Layer:  diff from tcp\/ip, it creates an end-to-end virtual channel between applications (source and destination), bypassing OS in both ends.<\/p>\n\n\n\n<p>&#8212; Network Layer: This is mainly at IB routers to connect IB subnets. Routers use GID as identifier for source and destinations.<\/p>\n\n\n\n<p>&#8212; Link Layer: each node is identified by a LID (local ID), managed by the Subnet Manager. Switch has a forwarding table with &#8220;port vs LID&#8221; &lt;- generated by Subnet Manager. You have flow-control for providing loss-less connections.<\/p>\n\n\n\n<p>&#8212; Physical Layer: Support for copper (DAC) and optical (AOC) connectors.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>NVIDIA provides this course for free. Although I surprised that there is no much &#8220;free&#8221; documentation about this technology. I wish they follow the same path as most networking vendors where they want you to learn their technology without much barriers. And it is quite pathetic that you can&#8217;t really find books about it&#8230; The &hellip; <a href=\"https:\/\/blog.thomarite.uk\/index.php\/2023\/06\/04\/infiniband-essentials\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Infiniband Essentials&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-1340","post","type-post","status-publish","format-standard","hentry","category-networks"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/1340","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=1340"}],"version-history":[{"count":1,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/1340\/revisions"}],"predecessor-version":[{"id":1341,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/1340\/revisions\/1341"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=1340"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=1340"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=1340"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}