Browse Source

Add NFS CSI plugin Nomad configuration

master
Magic_RB 4 months ago
parent
commit
42a14fbefa
3 changed files with 122 additions and 0 deletions
  1. +2
    -0
      infrastructure/plugin-nfs/.gitignore
  2. +55
    -0
      infrastructure/plugin-nfs/nfs-controller.org
  3. +65
    -0
      infrastructure/plugin-nfs/nfs-nodes.org

+ 2
- 0
infrastructure/plugin-nfs/.gitignore View File

@ -0,0 +1,2 @@
*.hcl
*.html

+ 55
- 0
infrastructure/plugin-nfs/nfs-controller.org View File

@ -0,0 +1,55 @@
#+PROPERTY: tangle "controller.nomad"
TODO The image that's currently being used is the "official" =nfs-csi-driver= image, while it works
it isn't very declarative, I should test the Nix based one.
#+NAME: docker-image
#+BEGIN_SRC hcl
image = "mcr.microsoft.com/k8s/csi/nfs-csi:latest" # "csi-driver-nfs:local"
#+END_SRC
The arguments passed to the =nfs-csi-driver= binary are actually quite interesting. First we
specify the endpoint, the socket is placed in =/csi= and =/csi= is actually a bind mounted directory,
which belongs to nomad as specified in the =csi_plugin= stanza. Nomad then connects to the socket.
All the arguments are standard and seem to be the same for almost all CSI plugins, I don't think
cmd line arguments are covered by the CSI specification though..
#+NAME: docker-args
#+BEGIN_SRC hcl
args = [
"--endpoint=unix://csi/csi.sock",
"--nodeid=contoller",
"--logtostderr",
"-v=5",
]
#+END_SRC
#+BEGIN_SRC hcl :noweb yes :tangle yes
job "plugin-nfs-controller" {
datacenters = [ "homelab-1" ]
a
group "controller" {
task "plugin" {
driver = "docker"
config {
<<docker-image>>
<<docker-args>>
}
csi_plugin {
id = "nfs"
type = "controller"
mount_dir = "/csi"
}
resources {
cpu = 250
memory = 128
}
}
}
}
#+END_SRC

+ 65
- 0
infrastructure/plugin-nfs/nfs-nodes.org View File

@ -0,0 +1,65 @@
#+PROPERTY: tangle "nodes.nomad"
For comments please see [[./nfs-controller.org][nfs-controller.org]], the org files are essentially the same, save for two difference. One being the
node ID, in this case it's necessary for them to be unique, which may be achieved with =consul-template=.
#+OPTIONS: toc:nil
#+TITLE: csi-nfs-driver Node Configuration
#+NAME: docker-args
#+BEGIN_SRC hcl
args = [
"--endpoint=unix://csi/csi.sock",
"--nodeid=blowhole-0",
"--logtostderr",
"--v=5",
]
#+END_SRC
Another difference is that we must explicitly set, what IP address wil the nodes get, this is not necessary if you don't
run your NFS server on the same machine as on of these nodes may run. However if this is the case, you must specify what
IP they'll get (or at the one sharing space with the NFS server). The NFS server will see the nodes internal IP, because
NAT isn't performed until the packet from the node, leave the host. That means that it's impossible to statically
configure your NFS server to properly identify that node, unless you enable Kerberos and whatnot. Note that this
requires that a static docker network be create on each machine you add into the Nomad cluster.
#+NAME: docker-ip
#+BEGIN_SRC hcl
network_mode = "nfs"
ipv4_address = "172.69.0.10"
#+END_SRC
Also, privileged is required here, because the nodes must be able to perform mounts.
#+BEGIN_SRC hcl :tangle yes :noweb yes
job "plugin-nfs-nodes" {
datacenters = [ "homelab-1" ]
type = "system"
group "nodes" {
task "plugin" {
driver = "docker"
config {
image = "mcr.microsoft.com/k8s/csi/nfs-csi:latest" # "csi-driver-nfs:local"
<<docker-args>>
privileged = true
}
csi_plugin {
id = "nfs"
type = "node"
mount_dir = "/csi"
}
resources {
cpu = 250
memory = 128
}
}
}
}
#+END_SRC

Loading…
Cancel
Save