Linux Containers/pt-br

Status

Como no kernel Linux 3.1.5, o LXC é útil por isolar seus próprios trabalhos de outros. Ele não está pronto ainda para isolar potencialmente usuários de outros ou do sistema hóspede (host). Para uma solução de containers mais madura, que é apropriada para ambientes de hospedagem, veja OpenVZ.

Containers LXC ainda não possui seu próprio sistema de atualização, e eles veem tudo que está na saída dmesg no host, entre outras coisas. Mas em geral, a tecnologia funciona.

Informações Básicas

• Linux Containers são baseados em:
  • Kernel namespaces para isolamento de recursos
  • CGroups para limitação e contabilidade de recursos

app-emulation/lxc é a ferramenta userspace para os Linux containers

Grupos de controle (Control groups)

• Control groups (cgroups) no kernel desde o 2.6.24
  • Permite agregação de tarefas e seus filhos (children)
  • Subsystems (cpuset, memory, blkio,...)
  • accounting - para medir a quantidade de recursos que certos sistemas utilizam
  • resource limiting (limitante de recurso) - grupos podem ser configurados para não excederem um determinado limite de memória
  • prioritization (priorização) - alguns grupos podem ter uma fatia maior do CPU
  • control - freezing/unfreezing (congelamento/descongelamento) de cgroups, checkpointing (ponto de verificação) e restarting (reinicialização)
  • No disk quota limitation ( -> image file, LVM, XFS, directory tree quota,...) [sem limitação de cota de disco (-> arquivo imagem, LVM, XFS, cota de arvore de diretório)]

Subsystems

root # cat /proc/cgroups 
subsys_name	hierarchy	num_cgroups	enabled
cpuset	
cpu	
cpuacct	
memory	
devices	
freezer	
blkio	
perf_event
hugetlb

1. cpuset -> limita tarefas para CPU/CPUs específicos
2. cpu -> compartilhamento de CPU
3. cpuacct -> contabilização de CPU
4. memory -> limitação de contabilização de memória e de swap
5. devices -> lista de dispositivo negado ou permitido
6. freezer -> suspend/resume tarefas
7. blkio -> priorização I/O (weight, throttle, ...)
8. perf_event -> suporte para monitoramento por-cpu por-cgroup perf_events
9. hugetlb -> recurso controlador do cgroup para páginas HugeTLB hugetlb

Configurando o Host system do Funtoo

Instale o LXC kernel

Qualquer kernel acima do 3.1.5 provavelmente funcionará. Pessoalmente, eu prefiro o No results , uma vez que ele possui suporte para todos os namespaces sem sacrificar o suporte a xfs, FUSE ou NFS, por exemplo. Essas verificações foram introduzidas posteriormente, a partir do kernel 3.5, o que pode também significar que o namespace do usuário não está funcionando de forma otimizada.

• User namespace (EXPERIMENTAL) depende do EXPERIMENTAL e do UIDGID_CONVERTED
  • config UIDGID_CONVERTED
    • True (verdadeiro) se todos os componentes de software selecionados forem conhecidos por terem uid_t and gid_t convertidos para kuid_t and kgid_t onde se apropriaram e são por outro lado seguros de utilizar com o user namespace.
      • Networking - depende do NET_9P = n
      • Filesystems - 9P_FS = n, AFS_FS = n, AUTOFS4_FS = n, CEPH_FS = n, CIFS = n, CODA_FS = n, FUSE_FS = n, GFS2_FS = n, NCP_FS = n, NFSD = n, NFS_FS = n, OCFS2_FS = n, XFS_FS = n
      • Opções de segurança (Security options) - Grsecurity - GRKERNSEC = n (se aplicável)

• • A partir do kernel 3.10.xx, todas as opções acima são seguras para se usar com User namespaces, excetuando XFS_FS. Consequentemente, com kernel >=3.10.xx, você deverá responder XFS_FS= n, caso queira suporte a User namespaces.
  • Em seu diretório da fonte do kernel (kernel source directory), você deve verificar o init/Kconfig e descobrir o que UIDGID_CONVERTED depende

Configuração do kernel

Estas opções devem ser habilitadas em seu kernel para ser capaz de tirar o máximo proveito do LXC.

• General setup
  • CONFIG_NAMESPACES
    • CONFIG_UTS_NS
    • CONFIG_IPC_NS
    • CONFIG_PID_NS
    • CONFIG_NET_NS
    • CONFIG_USER_NS
  • CONFIG_CGROUPS
    • CONFIG_CGROUP_DEVICE
    • CONFIG_CGROUP_SCHED
    • CONFIG_CGROUP_CPUACCT
    • CONFIG_CGROUP_MEM_RES_CTLR (em kernels superiores ao 3.6 essa opção é chamada de CONFIG_MEMCG)
    • CONFIG_CGROUP_MEM_RES_CTLR_SWAP (em kernels superiores ao 3.6 essa opção é chamada CONFIG_MEMCG_SWAP)
    • CONFIG_CPUSETS (on multiprocessor hosts)
• Networking support
  • Networking options
    • CONFIG_VLAN_8021Q
• Device Drivers
  • Character devices
    • Unix98 PTY support
      • CONFIG_DEVPTS_MULTIPLE_INSTANCES
  • Network device support
    • Network core driver support
      • CONFIG_VETH
      • CONFIG_MACVLAN

Uma vez que você tenha o lxc instalado, você pode verificar seu kernel config com:

root # CONFIG=/path/to/config /usr/sbin/lxc-checkconfig

Emerge lxc

root # emerge app-emulation/lxc

Configure a Rede para o Container

Tipicamente, alguém utiliza uma ponte (bridge) para permitir que os containers conectem a rede. Esse é o modo de se fazer isso no Funtoo Linux:

1. crie uma bridge utilizando os Funtoo network configuration scripts. Nomeie a bridge com algo como brwan (using /etc/init.d/netif.brwan). Configure sua bridge pata ter um endereço IP.
2. Faça a dua interface física, tal qual eth0, uma interface sem endereço de IP (utilize o template interface-noip do Funtoo.)
3. Torne o netif.eth0 um slave de netif.brwan em /etc/conf.d/netif.brwan.
4. Habilite sua nova rede já em bridge e certifique-se de que está funcionando corretamente no host.

Agora você será capaz de configurar LXC para adicionar automaticamente sua interface ethernet virtual do container para criar uma bridge quando ele inicializar, que a conectará a sua rede.

Definindo um LXC Container do Funtoo Linux

Aqui estão os passos necessários para por o Funtoo Linux para funcionar dentro de um container. Os passos abaixo mostram como definir um container utilizando um template OpenVZ existente do Funtoo Linux. It is now also possible to use Metro to build an lxc container tarball directly, which will save you manual configuration steps and will provide an /etc/fstab.lxc file that you can use for your host container config. See Metro Recipes for info on how to use Metro to generate an lxc container.

Create and Configure Container Filesystem

1. Start with a Funtoo LXC template, and unpack it to a directory such as /lxc/funtoo0/rootfs/
2. Create an empty /lxc/funtoo0/fstab file
3. Ensure c1 line is uncommented (enabled) and c2 through c6 lines are disabled in /lxc/funtoo0/rootfs/etc/inittab

That's almost all you need to get the container filesystem ready to start.

Create Container Configuration Files

Create the following files:

/lxc/funtoo0/config

and also create symlink from

/lxc/funtoo0/config to /etc/lxc/funtoo0/config

root # install -d /etc/lxc/funtoo0
root # ln -s /lxc/funtoo0/config /etc/lxc/funtoo0/config

   Note

Daniel Robbins needs to update this config to be more in line with http://wiki.progress-linux.org/software/lxc/ -- this config appears to have nice, refined device node permissions and other goodies. // note by Havis to Daniel, this config is already superior.

Read "man 5 lxc.conf" , to get more information about linux container configuration file.

## Container
lxc.utsname                             = funtoo0
lxc.rootfs                              = /lxc/funtoo0/rootfs/
lxc.arch                                = x86_64
#lxc.console                            = /var/log/lxc/funtoo0.console  # uncomment if you want to log containers console
lxc.tty                                 = 6  # if you plan to use container with physical terminals (eg F1..F6)
#lxc.tty                                = 0  # set to 0 if you dont plan to use the container with physical terminal, also comment out in your containers /etc/inittab  c1 to c6 respawns (e.g. c1:12345:respawn:/sbin/agetty 38400 tty1 linux)
lxc.pts                                 = 1024


## Capabilities
lxc.cap.drop                            = audit_control
lxc.cap.drop                            = audit_write
lxc.cap.drop                            = mac_admin
lxc.cap.drop                            = mac_override
lxc.cap.drop                            = mknod
lxc.cap.drop                            = setfcap
lxc.cap.drop                            = setpcap
lxc.cap.drop                            = sys_admin
#lxc.cap.drop                            = sys_boot # capability to reboot the container
#lxc.cap.drop                            = sys_chroot # required by SSH
lxc.cap.drop                            = sys_module
#lxc.cap.drop                            = sys_nice
lxc.cap.drop                            = sys_pacct
lxc.cap.drop                            = sys_rawio
lxc.cap.drop                            = sys_resource
lxc.cap.drop                            = sys_time
#lxc.cap.drop                            = sys_tty_config # required by getty

## Devices
#lxc.cgroup.devices.allow               = a # Allow access to all devices
lxc.cgroup.devices.deny                 = a # Deny access to all devices

# Allow to mknod all devices (but not using them)
lxc.cgroup.devices.allow                = c *:* m
lxc.cgroup.devices.allow                = b *:* m

lxc.cgroup.devices.allow                = c 1:3 rwm # /dev/null
lxc.cgroup.devices.allow                = c 1:5 rwm # /dev/zero
lxc.cgroup.devices.allow                = c 1:7 rwm # /dev/full
lxc.cgroup.devices.allow                = c 1:8 rwm # /dev/random
lxc.cgroup.devices.allow                = c 1:9 rwm # /dev/urandom
#lxc.cgroup.devices.allow                = c 4:0 rwm # /dev/tty0 ttys not required if you have lxc.tty = 0
#lxc.cgroup.devices.allow                = c 4:1 rwm # /dev/tty1 devices with major number 4 are "real" tty devices
#lxc.cgroup.devices.allow                = c 4:2 rwm # /dev/tty2
#lxc.cgroup.devices.allow                = c 4:3 rwm # /dev/tty3
lxc.cgroup.devices.allow                = c 5:0 rwm # /dev/tty
lxc.cgroup.devices.allow                = c 5:1 rwm # /dev/console
lxc.cgroup.devices.allow                = c 5:2 rwm # /dev/ptmx
lxc.cgroup.devices.allow                = c 10:229 rwm # /dev/fuse
lxc.cgroup.devices.allow                = c 136:* rwm # /dev/pts/* devices with major number 136 are pts
lxc.cgroup.devices.allow                = c 254:0 rwm # /dev/rtc0

## Limits#
lxc.cgroup.cpu.shares                  = 1024
lxc.cgroup.cpuset.cpus                 = 0        # limits container to CPU0
lxc.cgroup.memory.limit_in_bytes       = 512M
lxc.cgroup.memory.memsw.limit_in_bytes = 1G
#lxc.cgroup.blkio.weight                = 500      # requires cfq block scheduler

## Filesystem
#containers fstab should be outside it's rootfs dir (e.g. /lxc/funtoo0/fstab is ok, but /lxc/funtoo0/rootfs/etc/fstab is wrong!!!)
#lxc.mount                               = /lxc/funtoo0/fstab       

#lxc.mount.entry is prefered, because it supports relative paths
lxc.mount.entry                         = proc proc proc nosuid,nodev,noexec  0 0
lxc.mount.entry                         = sysfs sys sysfs nosuid,nodev,noexec,ro 0 0
lxc.mount.entry                         = devpts dev/pts devpts nosuid,noexec,mode=0620,ptmxmode=000,newinstance 0 0
lxc.mount.entry                         = tmpfs dev/shm tmpfs nosuid,nodev,mode=1777 0 0
lxc.mount.entry                         = tmpfs run tmpfs nosuid,nodev,noexec,mode=0755,size=128m 0 0
lxc.mount.entry                         = tmpfs tmp tmpfs nosuid,nodev,noexec,mode=1777,size=1g 0 0

##Example of having /var/tmp/portage as tmpfs in container 
#lxc.mount.entry                         = tmpfs var/tmp/portage tmpfs defaults,size=8g,uid=250,gid=250,mode=0775 0 0
##Example of bind mount
#lxc.mount.entry                        = /srv/funtoo0 /lxc/funtoo0/rootfs/srv/funtoo0 none defaults,bind 0 0

## Network
lxc.network.type                        = veth
lxc.network.flags                       = up
lxc.network.hwaddr                      = #put your MAC address here, otherwise you will get a random one
lxc.network.link                        = br0
lxc.network.name                        = eth0
#lxc.network.veth.pair                   = veth-example

Read "man 7 capabilities" to get more information aboout Linux capabilities.

Above, use the following command to generate a random MAC for lxc.network.hwaddr:

root # openssl rand -hex 6

It is a very good idea to assign a static MAC address to your container using lxc.network.hwaddr. If you don't, LXC will auto-generate a new random MAC every time your container starts, which may confuse network equipment that expects MAC addresses to remain constant.

It might happen from case to case that you aren't able to start your LXC Container with the above generated MAC address so for all these who run into that problem here is a little script that connects your IP for the container with the MAC address. Just save the following code as /etc/lxc/hwaddr.sh, make it executable and run it like /etc/lxc/hwaddr.sh xxx.xxx.xxx.xxx where xxx.xxx.xxx.xxx represents your Container IP.
/etc/lxc/hwaddr.sh:

#!/bin/sh
IP=$*
HA=`printf "02:00:%x:%x:%x:%x" ${IP//./ }`
echo $HA

/lxc/funtoo0/fstab

   Note

It is now preferable to have mount entries directly in config file instead of separate fstab:

Edit the file /lxc/funtoo0/fstab:

none /lxc/funtoo0/dev/pts devpts defaults 0 0
none /lxc/funtoo0/proc proc defaults 0 0
none /lxc/funtoo0/sys sysfs defaults 0 0
none /lxc/funtoo0/dev/shm tmpfs nodev,nosuid,noexec,mode=1777,rw 0 0

LXC Networking

• veth - Virtual Ethernet (bridge)
• vlan - vlan interface (requires device able to do vlan tagging)
• macvlan (mac-address based virtual lan tagging) has 3 modes:
  • private
  • vepa (Virtual Ethernet Port Aggregator)
  • bridge
• phys - dedicated host NIC

Linux Containers and Networking

Enable routing on the host: By default Linux workstations and servers have IPv4 forwarding disabled.

root # echo "1" > /proc/sys/net/ipv4/ip_forward
root # cat /proc/sys/net/ipv4/ip_forward
root # 1

Initializing and Starting the Container

You will probably need to set the root password for the container before you can log in. You can use chroot to do this quickly:

root # chroot /lxc/funtoo0/rootfs
(chroot) # passwd
New password: XXXXXXXX
Retype new password: XXXXXXXX
passwd: password updated successfully
(chroot) # exit

Now that the root password is set, run:

root # lxc-start -n funtoo0 -d

The -d option will cause it to run in the background.

To attach to the console:

root # lxc-console -n funtoo0

You should now be able to log in and use the container. In addition, the container should now be accessible on the network.

To directly attach to container:

root # lxc-attach -n funtoo0

To stop the container:

root # lxc-stop -n funtoo0

Ensure that networking is working from within the container while it is running, and you're good to go!

Starting LXC container during host boot

1. You need to create symlink in /etc/init.d/ to /etc/init.d/lxc so that it reflects your container.
2. ln -s /etc/init.d/lxc /etc/init.d/lxc.funtoo0
3. now you can add lxc.funtoo0 to default runlevel
4. rc-update add lxc.funtoo0 default

root # rc
 * Starting funtoo0 ...                  [ ok ]

LXC Bugs/Missing Features

This section is devoted to documenting issues with the current implementation of LXC and its associated tools. We will be gradually expanding this section with detailed descriptions of problems, their status, and proposed solutions.

reboot

• By default, lxc does not support rebooting a container from within. It will simply stop and the host will not know to start it.
• If you want your container to reboot gracefully, you need sys_boot capability (comment out lxc.cap.drop = sys_boot in your container config)

PID namespaces

Process ID namespaces are functional, but the container can still see the CPU utilization of the host via the system load (ie. in top).

/dev/pts newinstance

• Some changes may be required to the host to properly implement "newinstance" /dev/pts. See This Red Hat bug.

lxc-create and lxc-destroy

• LXC's shell scripts are badly designed and are sure way to destruction, avoid using lxc-create and lxc-destroy.

network initialization and cleanup

• If used network.type = phys after lxc-stop the interface will be renamed to value from lxc.network.link. It supposed to be fixed in 0.7.4, happens still on 0.7.5 - http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg01760.html

• Re-starting a container can result in a failure as network resource are tied up from the already-defunct instance: [1]

graceful shutdown

• To gracefully shutdown a container, it's init system needs to properly handle kill -PWR signal
• For funtoo/gentoo make sure that you have:
  • pf:12345:powerwait:/sbin/halt
  • in your containers /etc/inittab
• For debian/ubuntu make sure that you have:
  • pf::powerwait:/sbin/shutdown -t1 -a -h now
  • in your container /etc/inittab
  • and also comment out other line starting with pf:powerfail (such as pf::powerwait:/etc/init.d/powerfail start) <- these are used if you have UPS monitoring daemon installed!
• /etc/init.d/lxc seems to have broken support for graceful shutdown (it sends proper signal, but then also tries to kill the init with lxc-stop)

funtoo

• Our udev should be updated to contain -lxc in scripts. (This has been done as of 02-Nov-2011, so should be resolved. But not fixed in our openvz templates, so need to regen them in a few days.)
• Our openrc should be patched to handle the case where it cannot mount tmpfs, and gracefully handle this situation somehow. (Work-around in our docs above, which is to mount tmpfs to /libexec/rc/init.d using the container-specific fstab file (on the host.)
• Emerging udev within a container can/will fail when realdev is run, if a device node cannot be created (such as /dev/console) if there are no mknod capabilities within the container. This should be fixed.

References

• man 7 capabilities
• man 5 lxc.conf

Links

• There are a number of additional lxc features that can be enabled via patches: [2]
• Ubuntu User Namespaces page
• lxc-gentoo setup script on GitHub

• IBM developerWorks
  • LXC: Linux Container Tools
  • Secure Linux Containers Cookbook

• Linux Weekly News
  • Smack for simplified access control