FreeBSD Jails Containers

FreeBSD networking and containers (Jails) stacks are very mature and provide lots of useful features … yet for some reason these features are not properly advertised by the FreeBSD project … or not even documented at all. I remember when Solaris was still under Sun before ‘fatal’ 2008 Oracle acquisition and one of the advertised Solaris features was its networking capabilities – along with virtual switches etc. that were administrated with the ipadm(1M) and dladm(1M) commands. FreeBSD while having technologies like Netgraph or Jails lightweight containers – along with VNET Jails that have full independent of the host virtual network stack … almost does not advertise them at all. The VNET Jails – while being production ready and used by thousands of sysadmins – are still not documented in the FreeBSD Handbook or FreeBSD FAQ at all … you will not be able to find a single VNET mention in the FreeBSD Handbook. Even the FreeBSD Man Pages like jail.conf(5) does not mention it – only jail(8)partially mentions VNET feature.

There are however two FreeBSD Books dedicated to Jails … one free FreeBSD Jails Using VNETs (from 2020) and one non-free FreeBSD Mastery – Jails (from 2019).

The Table of Contents for this article is:

• FreeBSD Host Setup
• Classic Jails
• VNET Jails
• Thin Provisioning Jails
• Single Process Jails
• Removing Jails
• Summary

This guide aims to make VNET Jails networking little closer and simple. While one can use Netgraph bridge for this purpose – we will use the simpler and more obvious classic network bridge supported by if_bridge(4) driver on FreeBSD. I also encourage you to check the FreeBSD Handbook – Jails chapter.

FreeBSD Host Setup

The first thing we will do is to prepare the host networking. This is the typical static network configuration on FreeBSD with single IPv4 IP address without any VLANs in /etc/rc.conf file. We will also enable the Jails subsystem.

host # cat /etc/rc.conf
# NETWORK
  hostname="host"
  ifconfig_em0="inet 20.0.0.20/24"
  defaultrouter="20.0.0.1"

# JAILS
  jail_enable="YES"
  jail_parallel_start="YES"
  jail_list="classic vnet shadow"

With that setup – your classic Jails will be able to connect to the outside World.

It can be visualized more or less like that.

              +--------+    +-----------------+
              |GATEWAY |    |20.0.0.20    HOST|
(Internet)<==>|        |<==>|em0              |
              |20.0.0.1|    |                 |
              +--------+    +-----------------+

Below we will create a classic Jail for a start. Each such setup requires certain decisions to be made – one of them will be using /jail as our Jails root.

We will create one with ZFS datasets now.

host # zfs create -o mountpoint=/jail -p zroot/jail
host # zfs create                     -p zroot/jail/BASE
host # zfs create                     -p zroot/jail/classic

I have also create /jail/BASE and /jail/classic dirs. The first one will be used as a placeholder for various FreeBSD versions *-base.txz files. The latter will be used for our ‘classic’ FreeBSD Jail.

host # find /jail -maxdepth 1
/jail
/jail/BASE
/jail/classic

Classic Jails

I will be using FreeBSD 13.2-RELEASE for the host system so this is the Jail version I would use for ‘classic’ Jail.

host # fetch -o /jail/BASE/13.2-RELEASE-base.txz http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.2-RELEASE/base.txz

The FreeBSD host can run any other FreeBSD as long as its not newer then the host system version. That means that while we use FreeBSD 13.2-RELEASE you are able to run and consolidate a farm of any older FreeBSD version – along with legendary 4.11-RELEASE … or one of the most problematic 5.0-RELEASE that was the first one that introduced the SMP with its M:N model.

We will now create the ‘classic’ Jail.

host # tar -xf /jail/BASE/13.2-RELEASE-base.txz -C /jail/classic --unlink

We will now create our ‘classic’ Jail configuration. Usually the jail.conf(5) file is used for that … but as you grow more Jails it becomes less practical to scroll through this file to ‘find’ your desired Jail that you want to modify. This is where the /etc/jail.conf.d dir comes handy. You will be able to place each Jail config as /etc/jail.conf.d/JAILNAME.conf file. This is what we will use here – leaving /etc/jail.conf file empty or non-existent.

This is the config we will use for the ‘classic’ Jail.

host # cat /etc/jail.conf.d/classic.conf
classic {
  # STARTUP/LOGGING
    exec.start = "/bin/sh /etc/rc";
    exec.stop  = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
    allow.raw_sockets;
    exec.clean;

  # PATH/HOSTNAME
    path = "/jail/${name}";
    host.hostname = "${name}";

  # NETWORK
    ip4.addr = 20.0.0.50;
    interface = em0;
}

Now we will start that Jail.

host # service jail start classic
Starting jails: classic.

host # jls
   JID  IP Address      Hostname                      Path
    31  20.0.0.50       classic                       /jail/classic

Our ‘classic’ Jail successfully started.

The 20.0.0.50 IP address was added to the host em0 interface as shown below.

host # ifconfig em0
em0: flags=8963 metric 0 mtu 1500
        options=481009b
        ether 08:00:27:09:cc:a8
        inet 20.0.0.20/24 broadcast 20.0.0.255
        inet 20.0.0.50/32 broadcast 20.0.0.50
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

It can be visualized like that.

              +--------+    +-----------------------------+
              |GATEWAY |    |20.0.0.20                HOST|
(Internet)<==>|        |<==>|em0          +-------------+ |
              |20.0.0.1|    |20.0.0.50<==>|____jail0____| |
              +--------+    |20.0.0.51<==>|____jail1____| |
                            |(.......)<==>|    (...)    | |
                            |             +-------------+ |
                            +-----------------------------+

We can also ping(8) the Jail IP address from the host system.

host # ping -c 3 20.50
PING 20.50 (20.0.0.50): 56 data bytes
64 bytes from 20.0.0.50: icmp_seq=0 ttl=64 time=0.114 ms
64 bytes from 20.0.0.50: icmp_seq=1 ttl=64 time=0.049 ms
64 bytes from 20.0.0.50: icmp_seq=2 ttl=64 time=0.046 ms

--- 20.50 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.046/0.069/0.114/0.031 ms

Lets login into the ‘classic’ Jail.

host # jls
   JID  IP Address      Hostname                      Path
     1  20.0.0.50       classic                       /jail/classic

host # jexec classic

root@classic:/ # hostname
classic

Inside the ‘classic’ Jail we are able to ping(8) the host gateway.

root@classic:/ # ping -c 3 20.1
PING 20.1 (20.0.0.1): 56 data bytes
64 bytes from 20.0.0.1: icmp_seq=0 ttl=255 time=0.083 ms
64 bytes from 20.0.0.1: icmp_seq=1 ttl=255 time=0.314 ms
64 bytes from 20.0.0.1: icmp_seq=2 ttl=255 time=0.256 ms

--- 20.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.083/0.218/0.314/0.098 ms

We can also reach outside World popular DNS public servers.

root@classic:/ # nc -v -u 1.1.1.1 53
Connection to 1.1.1.1 53 port [udp/domain] succeeded!
^C

Lets configure one of those DNS servers for our ‘classic’ Jail and test it.

root@classic:/ # echo nameserver 1.1.1.1 > /etc/resolv.conf

root@classic:/ # drill freebsd.org | grep '^[^;]'
freebsd.org.    240     IN      A       96.47.72.84

Seems to work properly. Lets enable sshd(8) service on it.

root@classic:/ # service sshd enable
sshd enabled in /etc/rc.conf

root@classic:/ # service sshd start
Generating RSA host key.
3072 SHA256:mSVNDUSi14S+GiaFJgNHNLCqQi6ndFG9JaSyA/wev1k root@classic (RSA)
Generating ECDSA host key.
256 SHA256:Hij315+3C/IMVJ1RX+hNJynGtVSU7ALYN0AS9/lxpJY root@classic (ECDSA)
Generating ED25519 host key.
256 SHA256:qzQnJCHjhHB7jQzmimSLayBfOc3dkLzIVhmrL2r9qxM root@classic (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.

root@classic:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sshd       15049 3  tcp4   20.0.0.50:22          *:*
root     sendmail   18689 3  tcp4   20.0.0.50:25          *:*
root     syslogd    84689 5  udp4   20.0.0.50:514         *:*

Seems to work properly. We will now try to login to it from other host on the 20.0.0.0/24 network.

laptop % ssh 20.50
The authenticity of host '20.0.0.50 (20.0.0.50)' can't be established.
ED25519 key fingerprint is SHA256:qzQnJCHjhHB7jQzmimSLayBfOc3dkLzIVhmrL2r9qxM.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '20.0.0.50' (ED25519) to the list of known hosts.
([email protected]) Password for root@classic:
([email protected]) Password for root@classic:
([email protected]) Password for root@classic:
root@20.0.0.50: Permission denied (publickey,keyboard-interactive).

Works. I was not able to login as by default login to root account is disabled.

VNET Jails

On the other hand the VNET Jails (with options VIMAGE in the kernel) are quite other beasts. While on the file/dir/config level they work the same – on the network part they are a lot different. They come with separate network stack and do not add their IP to the host network interface.

Also the current configuration of the host (as repeated below) would also not work for the VNET Jails network connectivity with the outside World.

host # cat /etc/rc.conf
# NETWORK
  hostname="host"
  ifconfig_em0="inet 20.0.0.20/24"
  defaultrouter="20.0.0.1"

To allow VNET Jails entry to the World outside of the host system we need to use – for example – if_bridge(4) interface – and move out host IP address there. Below is the not working for the VNET Jails current host network configuration.

host # ifconfig
em0: flags=8963 metric 0 mtu 1500
        options=481009b
        ether 08:00:27:09:cc:a8
        inet 20.0.0.20/24 broadcast 20.0.0.255
        inet 20.0.0.50/32 broadcast 20.0.0.50
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29
lo0: flags=8049 metric 0 mtu 16384
        options=680003
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1/8
        groups: lo
        nd6 options=21


host # netstat -Win -f inet
Name      Mtu Network            Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
em0         - 20.0.0.0/24        20.0.0.20           164406     -     -    91934     -     -
em0         - 20.0.0.50/32       20.0.0.50              207     -     -      168     -     -
lo0         - 127.0.0.0/8        127.0.0.1                6     -     -        6     -     -

host # route get 0
   route to: default
destination: default
       mask: default
    gateway: 20.0.0.1
        fib: 0
  interface: bridge0
      flags: 
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

This is how now the rc.conf(5) file needs to look like.

host # cat /etc/rc.conf

# NETWORK
  hostname="host"
  cloned_interfaces="bridge0"
  ifconfig_em0="up"
  ifconfig_bridge0="inet 20.0.0.20/24 up addm em0"
  defaultrouter="20.0.0.1"
  gateway_enable=YES

# JAILS
  jail_enable="YES"
  jail_parallel_start="YES"
  jail_list="classic"

… and this is how it looks in the ifconfig(8) and netstat(8) commands.

host # ifconfig
em0: flags=8963 metric 0 mtu 1500
        options=481009b
        ether 08:00:27:09:cc:a8
        inet 20.0.0.50/32 broadcast 20.0.0.50
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29
lo0: flags=8049 metric 0 mtu 16384
        options=680003
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1/8
        groups: lo
        nd6 options=21
bridge0: flags=8843 metric 0 mtu 1500
        ether 58:9c:fc:10:ff:b6
        inet 20.0.0.20/24 broadcast 20.0.0.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9

host # netstat -Win -f inet
Name      Mtu Network            Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
em0         - 20.0.0.50/32       20.0.0.50              207     -     -      168     -     -
lo0         - 127.0.0.0/8        127.0.0.1                6     -     -        6     -     -
bridge0     - 20.0.0.0/24        20.0.0.20           164576     -     -    92109     -     -

host # route get 0
   route to: default
destination: default
       mask: default
    gateway: 20.0.0.1
        fib: 0
  interface: bridge0
      flags: 
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

The ‘classic’ Jail IP address is still bound to the em0 interface while the host IP address has been moved to the bridge0 bridge. If needed you can also move all the ‘classic’ Jails IP addresses to the bridge0 interface but its not mandatory.

It can be visualized like that. Keep in mind that em0 and epair*a interfaces are members of bridge0 interface.

                            +--------------------------------------------+
                            |                        +-------------+ HOST|
                            |       em0 20.0.0.50<==>|____jail0____|     |
                            |      /    20.0.0.51<==>|____jail1____|     |
              +--------+    |     /     (.......)<==>|    (...)    |     |
              |GATEWAY |    |20.0.0.20               +-------------+     |
(Internet)<==>|        |<==>|bridge0                                     |
              |20.0.0.1|    |               +-----------------------+ |
              +--------+    |     epairXa<==>|epairXb 20.0.0.60 vnet0| |
                            |                +-----------------------+ |
                            |      epairYa<==>|epairYb 20.0.0.61 vnet1| |
                            |                 +-----------------------+ |
                            |       (....)a<==>|(....)b (.......) (...)| |
                            |                  +-----------------------+ |
                            +--------------------------------------------+

Lets create now our first VNET Jail. The beginning is the same – we will extract base.txz file from the 13.2-RELEASE system.

host # zfs create -p zroot/jail/vnet

host # tar -xf /jail/BASE/13.2-RELEASE-base.txz -C /jail/vnet --unlink

We will now also need the VNET Jail config.

host # cat /etc/jail.conf.d/vnet.conf
vnet {
  # STARTUP/LOGGING
    exec.start = "/bin/sh /etc/rc";
    exec.stop  = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
    allow.raw_sockets;
    exec.clean;

  # PATH/HOSTNAME
    path = "/jail/${name}";
    host.hostname = "${name}";

  # VNET/VIMAGE
    vnet;
    vnet.interface = "${if}b";

  # NETWORKS/INTERFACES
    $id = "60";
    $ip = "20.0.0.${id}/24";
    $gw = "20.0.0.1";
    $br = "bridge0";
    $if = "epair${id}";

  # ADD TO bridge0 INTERFACE
    exec.prestart += "ifconfig ${if} create up";
    exec.prestart += "ifconfig ${if}a up descr jail:${name}";
    exec.prestart += "ifconfig ${br} addm ${if}a up";
    exec.start    += "ifconfig ${if}b ${ip} up";
    exec.start    += "route add default ${gw}";
    exec.poststop += "ifconfig ${if}a destroy";
}

Lets now start the VNET Jail.

host # service jail start vnet
Starting jails: vnet.

host # jls
   JID  IP Address      Hostname                      Path
     1  20.0.0.50       classic                       /jail/classic
     2                  vnet                          /jail/vnet

One thing to notice here is that the jls(8) tool does not show the VNET Jails IP addresses.

You can of course overcome that limitation – but IMHO after 10+ years of VNET Jails being production ready its PITA to say the least.

host # jexec vnet ifconfig | grep 'inet '
        inet 127.0.0.1/8
        inet 20.0.0.60/24 broadcast 20.0.0.255

Lets now try our VNET Jail network connectivity to the outside World.

host # jexec vnet

root@vnet:/ # ping -c 3 20.1
PING 20.1 (20.0.0.1): 56 data bytes
64 bytes from 20.0.0.1: icmp_seq=0 ttl=255 time=0.853 ms
64 bytes from 20.0.0.1: icmp_seq=1 ttl=255 time=0.474 ms
64 bytes from 20.0.0.1: icmp_seq=2 ttl=255 time=1.355 ms

--- 20.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.474/0.894/1.355/0.361 ms

root@vnet:/ # nc -v -u 1.1.1.1 53
Connection to 1.1.1.1 53 port [udp/domain] succeeded!
^C

Works as desired.

This is how the networking looks like inside the VNET Jail.

root@vnet:/ # ifconfig
lo0: flags=8049 metric 0 mtu 16384
        options=680003
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1/8
        groups: lo
        nd6 options=21
epair60b: flags=8863 metric 0 mtu 1500
        options=8
        ether 02:47:e8:2e:6b:0b
        inet 20.0.0.60/24 broadcast 20.0.0.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T )
        status: active
        nd6 options=29

root@vnet:/ # netstat -Win -f inet
Name       Mtu Network            Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
lo0          - 127.0.0.0/8        127.0.0.1             2848     -     -     2848     -     -
epair60b     - 20.0.0.0/24        20.0.0.60                5     -     -        9     -     -

root@vnet:/ # route get 0
   route to: default
destination: default
       mask: default
    gateway: 20.0.0.1
        fib: 0
  interface: epair60b
      flags: 
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Same as with ‘classic’ Jail – we will enable the sshd(8) service.

root@vnet:/ # service sshd enable
sshd enabled in /etc/rc.conf

root@vnet:/ # service sshd start
Generating RSA host key.
3072 SHA256:n9sGBV1bmz3bT4+qKuFE5fZHjnBcYlOMxXCq98z7/r0 root@vnet (RSA)
Generating ECDSA host key.
256 SHA256:A+gDtzkkrhNnGRGR4Yf27cqME8/NZk5NCHrxwyEO9oM root@vnet (ECDSA)
Generating ED25519 host key.
256 SHA256:aojc9Kbyd32HkllG9+noKL8GvKMjObuLrUNiq24+OFk root@vnet (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.

root@vnet:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sshd       37145 4  tcp4   *:22                  *:*
root     sendmail   99914 4  tcp4   127.0.0.1:25          *:*
root     syslogd    71123 6  udp4   *:514                 *:*

We will try to connect to it from a system other then the host.

laptop % ssh 20.60
The authenticity of host '20.0.0.60 (20.0.0.60)' can't be established.
ED25519 key fingerprint is SHA256:aojc9Kbyd32HkllG9+noKL8GvKMjObuLrUNiq24+OFk.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '20.0.0.60' (ED25519) to the list of known hosts.
([email protected]) Password for root@vnet:
([email protected]) Password for root@vnet:
([email protected]) Password for root@vnet:
root@20.0.0.60: Permission denied (publickey,keyboard-interactive).

Same as with the ‘classic’ Jail – we will not login with root user as its disabled by default – but that is not the task of this exercise.

Thin Provisioning Jails

While this will not be possible on UFS filesystem – the ZFS allows that without any hassle.

We will now create a template for the ‘classic’ Jails on ZFS.

host # zfs snapshot zroot/jail/classic@template

The only things we did with the ‘classic’ Jail was to setup DNS server in /etc/resolv.conf file and we enabled sshd(8) service. We generally want that from any of our future Jails – so its a good candidate for a template Jail.

As we created the classic@template ZFS snapshot – we can not create unlimited thin provisioning ‘classic’ Jails from it. Just make sure to create needed /etc/jail.conf.d Jail config files.

Lets create a new thin provisioning Jail from it now.

host # zfs list -t snapshot
NAME                          USED  AVAIL     REFER  MOUNTPOINT
zroot/jail/classic@template   224K      -      503M  -

host # zfs clone zroot/jail/classic@template zroot/jail/shadow

host # zfs list -t all
NAME                          USED  AVAIL     REFER  MOUNTPOINT
zroot                        3.31G  15.6G       96K  none
zroot/ROOT                    700M  15.6G       96K  none
zroot/ROOT/default            699M  15.6G      699M  /
zroot/jail                   1.86G  15.6G      194M  /jail
zroot/jail/BASE               191M  15.6G      191M  /jail/BASE
zroot/jail/classic            503M  15.6G      503M  /jail/classic
zroot/jail/classic@template   224K      -      503M  -
zroot/jail/shadow               8K  15.6G      503M  /jail/shadow
zroot/jail/vnet               503M  15.6G      503M  /jail/vnet
zroot/tmp                      96K  15.6G       96K  /tmp
zroot/usr                     780M  15.6G       96K  /usr
zroot/usr/home                 96K  15.6G       96K  /usr/home
zroot/usr/ports                96K  15.6G       96K  /usr/ports
zroot/usr/src                 780M  15.6G      780M  /usr/src
zroot/var                     684K  15.6G       96K  /var
zroot/var/audit                96K  15.6G       96K  /var/audit
zroot/var/crash                96K  15.6G       96K  /var/crash
zroot/var/log                 204K  15.6G      204K  /var/log
zroot/var/mail                 96K  15.6G       96K  /var/mail
zroot/var/tmp                  96K  15.6G       96K  /var/tmp

host # cp /etc/jail.conf.d/classic.conf /etc/jail.conf.d/shadow.conf

host # sed -i '' -e 's|classic|shadow|g' -e 's|20.0.0.50|20.0.0.51|g' /etc/jail.conf.d/shadow.conf

host # cat /etc/jail.conf.d/shadow.conf
shadow {
  # STARTUP/LOGGING
    exec.start = "/bin/sh /etc/rc";
    exec.stop  = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";

  # PERMISSIONS
    allow.raw_sockets;
    exec.clean;

  # PATH/HOSTNAME
    path = "/jail/${name}";
    host.hostname = "${name}";

  # NETWORK
    ip4.addr = 20.0.0.51;
    interface = em0;
}

host # service jail start shadow
Starting jails: shadow.

host # jls
   JID  IP Address      Hostname                      Path
     1  20.0.0.50       classic                       /jail/classic
     2                  vnet                          /jail/vnet
     3  20.0.0.51       shadow                        /jail/shadow

host # netstat -Win -f inet
Name      Mtu Network            Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
em0         - 20.0.0.50/32       20.0.0.50             2854     -     -     2746     -     -
em0         - 20.0.0.51/32       20.0.0.51                3     -     -        0     -     -
lo0         - 127.0.0.0/8        127.0.0.1                6     -     -        6     -     -
bridge0     - 20.0.0.0/24        20.0.0.20           195544     -     -   150555     -     -

We can now test its sshd(8) daemon connection.

laptop % ssh 20.51
The authenticity of host '20.0.0.51 (20.0.0.51)' can't be established.
ED25519 key fingerprint is SHA256:qzQnJCHjhHB7jQzmimSLayBfOc3dkLzIVhmrL2r9qxM.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '20.0.0.51' (ED25519) to the list of known hosts.
([email protected]) Password for root@shadow:
([email protected]) Password for root@shadow:
([email protected]) Password for root@shadow:
root@20.0.0.51: Permission denied (publickey,keyboard-interactive).

Yep. Works as it should.

You can create a template for VNET Jail in the same manner.

Single Process Jails

The Docker is mostly know from its single process execution. You choose a process/daemon that you want to run isolated – copy it into the Docker container with its dependencies and you start it.

Exactly the same can be achieved with FreeBSD Jails.

For the purposes of this guide we will execute /bin/sh POSIX shell interpreter in a separated FreeBSD Jail. To omit copying additional files we will use the statically compiled version from the well know FreeBSD Rescue System.

host # mkdir -p /jail/shell/dev

host # cp /rescue/sh /rescue/hostname /jail/shell/

host # jail -n shell 
            -c path=/jail/shell 
               mount.devfs 
               host.hostname=shell 
               ip4.addr=20.0.0.111 
               command=/sh

shell # /hostname
shell

shell # /sh
Cannot read termcap database;
using dumb terminal settings.

shell # for I in 1 2 3; do echo ${I}; done
1
2
3

shell # echo /*
/dev /hostname /sh

You can login to the host system from other SSH session to see that shell Jail is running among our other FreeBSD Jails.

host # jls
   JID  IP Address      Hostname                      Path
     1  20.0.0.50       classic                       /jail/classic
     2                  vnet                          /jail/vnet
     3  20.0.0.51       shadow                        /jail/shadow
     4  20.0.0.111      shell                         /jail/shell

Removing Jails

For that purpose we will remove the ‘classic’ Jail.

We will first stop our Jail.

host # service jail stop classic

host # rm -rf /jail/classic
rm: /jail/classic/var/empty: Operation not permitted
rm: /jail/classic/var: Directory not empty
rm: /jail/classic/libexec/ld-elf32.so.1: Operation not permitted
rm: /jail/classic/libexec/ld-elf.so.1: Operation not permitted
rm: /jail/classic/libexec: Directory not empty
rm: /jail/classic/lib/libthr.so.3: Operation not permitted
rm: /jail/classic/lib/libcrypt.so.5: Operation not permitted
rm: /jail/classic/lib/libc.so.7: Operation not permitted
rm: /jail/classic/lib: Directory not empty
(...)

Wait … you are root and you can not delete files? 🙂 This is just another FreeBSD features. The File Flags. They can also be used with FreeBSD Secure Levels. By default some files and directories are marked with some of these flags. To be able to remove these files you first need to remove these flags. We can do that with chflags(1) command.

host # chflags -R 0 /jail/classic
host # rm -rf /jail/classic
host # echo ${?}
0

Now as the File Flags have been removed we can delete all files and dirs with usual rm(1) command. As our ‘classic’ Jail files are gone we should also delete the Jail config at /etc/jail.conf.d/classic.conf file.

host # rm -f /etc/jail.conf.d/classic.conf

Summary

I am not sure what else I should add here … but I am sure that you will let me know in the comments section 🙂

Regards.

EOF