FreeBSD, FreeNAS

How to use GPT labels on FreeNAS – Part 1

I’ve recently been on a mission to make FreeNAS use GPT labels. By default, FreeNAS uses the GPT GUID format which makes it difficult to track down a drive in the event of a failure.

For example, the zpool status of a default FreeNAS setup will look something similar to:

[root@freenas] ~# zpool status tank
  pool: tank
 state: ONLINE
  scan: none requested
config:

	NAME                                            STATE     READ WRITE CKSUM
	tank                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/c16b5f1a-2abf-11e5-874b-000c29f1f845  ONLINE       0     0     0
	    gptid/c181b4bb-2abf-11e5-874b-000c29f1f845  ONLINE       0     0     0

errors: No known data errors

Unfortunately, this only tells you the GUID of the disk which is not available in the UI at all. Therefore, in the event of a failure, you’ll have to poke around to pinpoint which disk has failed.

Let’s fix that… Continue reading

Linux

Multiple DNS Search Suffixes in dhcpd.conf

So this a quick post because I spent a few hours resolving this issue and wanted to share. The DHCP option domain-search (code 119) is used to specify the DNS domain search list (i.e. “search example.com sales.example.com marketing.example.com” in /etc/resolv.conf) but it’s not clearly stated the format for dhcpd.conf. After trying many different combinations I finally nailed it down:

subnet 192.168.1.0 netmask 255.255.255.0 {
    range 192.168.1.101 192.168.1.199;
    option domain-name "example.com";
    option domain-search "example.com", "sales.example.com", "marketing.example.com";
    option domain-name-servers 192.168.1.1;
    option routers 192.168.1.1;
}

So the format is: option domain-search “SUFFIX 1”, “SUFFIX 2”, …, “SUFFIX N”;

Cheers,
–The IT Department

Linux

OpenWRT + OpenVPN + Routed LANs

So I’ve been using OpenVPN for a while now in a “point-to-point” (i.e. road-warrior) fashion. On the server I’m running OpenVPN on CentOS 5.3 using client certificate authentication + TLS-AUTH. I have 2 clients connecting, 1 client (me) connects from either a Windows host (OpenVPN GUI) or a Mac host (Tunnelblick). The other client connects from a Linux host (OpenVPN NetworkManager plugin).

Now whenever I connect, it’s from home – either I’m on my laptop or I’m on my mini. I have a Linksys WRT54GL running OpenWRT as my home router. Now I’ve configured the OpenWRT client in a bridged-fashion before but I wasn’t too fond of it since it doesn’t allow other clients to connect if they’re on a different network than the server without setting up another server instance.

So I set out today to figure out how to configure the routed vpn tunnel to allow the private networks on either side of the tunnel to be able to communicate with each other.

I found this post on the OpenWRT forums describing the same setup but it had no solution. I believe the problem was trying to manipulate the routing tables on the individual hosts directly when you actually need some extra routing configuration in OpenVPN. You have to use the client-config-dir option and the iroute option so OpenVPN can correctly route between the 2 LANs.

On the server we have the following setup:

eth0: 10.0.0.200/24
tun0: 192.168.255.1/24

[root@server openvpn]# cat /etc/openvpn/server.conf
port 1194
proto udp
dev tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key
dh /etc/openvpn/dh2048.pem
tls-auth /etc/openvpn/ta.key 0
server 192.168.255.0 255.255.255.0
client-config-dir /etc/openvpn/clients
route 192.168.1.0 255.255.255.0
keepalive 10 60
persist-key
persist-tun
comp-lzo

[root@server openvpn]# cat /etc/openvpn/clients/client1
iroute 192.168.1.0 255.255.255.0
push “route 10.0.0.0 255.255.255.0”
ifconfig-push 192.168.255.2 192.168.255.1

An explanation of the /etc/openvpn/clients/client1 config:

The ‘client1’ name comes from the common name of the certificate which the client is connecting with. For example, in my actual setup my common name is my full name so the client config file is /etc/openvpn/clients/FirstName_LastName

  • the ‘iroute’ option tells the server the 192.168.1.0/24 network should be routed through ‘client1’
  • the ‘push’ option tells client1 the 10.0.0.0/24 network is routed through the vpn tunnel
  • the ‘ifconfig-push’ option assigns client1 the vpn ip 192.168.255.2 routed through 192.168.255.1

On the client we have the following setup:

eth0: 192.168.1.1/24
tun0: 192.168.255.2/24

[root@client openvpn]# cat /etc/openvpn/server.conf
client
dev tun
proto udp
remote 1.2.3.4 1194
nobind
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client1.crt
key /etc/openvpn/client1.key
tls-auth /etc/openvpn/ta.key 1
persist-tun
persist-key
comp-lzo

After starting openvpn on the two hosts, there should be additional route entries similar to the following on the client:

[root@client openvpn]# route add -net 192.168.255.1 netmask 255.255.255.255 dev tun0
[root@client openvpn]# route add -net 10.0.0.0 netmask 255.255.255.0 gw 192.168.255.1 dev tun0

And on the server:

[root@server openvpn]# route add -net 192.168.255.2 netmask 255.255.255.255 dev tun0
[root@server openvpn]# route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.255.2 dev tun0

Now you may have noticed my OpenVPN server’s internal IP address is 10.0.0.200. It’s not the border router for the 10.0.0/24 network – 10.0.0.1 is (i.e. it’s the default gateway on the 10.0.0/24 network). It’s a FreeBSD router so simply adding a route entry for 192.168.1/24 via 10.0.0.200 allows the server-side LAN to fully communicate with client-side LAN:

[root@bsd-rtr ~]# route add -net 192.168.1/24 10.0.0.200

One last thing is to add a few firewall rules on each vpn end point to allow proper communication over the tunnel:

[root@client openvpn]# iptables -t nat -A POSTROUTING -s 192.168.255.2 -d 10.0.0.0/24 -j SNAT –to-source 192.168.1.1
[root@client openvpn]# iptables -A FORWARD -i eth0 -o tun0 -s 192.168.1.0/24 -d 10.0.0.0/24 -j ACCEPT
[root@client openvpn]# iptables -A FORWARD -i tun0 -o eth0 -s 10.0.0.0/24 -d 192.168.1.0/24 -j ACCEPT

[root@server openvpn]# iptables -t nat -A POSTROUTING -s 192.168.255.1 -d 192.168.1.0/24 -j SNAT –to-source 10.0.0.200
[root@server openvpn]# iptables -A FORWARD -i eth0 -o tun0 -s 10.0.0.0/24 -d 192.168.1.0/24 -j ACCEPT
[root@server openvpn]# iptables -A FORWARD -i tun0 -o eth0 -s 192.168.1.0/24 -d 10.0.0.0/24 -j ACCEPT

If you’ve reached this point and everything has made sense so far, you should be able to successfully ping any host on the 10.0.0.0/24 network from the 192.168.1.0/24 network and vice-versa. If not, leave a comment and I’ll try to help you out.

Cheers,
–The IT Department

Development

CentOS 5.2 + Qt 4.5 + FcFreeTypeQueryFace

So with all the buzz around the new release of Qt 4.5, you may be like me and have been desperately wanting to give it a whirl. Has this happened to you yet?

/opt/qtsdk-2009.01/qt/lib/libQtGui.so: undefined reference to `FcFreeTypeQueryFace'
collect2: ld returned 1 exit status

Someone’s got some ‘splainin to do. Lucky for you, that’s me. Basically, the people at fontconfig did not add this API call until 2.4.2. From fontconfig’s ChangeLog 2.4.2:

commit 72ffe6536a6825a32095c8185aff836a12326ac5
Author: Keith Packard 
Date:   Sat Dec 2 13:22:27 2006 -0800

    Add FcFreeTypeQueryFace external API. Bug #7311.
    
    Expose ability to build an FcPattern directly from an FT_Face
    object.

 configure.in            |    4 ++--
 doc/fcfreetype.fncs     |   17 +++++++++++++++-
 fontconfig/fcfreetype.h |    6 ++++++
 src/fcfreetype.c        |   50 ++++++++++++++++++++++++++---------------------
 4 files changed, 52 insertions(+), 25 deletions(-)

Problem is for us CentOS 5.2 users, we’re given a version 2.4.1-7 from “upstream” 😦 It’s not a big deal, we can just compile fontconfig from source … wait, did I just say compile? No worries, it’s easy.

1. Compile fontconfig 2.4.2 from source

Note: Throughout this step, I used fontconfig 2.4.2 since this is the minimum required version, but I tested this with the latest version 2.6.0 so just replace the version numbers accordingly if you choose to use 2.6.0.

[user@localhost ~]$ cd /usr/src
[user@localhost /usr/src]$ wget http://fontconfig.org/release/fontconfig-2.4.2.tar.gz
[user@localhost /usr/src]$ tar -zxf fontconfig-2.4.2.tar.gz
[user@localhost /usr/src]$ cd fontconfig-2.4.2
[user@localhost /usr/src/fontconfig-2.4.2]$ ./configure --prefix=/opt/fontconfig-2.4.2
... (note: 'configure' output omitted for brevity) ...
[user@localhost /usr/src/fontconfig-2.4.2]$ make
... (note: 'make' output omitted for brevity) ...
[user@localhost /usr/src/fontconfig-2.4.2]$ make install
... (note: 'make install' output omitted for brevity) ...

Note: I chose to install into /opt/fontconfig-2.4.2 since I like keeping my “deviations” from upstream organized.

2. Add the fontconfig library location to your project’s .pro file

LIBS += -L/opt/fontconfig-2.4.2/lib

3. Recompile and that’s it! 😎

Now I can’t guarantee you will not have any ‘make’ errors due to library dependencies but I’m pretty sure that if you got as far as you did to receive this error, you will have every *-devel package needed by fontconfig already installed on your system.

Zend Framework

Zend Framework 1.7 + Zend Studio 6.1 + PHPUnit

With the new release of Zend Framework 1.7 , like many others, I was eager to download it and give it a whirl. Now I’m currently developing a ZF web app and we’re still in the early stages so I upgraded our trunk version of ZF to 1.7 and re-ran all my PHPUnit test cases using Zend Studio. To no suprise there was a slight problem:

Debug Error: /foo/library/Zend/Test/PHPUnit/ControllerTestCase.php line 1099 – Call to undefined method FooControllerTest::incrementAssertionCounter()

Now I did manage to track down exactly what the issue is. I won’t go into details about how but I’ll just say this: Zend Debugger is a god send for debugging PHP applications (and in this case a PHP test suite).

Anywho, basically with the release of ZF 1.7, Zend_Test_PHPUnit_ControllerTestCase::_incrementAssertionCounter() was added and is called by every Zend_Test_PHPUnit_ControllerTestCase::assertBlah(…) function. It simply does a PHPUnit version check and loops through to PHPUnit and calls either PHPUnit_Framework_TestCase::incrementAssertionCounter() if your PHPUnit version < 3.3.0 or PHPUnit_Framework_TestCase::addToAssertionCounter(1) if your PHPUnit version >= 3.3.0.

The problem is this, PHPUnit did not add PHPUnit_Framework_TestCase::incrementAssertionCounter() until version 3.3.0. Hence if you have a version of PHPUnit < 3.3.0, ZF 1.7 is not compatible 😦 This is true for all Zend Studio 6.1 users as it ships with PHPUnit v3.2.18. Why did the ZF devs not add a check to make sure the version was above 3.3.0 is beyond me. This breaks ZF 1.7 backwards compatibility with ZF 1.6 as this “increment/add to assertion counter” was not even present in ZF 1.6.

Now here’s the temporary solution: Since you can’t change the PHPUnit location in Zend Studio, you have to patch your Zend Studio install with an updated PHPUnit library. At the time of this writing the latest was 3.3.4. Grab it from PEAR and do the following:

  1. Navigate to your ${ZEND_STUDIO_INSTALL_DIR}/plugins/com.zend.php.phpunit_6.1.0.v20080907/resources/library directory.
  2. Rename PHPUnit to PHPUnit-3.2.18
  3. Move the PHPUnit v3.3.4 folder you got from PEAR into the same directory and make sure it’s named PHPUnit

Now your PHPUnit test cases should happily run in Zend Studio 😀

I’m going to take this issue up with the ZF devs and hopefully they’ll be a bugfix in 1.7.1 but who knows. Just wanted to help anyone who may be experiencing the same problem.

Linux

Howto Resize a Xen DRBD LVM VBD

I decided to put together this howto because I spent practically an entire day at work trying to get this to work without needing a full resync after it was all said and done. So a quick run through of the setup and then off we go.

Note: This HOWTO describes shrinking a volume but will work exactly the same … no-resync necessary … for growing a DRBD volume backed by LVM 😀

I have 3 CentOS 5.1 domains (1x dom0 and 2x domU). The dom0 is running kernel 2.6.18-53.1.21.el5xen and Xen 3.0.3 and the 2 domUs are running kernel 2.6.18-53.1.19.el5xen and DRBD 8.2.5-1.el5. All of this came from the standard CentOS 5 yum repos. Each domU has two VBDs, namely /dev/xvda and /dev/xvdb. The OS is on /dev/xvda and the backing for DRBD is /dev/xvdb. Both of these are mapped to individual LVs on the dom0, namely /dev/vg0/node1 -> /dev/xvda and /dev/vg0/node1-drbd -> /dev/xvdb on node1 and /dev/vg0/node2 -> /dev/xvda and /dev/vg0/node2-drbd -> /dev/xvdb on node2. The filesystem on /dev/drbd0 is ext3 and is currently at 512MB for the duration of this howto. The current size of /dev/drbd0 is 3G and we want to shrink it to 2G.

Here’s my drbd.conf for reference:

resource res {
        protocol C;
        startup { wfc-timeout 0; degr-wfc-timeout 120; }
        disk { on-io-error detach; }
        net { }
        syncer { rate 100M; # we're using 1Gbps crossover link }
        on node1.domain.com {
                device /dev/drbd0;
                disk /dev/xvdb;
                address 10.0.0.1:7788;
                meta-disk internal;
        }
        on node2.domain.com {
                device /dev/drbd0;
                disk /dev/xvdb;
                address 10.0.0.2:7788;
                meta-disk internal;
        }
}

Whew … ready, steady go!

!!! READ ME — YOU ARE PROCEEDING BEYOND THIS POINT AT YOUR OWN RISK !!! THE IT DEPARTMENT CAN NOT AND WILL NOT BE HELD RESPONSIBLE / LIABLE / ACCOUNTABLE FOR THE RESULTS OF ANY OF YOUR ACTIONS. THIS IS PURELY AN EDUCATIONAL WRITING AND SHOULD NOT BE USED IN PRODUCTION — READ ME !!!

Now that I’ve scared the $h!t out of you … please let us continue.

0. Ensure your filesystem is smaller than the target size of the resource

In this example, our filesystem is 512MB which is already smaller than our target resource size of 2GB. Whenever you attempt this, simply resize your filesystem down smaller than your target (say -1GB or so than your target size) and when everything is said and done, you can simply grow the filesystem up to it’s maximum size possible.

1. Take down the resource on each node

[root@node1 ~]# drbdadm down res
[root@node2 ~]# drbdadm down res

2. Dump the resource metadata to disk on each node

[root@node1 ~]# drbdadm dump-md res > /tmp/metadata
[root@node2 ~]# drbdadm dump-md res > /tmp/metadata

Do not dump the meta data on one node, and simply copy the dump file to the peer. This will not work. You must dump the resource metadata on each node!

3. Detach the VBDs from the domUs

To actively resize the LVM without restarting the domU you have to detach the VBD from the domU / resize the LV in the dom0 / reattach the VBD to the domU. To detach a VBD, you need it’s dev-id which can be found using xm block-list

[root@dom0 ~]# xm block-list node1
Vdev  BE handle state evt-ch ring-ref BE-path
51713    0    0     4      6      8     /local/domain/0/backend/vbd/57/51713
51728    0    0     4      8      1377  /local/domain/0/backend/vbd/57/51728
[root@dom0 ~]# cat /etc/xen/node1 | grep disk
disk = [ "phy:/dev/vg0/node1,xvda,w", "phy:/dev/vg0/node1-drbd,xvdb,w" ]

From what I can tell, this is linear to your domU configuration. The first device in the xm block-list output corresponds to the first device in your configuration, the second corresponds to the second device, and so forth. Therefore, in my configuration, my domU node1’s VBD /dev/xvda has dev-id 51713 and it’s VBD /dev/xvdb has dev-id 51728.

Same thing goes for domU node2:

[root@dom0 ~]# xm block-list node2
Vdev  BE handle state evt-ch ring-ref BE-path
51713    0    0     4      6      8     /local/domain/0/backend/vbd/57/51713
51728    0    0     4      8      1377  /local/domain/0/backend/vbd/57/51728
[root@dom0 ~]# cat /etc/xen/node2 | grep disk
disk = [ "phy:/dev/vg0/node2,xvda,w", "phy:/dev/vg0/node2-drbd,xvdb,w" ]

Now you can detach…

[root@dom0 ~]# xm block-detach node1 51728
[root@dom0 ~]# xm block-detach node2 51728

4. Resize the LVs which are the backing for the VBDs

In my case, I have /dev/vg0/node1-drbd and /dev/vg0/node2-drbd. In this example, I am shrinking my DRBD device down to 2GB:

[root@dom0 ~]# lvresize -L2G /dev/vg0/node1-drbd
  WARNING: Reducing active logical volume to 2.00 GB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce node1-drbd? [y/n]: y
  Reducing logical volume node1-drbd to 2.00 GB
  Logical volume node1-drbd successfully resized
[root@dom0 ~]# lvresize -L2G /dev/vg0/node2-drbd
  WARNING: Reducing active logical volume to 2.00 GB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce node2-drbd? [y/n]: y
  Reducing logical volume node2-drbd to 2.00 GB
  Logical volume node2-drbd successfully resized

Enter yes when it asks you if “you really want to reduce” though you “MAY DESTROY YOUR DATA” … the force “The IT Department” is with you

5. Attach the VBDs back to the domUs

This step is simple enough:

[root@dom0 ~]# xm block-attach node1 phy:/dev/vg0/node1-drbd xvdb w
[root@dom0 ~]# xm block-attach node2 phy:/dev/vg0/node2-drbd xvdb w

6. Restore the resource metadata

This is probably the most complicated step. The jist of it is you have to create new metadata, then overwrite it with the old metadata you dumped earlier with a slight modification. You’ll see …

First create new metadata on each node

[root@node1 ~]# drbdadm create-md res
md_offset 2147479552
al_offset 2147446784
bm_offset 2147381248

Found ext3 filesystem which uses 524288 kB
current configuration leaves usable 2097052 kB

 ==> This might destroy existing data! <==
 
Do you want to proceed?
[need to type 'yes' to confirm] yes

[root@node2 ~]# drbdadm create-md res
md_offset 2147479552
al_offset 2147446784
bm_offset 2147381248

Found ext3 filesystem which uses 524288 kB
current configuration leaves usable 2097052 kB

 ==> This might destroy existing data! <==
 
Do you want to proceed?
[need to type 'yes' to confirm] yes

If you get a warning about there already being a v08 style flexible-size internal meta data block …

v07 Magic number not found
v07 Magic number not found
You want me to create a v08 style flexible-size internal meta data block.
There apears to be a v08 flexible-size internal meta data block
already in place on /dev/xvdb at byte offset 2147479552
Do you really want to overwrite the existing v08 meta-data?
[need to type 'yes' to confirm] yes

Go ahead and type ‘yes’ to overwrite it … again “The IT Department” is with you

Now do you see the line where DRBD says “current configuration leaves usuable 2097052 kB” … well we need that number. That number is the answer to life the amount of data the device can use minus the size of the internal metadata DRBD needs. The only problem is it’s in kB and we need it in sectors. You’re right … simple conversion: x KB / 1024 = y MB … y MB x 2048 = z Sectors

Therefore: 2097052 KB / 1024 = 2047.90234375 MB * 2048 = 4194104 Sectors. Now before we can restore the original metadata dump, we have to update it with the new size. Then we can restore it:

[root@node1 ~]# sed -i -e 's/la-size-sect.*/la-size-sect 4194104;/g' /tmp/metadata
[root@node1 ~]# drbdmeta_cmd=$(drbdadm -d dump-md res)
[root@node1 ~]# ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata

Valid meta-data in place, overwrite?
[need to type 'yes' to confirm] yes

reinitialising
Successfully restored meta data
[root@node2 ~]# sed -i -e 's/la-size-sect.*/la-size-sect 4194104;/g' /tmp/metadata
[root@node2 ~]# drbdmeta_cmd=$(drbdadm -d dump-md res)
[root@node2 ~]# ${drbdmeta_cmd/dump-md/restore-md} /tmp/metadata

Valid meta-data in place, overwrite?
[need to type 'yes' to confirm] yes

reinitialising
Successfully restored meta data

7. Bring up the resource on each node

If all goes well … you should simply be able to issue a drbdadm up on each node

[root@node1 ~]# drbdadm up res
[root@node2 ~]# drbdadm up res

And then verify everything is connected and UpToDate…

[root@node1 ~]# service drbd status
version: 8.2.5 ( api:88/proto:86-88 )
GIT-hash: 9faf052fdae5ef0c61b4d03890e2d2eab550610c build by buildsvn@c5-i386-build, 2008-05-11 03:43:50
m:res  cs         st                   ds                 p  mounted  fstype
0:res  Connected  Secondary/Secondary  UpToDate/UpToDate  C
[root@node2 ~]$ service drbd status
version: 8.2.5 ( api:88/proto:86-88 )
GIT-hash: 9faf052fdae5ef0c61b4d03890e2d2eab550610c build by buildsvn@c5-i386-build, 2008-05-11 03:43:50
m:res  cs         st                   ds                 p  mounted  fstype
0:res  Connected  Secondary/Secondary  UpToDate/UpToDate  C

Everything is a-ok and no need for a full-resync!!!

8. Conclusion

This was a simple example with a filesystem of 512MB and we only shrank the resource from 3GB to 2GB but this will work for larger configurations (I personally did it from 50GB down to 25GB earlier today). Maybe one day this won’t be as complicated but for now this works and I have no complaints.