Archive for the 'Unix & Linux' Category

Opensolaris 2009.06 live-CD

opensolarisI’ve received today the original Opensolaris Live-CD I ordered for free some days ago… thanks Sun!

You can order a free live CD from Sun: https://oscd.sun.com/ after registrering a free My Sun account, or download it from here.

Upgrading to Opensolaris 2009.06

Released few days ago, Opensolaris 2009.06 comes with a lot of new features (see Release notes and “What’s new“), like for example support for Sparc hardware (sun4u, sun4v), network virtualization with Crosbow, improvements in CIFS and ZFS time-slider snapshots (read this!),…

So last night I decided to upgrade my Opensolaris 2008.11 server to 2009.06. I have more than 7 years working in Unix administration, but I have to recognize that a full OS upgrade is always something a bit scary and risky. But honestly, upgrading Opensolaris is scarily easy — much like Debian or Ubuntu — and also safe!

  • Easy upgrade:

The upgrade process can be run with two command:

$ pfexec pkg install SUNWipkg
$ pfexec pkg image-update -v

Or more easily by running the IPS Package Manager graphical interface.

  • Safe upgrade

Yes, upgrading the full Opensolaris OS is safe, thanks to ZFS and Boot Environments. Whenever you upgrade a critical package or the full system: Opensolaris creates a new BE, that is a bootable ZFS clone (writable snapshot) of the actual running OS filesystem. The corresponding entry in GRUB is automatically added, of course!

The upgraded packages are actually installed on the recently created BE. Once finished the package installation, just reboot into the new BE and you get your new OS release runnung. Should something go wrong, just reboot into the previous BE.

BEs consum space based on the upgraded files only, so if you upgrade just a few packages, it won’t eat a lot of disk. Of course, upgrading the full OS is more space consuming:

NAME                        USED  AVAIL  REFER  MOUNTPOINT
rpool/ROOT                 7.24G  78.4G    18K  legacy
rpool/ROOT/opensolaris     8.31M  78.4G  2.60G  /
rpool/ROOT/opensolaris-1   7.60M  78.4G  3.19G  /
rpool/ROOT/opensolaris-2   35.3M  78.4G  4.21G  /
rpool/ROOT/opensolaris-3   7.19G  78.4G  4.59G  /

You can manage (mount, umount, activate, and destroy…) BEs with beadm(1M):

  • Unfortunately upgrading Opensolaris is a bit slow! Yes IPS is great, but at the moment it is really slow — compared to Debian’s APT for example. In my case I left the upgrade process all night and when waking up it was upgraded ;-):

Et voila:

$ cat /etc/release
                         OpenSolaris 2009.06 snv_111b X86
           Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                              Assembled 07 May 2009

ZFS Auto snapshots and Samba

You may have constated that snapshots generated by timf’s ZFS Auto Snapshots for Opensolaris are not Samba friendly.

If you look at the snapshots name, you’ll see they include a colon:

root# zfs list -t snapshot | grep mybook/share
NAME                                                                USED  AVAIL  REFER  MOUNTPOINT
mybook/share@zfs-auto-snap:daily-2009-05-12-00:00                    35K      -   322M  -
mybook/share@zfs-auto-snap:daily-2009-05-13-00:00                      0      -  6.29G  -
mybook/share@zfs-auto-snap:daily-2009-05-14-00:00                      0      -  6.29G  -
mybook/share@zfs-auto-snap:weekly-2009-05-15-00:00                     0      -  6.29G  -
mybook/share@zfs-auto-snap:daily-2009-05-15-00:00                      0      -  6.29G  -
mybook/share@zfs-auto-snap:daily-2009-05-16-00:00                    79K      -  6.29G  -
mybook/share@zfs-auto-snap:daily-2009-05-17-00:00                  7.18M      -  16.7G  -
mybook/share@zfs-auto-snap:daily-2009-05-18-00:00                      0      -  16.7G  -
mybook/share@zfs-auto-snap:daily-2009-05-19-00:00                      0      -  16.7G  -
mybook/share@zfs-auto-snap:daily-2009-05-20-00:00                      0      -  16.7G  -
...

When exported with Samba, Windows clients will see the snapshots like this:

zfs-snapshots-samba

It’s not even possible to open the snapshot directories from Windows.

How does timf’s ZFS Auto Snapshots work? They are launched by zfssnap user’s crontab:

root# crontab -l  zfssnap
0 0 1 1,2,3,4,5,6,7,8,9,10,11,12 * /lib/svc/method/zfs-auto-snapshot svc:/system/filesystem/zfs/auto-snapshot:monthly
0 0 1,8,15,22,29 * * /lib/svc/method/zfs-auto-snapshot svc:/system/filesystem/zfs/auto-snapshot:weekly
0 0 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31 * * /lib/svc/method/zfs-auto-snapshot svc:/system/filesystem/zfs/auto-snapshot:daily
0 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * * /lib/svc/method/zfs-auto-snapshot svc:/system/filesystem/zfs/auto-snapshot:hourly
0,15,30,45 * * * * /lib/svc/method/zfs-auto-snapshot svc:/system/filesystem/zfs/auto-snapshot:frequent

The script launched is actually the same as the service method: /lib/svc/method/zfs-auto-snapshot

If we look at it, we can see, among other options, that there is a SEP variable to change the “:” separator character (around line 65):

61
62 # A separator character for date strings, and to delimit
63 # snapshot label names. Needed because apparantly Samba
64 # clients can get confused by colons. Who knew?
65 SEP=":"
66

Change SEP to be for example “_”, and you’ll be able to see and open your snapshots from a Windows client!

Now what happends to all the snapshots created before we change the this? If we leave them with the colon, they won’t be destroyed when their time arrives…

So we need to rename them accordingly to the new separator. This can be done with the following scriptlet:

zfs list -t snapshot -H -o name |
   grep "zfs-auto-snap" | grep ":" |
while read NAME
do
   NEW="$( echo "${NAME}" | tr ":" "_" )"
   echo Renaming ${NAME} ${NEW}
   zfs rename ${NAME} ${NEW}
done

Now this is how the snapshots will appear from a Windows client: their names are now displayed correctly, and you can enter any of them to browse the files and directories at the date of the snapshot. Great!

zfs-snapshots-samba2

Project: Home file server powered by ZFS

WD Mirror Box

My Book® Mirror Edition™ package

As related in a previous post, my old home server was, amongst other things, acting as a home file server. This post is about how I setup my new Opensolaris eMachine home server as a file server. As requirements, I wanted my data to be safe from both hardware failure (drive redundancy) and human mistakes (backups).

I could have bought a commercial NAS (Network Attached Storage). The most commun NASes in the market are thoses ethernet hard drives. As far as I know, they are powered by some kind of embedded Linux, which cannot be that easily modified and feature extended (when possible it requires some obscure hacking…). Another cons are that multidrives models are quite expensive, and also – sorry Linux -, they don’t offer ZFS powerfull features!

To  add storage to my Opensolaris eMachine home server, I’ve bought — for my birthday — an external drive: a Western Digital “My Book® Mirror Edition 2TB“, that is:

  • a dual-drive Raid-1 (Mirror) external USB enclosure (also Raid-0 (Striped) configurable) with
  • two Western Digital Caviar 1TB Green Power hard drives: Cool, eco-friendly drives that consume approximately one-third less power than standard one,
  • User serviceable: enables me to open the enclosure and replace the drives inside (not hotplug though)
  • efficient convection cooling architecture (no fan: less noise and less power consumption),
  • intelligent drive management features, including automatic power-up, Safe Shutdown™ (what is that?) and LED status and activity lights (not sure wether this cool feature will work well with the Opensolaris/ZFS combo).
My Book® Mirror Edition™

My Book® Mirror Edition™

Upon plugging it into my eMachine, the new “disk” is recognized (using format in expert mode or rmformat):

root# format -e 
          /pci@0,0/pci10de,cb84@2,1/storage@a/disk@0,0
       1. c4d0
          /pci@0,0/pci-ide@8/ide@1/cmdk@0,0
       2. c8t0d0
          /pci@0,0/pci10de,cb84@2,1/device@8/storage@0/disk@0,0
Specify disk (enter its number):

root# rmformat -l
Looking for devices...
[...]
     3. Logical Node: /dev/rdsk/c8t0d0p0
        Physical Node: /pci@0,0/pci10de,cb84@2,1/device@8/storage@0/disk@0,0
        Connected Device: WD       My Book          1008
        Device Type: Removable
        Bus: USB
        Size: 953.9 GB
        Label:
        Access permissions: Medium is not write protected.

Let’s use the new disk c8t0d0 to create a new ZFS pool (zpool):

root# zpool create mybook c8t0d0
root# df
Filesystem           1K-blocks      Used Available Use% Mounted on
[...]
mybook               957874125        18 957874107   1% /mybook
Disk Replacement

Disk Replacement

That’s it: 1 TB of RAID1 protected disks! It couldn’t be more easy, could it?

This will protect my precious data against a drive failure. But what about human mistakes, which actually are much more likely to happend?

To recover from human mistakes, like unintentionally deleted and/or modified files, ZFS brings a killer feature: automatic snapshots at regular intervals, each with different retention times:

  • frequent snapshots every 15 mins, keeping 4 snapshots,
  • hourly snapshots every hour, keeping 24 snapshots,
  • daily snapshots every day, keeping 31 snapshots,
  • weekly snapshots every week, keeping 7 snapshots,
  • monthly snapshots every month, keeping 12 snapshots.

That is, providing you previously enables the feature on the ZFS in question, you can browse filesystem “back in time”! It’s amazing! (High end filer solutions like NetApp’s do this!). And it’s automatic! You just need to enable it, and that’s it!

This is the way to enable automatic snapshots on a ZFS. In my case I enable it on the first ZFS (the ZFS created upon zpool creation), so all the ZFS created under it will also inherit the property ;-).

root# zfs set com.sun:auto-snapshot=true mybook

I also configure the zfs to show the snapshot (they will appear under ./.zfs/snashot/ in any zfs filesystem). (The same apply here regarding inheritance).

root# zfs set snapdir=visible mybook
root# zfs get all mybook
NAME    PROPERTY               VALUE                  SOURCE
mybook  type                   filesystem             -
mybook  creation               Fri May  8 21:51 2009  -
mybook  used                   79.5K                  -
mybook  available              913G                   -
mybook  referenced             18K                    -
mybook  compressratio          1.00x                  -
mybook  mounted                yes                    -
mybook  quota                  none                   default
mybook  reservation            none                   default
mybook  recordsize             128K                   default
mybook  mountpoint             /mybook                default
mybook  sharenfs               off                    default
mybook  checksum               on                     default
mybook  compression            off                    default
mybook  atime                  on                     default
mybook  devices                on                     default
mybook  exec                   on                     default
mybook  setuid                 on                     default
mybook  readonly               off                    default
mybook  zoned                  off                    default
mybook  snapdir                visible                local
mybook  aclmode                groupmask              default
mybook  aclinherit             restricted             default
mybook  canmount               on                     default
mybook  shareiscsi             off                    default
mybook  xattr                  on                     default
mybook  copies                 1                      default
mybook  version                3                      -
mybook  utf8only               off                    -
mybook  normalization          none                   -
mybook  casesensitivity        sensitive              -
mybook  vscan                  off                    default
mybook  nbmand                 off                    default
mybook  sharesmb               off                    default
mybook  refquota               none                   default
mybook  refreservation         none                   default
mybook  primarycache           all                    default
mybook  secondarycache         all                    default
mybook  usedbysnapshots        0                      -
mybook  usedbydataset          18K                    -
mybook  usedbychildren         61.5K                  -
mybook  usedbyrefreservation   0                      -
mybook  com.sun:auto-snapshot  true                   local
root#

After some time (even with no file created on the new disk), I have:

root# find /mybook/
/mybook/
/mybook/.zfs
/mybook/.zfs/snapshot
/mybook/.zfs/snapshot/zfs-auto-snap:hourly-2009-05-08-22:00
/mybook/.zfs/snapshot/zfs-auto-snap:hourly-2009-05-08-22:00/share
/mybook/.zfs/snapshot/zfs-auto-snap:frequent-2009-05-08-22:00
/mybook/.zfs/snapshot/zfs-auto-snap:frequent-2009-05-08-22:00/share
/mybook/share
/mybook/share/.zfs
/mybook/share/.zfs/snapshot
/mybook/share/.zfs/snapshot/zfs-auto-snap:frequent-2009-05-08-22:00
/mybook/share/.zfs/snapshot/zfs-auto-snap:hourly-2009-05-08-22:00

Now I can share some of the new ZFS files using Samba to my other boxes, and my precious files are protected!

Nevertheless, there is some (annoying) drawbacks to this solution:

  • Portability: a ZFS formated USB drive is obviously less portable than a NTFS or FAT32 one, that you could plug on almost any PC. However, keeping a copy of the latest Opensolaris livecd with the drive, I should be able to re-import it and recover the files on another PC with not much trouble (I need to try the procedure!)
  • Energy saving: I found ZFS not very friend of smart energy saving drive features. At the beginning o my tests, ZFS would hang after some time (and the drive would be quite hot!). I suspect the drives power saving mode that probably tries to standby them. For the moment a script running every minute creating a random file on the drive is keeping it under some activity, and seems to do the trick (ZFS is’t hungging any more now). It’s not a very nice solution though.
  • Booting: the eMachine “server” won’t boot with the USB My Book® Mirror Edition™ plugged, so I manually have to unplug it, boot and plug it (before ZFS imports the zpools). It’s a bit of an hassle, but hopefully shouldn’t happend very often.
  • No support for drive status under Opensolaris (for the moment the drivers and tools are for Windows and MacOS), so should a drive fail I’ll have to notice it from the led status. I’m not very satisfied with that either, but I also hope it won’t happend!
  • Enclosure failure: should the enclosure fails, the only way to recover the data will be buying the same model (this is the problem with hardware Raid).

What’s next (some open ideas)?

  • implement Opensolaris CIFS rather than Samba; I’ll first have to understand Windows and CIFS ACLs…
  • find a way to get the status of both physical drives and the Raid status, from Opensolaris…

Información Hardware en Opensolaris

Leyendo el Blog O’Matty he descubierto este excelente comando para obtener informacion detallada del hardware de una maquina (menos criptico que los prtdiag/prtconf), con Opensolaris: smbios(1M) obtiene la informacion de la BIOS de la maquina.

Extraido del man:

DESCRIPTION
The smbios utility  displays  the  contents  of  the  System
Management  BIOS (SMBIOS) image exported by the current sys-
tem or stored in a  file.  SMBIOS  is  an  industry-standard
mechanism  for  low-level system software to export hardware
configuration information to higher-level system  management
software.  The  SMBIOS  data format itself is defined by the
Distributed  Management  Task   Force   (DMTF).   Refer   to
http://www.dmtf.org for more information about SMBIOS and to
obtain a copy of the SMBIOS specification and implementation
guidelines.

Por ejemplo, en mi nuevo servidor veo:

Informacion de la BIOS:

ID    SIZE TYPE
0     79   SMB_TYPE_BIOS (BIOS information)

Vendor: Phoenix Technologies, LTD
Version String: R01-A0L
Release Date: 09/17/2008
Address Segment: 0xe000
ROM Size: 1048576 bytes
Image Size: 131072 bytes

Informacion de la maquina:

ID    SIZE TYPE
1     77   SMB_TYPE_SYSTEM (system information)

Manufacturer: eMachines
Product: EL1200
Version: R01-A0
Serial Number: 98Y2D7ZEF0841077483001

ID    SIZE TYPE
4     94   SMB_TYPE_PROCESSOR (processor)

Manufacturer: AMD
Version: AMD Athlon(tm) Processor LE-1600
Serial Number:
Asset Tag:
Location Tag: Socket AM2
Part Number:

Family: 1 (other)
CPUID: 0x78bfbff00050ff3
Type: 3 (central processor)
Socket Upgrade: 17 (socket 940)
Socket Status: Populated
Processor Status: 1 (enabled)
Supported Voltages: 1.2V
External Clock Speed: Unknown
Maximum Speed: 2200MHz
Current Speed: 2211MHz
L1 Cache: 8
L2 Cache: 9
L3 Cache: None

Informacion de la memoria:

ID    SIZE TYPE
19    15   SMB_TYPE_MEMARRAY (physical memory array)

  Location: 3 (system board or motherboard)
  Use: 3 (system memory)
  ECC: 3 (none)
  Number of Slots/Sockets: 2
  Memory Error Data: Not Supported
  Max Capacity: 8589934592 bytes

ID    SIZE TYPE
20    86   SMB_TYPE_MEMDEVICE (memory device)

  Manufacturer: 7F4F000000000000
  Serial Number: 0007CA3F
  Asset Tag: None
  Location Tag: A0
  Part Number: JM800QLU-1G

  Physical Memory Array: 19
  Memory Error Data: Not Supported
  Total Width: 64 bits
  Data Width: 64 bits
  Size: 1073741824 bytes
  Form Factor: 9 (DIMM)
  Set: None
  Memory Type: 19 (DDR2)
  Flags: 0x0
  Speed: Unknown
  Device Locator: A0
  Bank Locator: Bank0/1

En Linux se puede sacar esta información con lshw.




Close
Powered by ShareThis