At the beginning of this nice month, I'll try to bring in some light in the storage matter as many readers asked me to do so.
This blog entry will deal with block based storage-sharing and file-based storage sharing.
FiberChannel-FCP ("SAN") and iSCSI are representants of the block share paradigm, CIFS/SMB ("Windows Sharing") and NFS are representants of the file-based sharing method.
The graphic on the right shows a simplified communication layer diagram of FiberChannel- and iSCSI storage compared to a local hard disk.
Block-based sharing of storage
Let's begin from the top. At the top there's the application or just you: You want to save files, create directories, delete files, copy them, read them, define access restrictions.
These requests get into the filesystem layer. There are many of them. On Solaris ufs and zfs are used widely. In Windows, NTFS is normally used. Linux offers many filesystems like ext2, ext3, ext4, xfs and jfs - and each of them has pros and cons.
The filesystem layers translates the application and user requests into requests that a "dumber" device can understand: In reads and writes of simple blocks (a fixed length of data). A block on a hard disk can be a sector of a track of a platter.
In Solaris these requests gets filed to the "sd" device driver which means "SCSI disk". This driver routes the block requests (via a hardware driver of the installed SCSI/SATA/SAS-chip) to the disk which executes the request.
Instead of a real disk you can attach FiberChannel or iSCSI devices. The relevant layers are printed in red on the graphic. In case of a network disk the "ssd" driver comes in action (in Solaris SPARC versions, in x86 Solaris it is the same "sd" driver. This is a historic behaviour).
The information which blocks have to be read or written gets packed in FCP packets (in the case of a FiberChannel SAN) or in iSCSI packets. FCP is sent over the FiberChannel network and iSCSI via a normal TCP connection - just like you did when loading this web page.
As a result the filesystem layer does not know that the underlying "block device" is not really a local connected disk but a device somewhere in the network offering this basic block services.
Consequence? You may use whatever filesystem offered by your operating system when using these networked storage devices. A distant storage system with FiberChannel sharing can be mounted/formatted/used just as a normal hard disk in your system.
Another advantage: Applications or Servers using raw partitions for their data are perfectly usable in this scenario.
The downside is quite easy: As the network storage does not know which filesystem is used nor which blocks are really used or not it can not optimize its behaviour, it stays "dumb". Only blocks can be cached by the storage device, more complex caching must occur in the filesystem layer of the host using this network storage. Nowadays, nearly every serious file system has complex caching algorithm (imagine ARC and ZIL in ZFS) and the performance of these network disks are skyrocketing.
The second downside: Unless the used filesystem does not guarantee some kind of cluster functionality you cannot access the same storage disk for writing with two distinct hosts at the same time.
File-based sharing
This is the traditional form of sharing storage data. A special network file system (N F S) is used to translate the user/application file requests into a network language. SMB/CIFS and NFS are widely used nearly everywhere in the world. The translation from these requests to a "dumb" form which a disk can understand is made at the storage system, not at the host using nfs or cifs as filesystem. The graphic on the right tries to illustrate the host side.
In this scenario the storage system/server has more CPU cycles to burn to do disk accesses and translations. The storage system can heavily cache file accesses because it knows which FILES (or parts from them) are read or to be written.
The downside is simple: You are limited to nfs or cifs as a file system. And performance of these beasts are not always satisfying. If your application wants to use its own file system on a raw partition, this paradigma won't work.
Both of them - CIFS and NFS - had a quite difficult history with many nightmares. Neither of them were designed with performance and security in mind. If you keep your NFS/CIFS networks apart from other networks you'll get the same security focus like on a SAN or iSCSI LAN (also with separated networks).
Using ZFS, you may share storage block-based and file based.
Continue reading on the next blog entry...
Let's begin from the top. At the top there's the application or just you: You want to save files, create directories, delete files, copy them, read them, define access restrictions.
These requests get into the filesystem layer. There are many of them. On Solaris ufs and zfs are used widely. In Windows, NTFS is normally used. Linux offers many filesystems like ext2, ext3, ext4, xfs and jfs - and each of them has pros and cons.
The filesystem layers translates the application and user requests into requests that a "dumber" device can understand: In reads and writes of simple blocks (a fixed length of data). A block on a hard disk can be a sector of a track of a platter.
In Solaris these requests gets filed to the "sd" device driver which means "SCSI disk". This driver routes the block requests (via a hardware driver of the installed SCSI/SATA/SAS-chip) to the disk which executes the request.
Instead of a real disk you can attach FiberChannel or iSCSI devices. The relevant layers are printed in red on the graphic. In case of a network disk the "ssd" driver comes in action (in Solaris SPARC versions, in x86 Solaris it is the same "sd" driver. This is a historic behaviour).
The information which blocks have to be read or written gets packed in FCP packets (in the case of a FiberChannel SAN) or in iSCSI packets. FCP is sent over the FiberChannel network and iSCSI via a normal TCP connection - just like you did when loading this web page.
As a result the filesystem layer does not know that the underlying "block device" is not really a local connected disk but a device somewhere in the network offering this basic block services.
Consequence? You may use whatever filesystem offered by your operating system when using these networked storage devices. A distant storage system with FiberChannel sharing can be mounted/formatted/used just as a normal hard disk in your system.
Another advantage: Applications or Servers using raw partitions for their data are perfectly usable in this scenario.
The downside is quite easy: As the network storage does not know which filesystem is used nor which blocks are really used or not it can not optimize its behaviour, it stays "dumb". Only blocks can be cached by the storage device, more complex caching must occur in the filesystem layer of the host using this network storage. Nowadays, nearly every serious file system has complex caching algorithm (imagine ARC and ZIL in ZFS) and the performance of these network disks are skyrocketing.
The second downside: Unless the used filesystem does not guarantee some kind of cluster functionality you cannot access the same storage disk for writing with two distinct hosts at the same time.
File-based sharing
This is the traditional form of sharing storage data. A special network file system (N F S) is used to translate the user/application file requests into a network language. SMB/CIFS and NFS are widely used nearly everywhere in the world. The translation from these requests to a "dumb" form which a disk can understand is made at the storage system, not at the host using nfs or cifs as filesystem. The graphic on the right tries to illustrate the host side.In this scenario the storage system/server has more CPU cycles to burn to do disk accesses and translations. The storage system can heavily cache file accesses because it knows which FILES (or parts from them) are read or to be written.
The downside is simple: You are limited to nfs or cifs as a file system. And performance of these beasts are not always satisfying. If your application wants to use its own file system on a raw partition, this paradigma won't work.
Both of them - CIFS and NFS - had a quite difficult history with many nightmares. Neither of them were designed with performance and security in mind. If you keep your NFS/CIFS networks apart from other networks you'll get the same security focus like on a SAN or iSCSI LAN (also with separated networks).
Using ZFS, you may share storage block-based and file based.
Continue reading on the next blog entry...
Leave a comment