Synco Documentation

  1. Introduction
  2. Installation
  3. Concepts
  4. Backup Policy
  5. Storage Solutions
  6. Backup Scripts
  7. Basic Server Backup
  8. Backup Staging
  9. DNS Indirection
  10. Server Synchronization

Many small and agile companies rely on the availability of low cost equipment to support their storage and networking requirements. However, without considering the overall goals and requirements, this may result in insufficient throughput and poor reliability. I am going to discuss the structural elements in a small storage network which can cater for the majority of small/medium sized businesses.

Storage Requirements

There are a variety of different kinds of storage requirements, but businesses typically have two: computers which are used by staff members and centralised storage.

Workstation Backup

The majority of companies have both desktop and laptop computers used by staff. These devices contain many forms of important information which should be backed up on a regular basis.

  • User accounts, preferences and system configuration.
  • Locally stored company information and documents.
  • Personal files including photos, music, and other non-essential data.

Preferably, a workstation should be incrementally backed up once per hour.

Shared Storage

Many companies require a shared storage area. There are typically two kinds of businesses - those where data is the primary unit of work, and those where it is a side-effect. As an example, a video production house will produce large amounts of data, but an accounting firm will only produce small amounts of data. In both cases the data is critically important, but when data is the primary output, special consideration needs to be paid to the capacity of primary and backup storage devices, and the speed of the network between the storage device and the host computers.

Some businesses have both kinds of data, and ideally in these cases having separate storage servers for each type of data will provide the best performance and availability.

Small/Medium Storage Network Solution

The following diagram outlines a basic storage network. Depending on the devices and connectivity chosen, the performance can be improved.

There are several issues to consider when putting together a storage network.

Total Throughput

The total throughput between the storage infrastructure and the network will be divided between the number of machines in the workspace. Depending on the usage pattern, the effects can be severe.

Alice's storage server has a throughput of 60-80MB/s (typical for a single 1Gb/s ethernet). She also has 6 machines on a switched network. Thus, each machine has approximately 10MB/s of transmission to the storage server. If all 6 machines are in use and accessing data on the shared storage, the throughput degrades significantly.

Having separate storage devices allocated for different tasks can improve overall throughput if the services are accessed concurrently by multiple users.

Alice sets up a backup server on a separate physical ethernet port. Now each machine will have approximately 10MB/s of throughput to the shared storage (if accessing simultaneously) and 10MB/s of throughput to the backup storage. She sets up incremental backups and notices that the overall network performance reduces slightly, but that the shared storage is still as fast as before.

Upgrading hardware and connectivity can yield a net gain in terms of throughput; however care should be taken to find and improve the slowest part of the storage network first — improvements in other areas may not provide a significant benefit. Finding the weakest link can be challenging, as it not only depends on the static characteristics of the hardware, but also the usage patterns.

Alice's coworkers are complaining that editing data on the shared storage is slow. Alice bonds two 1Gb/s ethernet connections to the storage server. Now each machine receives approximately 20MB/s of throughput to the shared storage (if accessing simultaneously). However, if 6 users are accessing different projects all at the same time, they only receive approximately 15MB/s due to limitations in the storage hardware.

Storage Utilisation

The utilisation of the shared storage will heavily affect the speed at which workstations can write new data to the shared storage. Hard drives physically become slower (in layman's terms) as more data is stored on them, and data fragmentation issues make it hard for the filesystem to allocate storage on the disk in contiguous blocks. As space is used beyond 80% capacity, hard drives will become progressively slower.

Bob buys a DroboPro and fills it with 6x 2TB drives giving approximately 5TB of usable space. He then proceeds to copy 4.2TB of photographs to the device, which takes several hours but is relatively quick. His coworker drops in a Drobo S with 500GB more photographs, but it takes significantly longer than expected to copy them to the DroboPro.

Many storage systems perform integrity checking and defragmentation in the background. If a large amount of data is added, modified and deleted, it is possible that the performance will be reduced due to fragmentation and replication issues.

Bob has been editing and deleting the photographs. Even though there is now only 3TB of photographs on the device, the performance still seems less than typical. After leaving the device running over night in standby mode, the next day performance is back to normal.

Large files can be efficiently stored and accessed in modern file systems, but many small files often reduces performance of both read and write. This is primarily because the disk head must physically move to many different locations to access each separate file.

Bob is done with the video data and deletes it all, and then starts copying hundreds of small company documents to the DroboPro. He notices it is considerably slower than copying the large photographic files.

Incremental Backup Provisioning

The capacity of the primary shared storage and the capacity of the workspace infrastructure will dictate the required size of the backup subsystem. Incremental backups are essential to a data retention policy, and thus as a rule of thumb, having about twice as much storage in the backup subsystem is recommended.

Bob has a network with 1TB of shared storage and 3 laptops each with 500GB drives. He has a total of 2.5TB of primary storage, and so he provisions 5TB of storage attached to the the backup server.

Advanced Requirements

Businesses often have a variety of different storage requirements. The basic storage network architecture can be improved in many different ways.

Multiple computers accessing the network, throughput a high priority.
Workstations that are constantly accessing the same data (e.g. a database) will benefit from additional cache memory in the server.
Workstations that are accessing large amounts of data in a project-centred fashion will benefit from having the data distributed across multiple storage servers.
Use high quality networking equipment and servers.
Multiple paths to shared storage can improve throughput.
Improve security of data.
Encrypted shared storage, backups and offsite storage.
Encrypted workstations and local user accounts.
Physically secure access and monitoring.
Improve the reliability of shared storage and backup.
Storage virtualisation and load balancing.
Real time incremental backups and offsite synchronisation.

Lucid Information Systems Ltd is able to provide storage and network consultation services. We are happy to assist with the design and deployment of storage networks.