Tech Overview

Start With Just One Single, Multi-Drive Server. Scale to Thousands of TBs!

Huge.Global serves four primary tasks for Ubuntu cluster programmers:

  1. Replicate Your Databases: Replicate database records across all servers so that the hub and satellite servers remain perfect replicates of each other. This allows for users to perform CRUD operation on any of the cluster’s machines
  2. Replicate Your Files: Copy file duplicates across multiple servers and drives to ensure a minimal level of replication and fault tolerance → This allows for uploads and lookup queries to any machine
  3. Automate Your Cluster Updates:Automatically update any application script files or other resource files based on last modified from a central update folder on the Hub machine
  4. Monitor Your Hardware: Orchestrate and monitor the entire server hardware stack: hard drives, RAM, CPU, Network, Disks, Web Servers


Distribution Methodology

Huge.Global uses a hub and spoke(or "star network") synchronization methodology called Flo and can work in all of the following scenarios:

  • LAN: All servers are on a single LAN and communicate over the local network. (Fastest and most responsive replication)
  • ExposedTeam: All servers have Internet accessible endpoints and communicate over the internet (Geo- Distributed)
  • ExposedPresident: At least the Hub’s sync endpoint is accessible over the Internet- all Satellites will “phone home” (instigate syncing and push and pull data from the hub)

A cluster is defined as having multiple servers each having one or more drives. One server is designated the King Server and the others are referred to as Pawn Servers. The king server is always visible to each of the pawn servers.

Server Roles: King and Pawns

The King Server is the center hub of the system that all of the pawn servers sync with. There are a vast number of configuration options to decide exactly when imports and exports occur between a pawn and the king.

Cluster Modes

Huge.Global has two cluster modes: Single Local Area Network or Multiple Networks. To further optimize connectivity in cases where the cluster is on the same Local Area Network, Huge.Global allows the king to initiate syncs with visible pawn servers. Visible pawn servers are those marked On King’s LAN or have an external http endpoint configured.

Hardware Management

Easily monitor the health of your Linux servers, their storage drives, RAM, CPU, Network, and temperatures.

Drives can easily be added, changed to read-only(for file replication), removed or marked as corrupt or irreparable (recalculates replication rates and replicates more when needed and possible)

Drives can be marked as available, full, and you can set a free space threshold past which the drive will automatically be classified as full if storage exists elsewhere in the system

The Flo Database: Important Tables:

  • Servers – the cluster’s team members, their roles(king or pawn), availability status, visibility by and to king,
  • Drives – Which drives are part of the system, which servers they are connected to, availability status, remaining space, partition name, mounted on
  • Collections – Which collections are being synced, the corresponding PostgreSQL database/ data table information (host, port, database, datatable)
  • Speedtests – What is the latency and bandwidth(upload and download) between your servers and speed test servers located around the world
  • Files – Which files are in the HugeGlobal repository, their access id, target replication, filename and path
  • FileManifest – Which files are on which drives, their statuses
  • SyncLog – When the server last synced with the other server(s) and which records were successfully moved (and in which direction)

Connect Any Number of Servers & Disks

The more the merrier!

Distribute and Amplify

Perform CRUD operations on any of the servers. Upload files to any server. File storing and fetching is distributed across servers and disks.

For a More Connected World

Websites of the future will need to offer more and more computing resources to the user in order to compete. The sun sets on the days of running a top million ranked site off a single drive.

DataBase Synchronization

Specify the databases you wish to replicate on each of the cluster's servers. Each Replicated Database has a single server specified as the king server and the others are pawn servers. CRUD operations can be made on any server(just update the timestamp and _srvr fields in conjunction). Make any change to the structure of the database on the King Server and it will permeate to each of the pawns.

A server can be added to a cluster by running the flo.php command --JoinCluster with the new cluster King's endpoint and access code specified

Amplify With Confidence

Huge.Global

Happy Clients

Join the growing list of developers using Huge.Global!

  • BigDrive.io securely and redundantly stores all of its users beloved files with ease using the Huge.Global framework.

    BigDrive.io

File Replication

Files should be stored in multiple disks(on separate servers, preferably) to prevent any data loss if hardware failure should occur.