ICT platforms, HPC & Big data
The foundations of Calculus Energy's big data solutions are cemented on the design, development and the deployment of a bespoke, efficient and dynamically scalable big data cloud platform. It forms the grounding basis for providing truly innovative and integral solutions within the big data landcape. Fast integration and flexible data management are paramount for handling the vast amounts of data generated in the context of machine to machine communication, Internet of Things (IoT) and its state of the art digital connections such as blockchain. Our bespoke platforms answers criticial questions by employing bleeding edge, inovative technological solutions.
Big data platform based on Apache Spark
Our technology platform experitise implies that there are no limits in terms of quantity of data and type of data source. To integrate and aggregate huge amounts of data with different sizes, Calculus Energy utilizes a heavily customised inhouse application, which is essentially Spark in conjunction with the Hadoop Distributed File System (HDFS) - which forms the backbone for this big data platform. This robust framework is suitable for the collection, backup and processing of vast, structured and unstructured amounts of data. At the same time, the application is expandable up to any number of computing nodes. This makes the platform highly flexible and enables enormous storage capacity up to petabyte ranges.
When the data is well structured, we utilise PostgreSQL and have developed the expertise to enable fast read writes of around 1M/sec. To enable fast search of the data, we leverage an inhouse version of Apache Solr which delivers blazing speed for information retrival.
Cloud computing on large datasets
Cloud computing and the map-reduce paradigm have proven to be the key drivers for big data and its applications. A fast and efficient backup of vast data (in TBs/PBs) and the unlimited availability of data are only realisable by using scalable infrastructure resource such as AWS or Google Cloud Compute. Custom databases such as MySQL and Postgres quickly reach their limits when processing huge amounts of data – and infrastructure, combined with DevOps investment costs for an owned high performance big data platform are enormous.
To help clients overcome this bottleneck and improve cost efficiency and flexibility, Calculus Energy provides the state of the art big data technologies as a platform-as-a-service (PaaS) in a highly secure cloud environment. This means, companies can use a novel architecture and scale on demand without any financial risk.
