Marcel Blokpoel, senior infrastructure engineer at Textkernel, talks about scaling up the US colocation for Textkernel’s products and his experiences in Dallas, USA.
by Marcel Blokpoel
Oftentimes, laws and customer preference request that sensitive and personal data should be kept in the same part of the world, in some cases even the same country, where the customer is located. To meet those demands, Textkernel cloud services are distributed across different data centres throughout the world. Having our services located near to our customers also makes them more responsive. Despite transatlantic connections consisting of very fast fibre-optic links these days, a few tenths of a second latency is still difficult to avoid. That’s enough to make a Web service noticeably less responsive.
Expansion in the US
When we joined forces with CareerBuilder in July last year, more and more US customers started using our products. Therefore, we decided for a US-based high-performance Internet infrastructure provider with a certified state-of-the-art facility in Dallas, Texas, to house our US platform featuring local deployments of most of our product portfolio. As a secondary (fall-back) data centre, we went for another US-based cloud computing service.
Our facility features hot and cold aisles (for optimised airflow and cooling of server hardware), biometric security, redundant power feeds, three 2MW diesel generators and much more. Any one component of this data centre’s infrastructure, and in some cases multiple components, can fail without any effect on service availability. That adds up to an advertised “five nines” – that’s 99.999% availability of the data centre itself, and a sizeable collection of data protection, security and environmental certifications.
On top of that, our own platform is built as a secure private cloud, fully virtualised and redundant for high availability. Redundancy at this level basically means there is two of everything, and failover (switching) and/or load balancing mechanisms are in place, minimising impact when something bad happens.
Doubling our capacity in the US
Just 6 months after the US platform was first built, as sales in the US market went well and US-based installations of the full range of our services were desired in both staging and production environments, we already felt the need to scale out. In order to more than double the capacity of the US platform, a number of additional servers were configured in-house and then shipped to Dallas for installation by local engineers.
However, during that process, rather than just adding hardware, we reviewed the underpinning network infrastructure and decided to take the opportunity to upgrade that as well. The new situation consists of two switched gigabit networks, to which all hardware is connected with dual network cards for redundancy. A pair of hardware routers were also added, to further improve performance and reliability of the connection to our cloud services.
Preparing to welcome this growing number of customers in the US market, the US platform doubled their CPU cores as well as their RAM and storage. Some of the new hardware is optimised for raw processing speed and will be dedicated to vacancy and CV parsing models. The rest is optimised for parallelisation and storage; for databases, Search! and more.
No downtime of any services
Eventually, due to the complexity of this change and the fact that we would no longer leave the existing part of the platform untouched, we preferred to have our own engineer on-site, so right after the retreat, I flew to Dallas to install the previously shipped hardware.
After much careful planning, preparation and collaboration with our US customers, a service window was planned during my stay in Dallas, during which I was to reconfigure the US platform.
Fortunately, all went well. Even though I replaced every single network cable and rewired most to a different configuration, there was no downtime on any service. It’s times like these when a redundant setup pays off. This was like replacing all of your car’s wheels and tyres – while doing 65 mph on the interstate. So while I was on the plane back to Amsterdam, my colleagues were already deploying new virtual machines in order to utilise the increased capacity.
About the author
Marcel is a senior Infrastructure engineer at Textkernel, responsible for managing our shared web and API services. After working in the staffing industry for a number of years, and then on high availability cloud and television services, he now combines his understanding of the two at Textkernel. Up to last year, he was also active in local politics. When he’s not working, he likes to spend time relaxing on or racing his sailboat, ride his motorbike and basically mess with anything technical.
Curious about Textkernel? We are growing and hiring!
