Portable datacenter

From CatchChallenger wiki
Jump to: navigation, search

Target

Have fix url or access, included with change of internet provider, or have multiple internet connexion. This include for other service than http, like: mail (pop, smtp), irc, xmpp, ... Have access to site web via the normal web thanks to inproxy (XXXXXX.b32.i2p.us), and less sensible to dns or routing security leak.

Have decentralised service, with minimum latency, when you can't/wan't dynamic dns and ipv6. Or multiple same service on same port, on you network. You can route 2 server on port 80 without change on internet port or have reverse proxy (haproxy and i2p).

Topologie and installation

Personnal-network-topologie.png
Nagios-statusmap.png
Nagios-map-4.1.png
Photo-portable-datacenter.jpg
Nagios-bi-datacenter.jpg

On toc and utilite:

*nat
:PREROUTING ACCEPT [966:55985]
:INPUT ACCEPT [32:2024]
:OUTPUT ACCEPT [19:1212]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o ethToInternet -j MASQUERADE
COMMIT

*filter
:INPUT ACCEPT [112:7798]
:FORWARD DROP [1:60]
:OUTPUT ACCEPT [0:0]

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.142.0/24 ! -d 192.168.142.0/24 -i ethLocalNetwork -j ACCEPT
# for utilite only
-A FORWARD -s 192.168.0.0/24 -d 172.16.0.0/16 -i ethToInternet -j ACCEPT

COMMIT

And:

sysctl net.ipv4.ip_forward=1

On toc:

route add -net 172.16.0.0 netmask 255.255.0.0 gw 192.168.0.2 ethToInternet

On utilite, deny the access to 192.168.X.X, only forward to internet. But from 192.168.142.X will access to 192.168.0.X, 172.16.X.X and internet.

Kernel

Vanilla kernel

The reason to have an vanilla kernel:

  • Have the last fix for the kernel, most of ARM vendor ship an kernel and only update their part, mean forget all the security fixes since it have snapshoted their version
  • Have access to patch as vserver, Grsec, ... the repatched ARM kernel is mostly not capable of this and their new driver not ship the corresponding part. In case of Grsec, some drivers have memory problem reveled with grsec and then not boot
  • Kernel 4.1+ is highly recommander for LXC

Modern kernel

The modern kernel:

  • Support better the ARM arch, for example by converting more assembly code to C code to be more same and have same behavoiur on all platform
  • Better ARMv8
  • More lock less code to greatly improve the performance on hardware without barrier/lock hardware (mean: ARMv6, ARMv7, ARMv8, support is included into: ARMv8.1+)
  • More code optimisation to have less system time

Income route

To route the income connexion, for http I use haproxy for normal ip trafic. But I route via i2p too for secure and more general routing (http, MMORPG server, ...).

Backup

The backup is snapshoted by btrfs, rsync to be exported, and keep as snapshot to have backup diff as high efficienty and compressed backup

Hot switching

Hot switching is allowed by lxc 1.1 to migrate transparently the vm from one hardware to other. LXC 2 give:

  • More consistent user experience between the various LXC tools.
  • Improved checkpoint/restore support.
  • Complete rework of our CGroup handling code, including support for the CGroup namespace.
  • A massive amount of bugfixes.

That allow physical mobility: Datacenter-truck.jpg Decentralised-datacenter.jpg Bolivia-load.jpg

  • Place geographically where you have the best bandwith for datacenter and customer
  • Easy close datacenter node
  • Equilibrate the load on the hardware
  • Nearest database to their site

Routing and bandwidth

Advanced router

Standard level 2 switch is used, not administrable, then how do you protect? By the virtualisation layer, it force the ip and mac address, route de packet, do some firewall filter.

Bandwidth

Network-link-bandwidth.png The first case is over sized network, the second case is similar network.

The over sized network

It's transparent seam less bandwidth. The final server customer don't need care of the server bandwidth because the top level network is over sized.

The customer don't think before buy. Mean if the need it's just have load balancing, it take 2 server anywhere, producing over consuming of top level network as the backbone. Same for server for http + php and another to database.

Advantage: No over loaded top level network. Effective for high availability strategies.

Disavantage: More cost for the customer (more expensive hardware and over usage of bandwidth). Tend to random place of server, mean higher latency.

The similar network

You have the plain bandwidth in local, and slow on server into another rack. The final customer need think where place it.

The customer need studie few bit the server topologie in case of multiple dependent server.

Advantage: Lower price, more accessible technologies

Disavantage: Lower bandwidth between server of different place. Need compression and take care for high availability strategies.

Links


More advanced need

More advanced need for complex structure.

Actual-need-network-route.png
Final-network-route.png

Temperature cooling

This is the result of overall cluster temperature of each node after 2h at idle (minor burning time) at 14C ambiant temperature without cooling.

Datacenter-temperature-without-cooling.png

Extracted form internet: Odroid-temperature.jpg

Power consumption

Professional-pannel-solar-rpi.jpg

The power consumption on 5V line is here with kernel 3.14.77+ and connected with 1000Mbps is:

  • With governor performance at idle: 0.32A
  • With governor performance at 100% of CPU: 0.67A
  • With governor ondemand at idle: 0.29A
  • With governor ondemand at 100% of CPU: 0.65A
  • With governor powersave at idle: 0.29A
  • With governor powersave at 100% of CPU: 0.30A
  • With governor powersave at 100% of network: 0.315A

Datacenter in bolivia

More info on: Datacenter in bolivia

I have done the electrical power bus, it's 2*23 Odroid C2, 1000Mbps. I used 46 MicroSD of 64GB. Power used (mesured at the 220V in Seasonic serie G power supply): 56W, disconnected power supply: 7W. 49W at 100Mbps. Peak: 111W When all is working and used, I thinks buy 1000 Odroid C2 or Odroid XU5 (if is released). I really need vanilla linux kernel support (for headless server: Ethernet, MMC, ...), this allow usage of BTRFS to pass from 20MB/s read and 4MB/s write to 100/MBs read and 20MB/s write for my common case, mean average 5x boost without speak about space improvements.

  • Other hardware used: TL-SG1024D (24 1000Mbps port switch not administrable), Industrial power supply Ebest CFS-200-5: OUTPUT: 5V 40A. Custom network wire to have the correct length and cat 5e due to sort distance.
  • OS: Gentoo ARMv8, Official Ubuntu 16.04 kernel (3.14LTS), hypervisor: Lxc + LXCFS (need more isolation because remain see the ressources usage of other VM), not vserver or openvz because ARM provide only vanilla kernel
  • Hardware by node: Amlogic S905 (4*Cortex A53 at 1.5GHz with 512KB of L2), memory: Samsung K4B4G1646D: 2GByte DDR3 32bit RAM, Ethernet: Realtek RTL8211F 1000Mbps
  • From /dev/cpuinfo (Note: NO AES/SHA1/SHA2 for crypto acceleration):
    • Features  : fp asimd crc32
    • CPU implementer : 0x41
    • CPU architecture: 8
    • CPU variant  : 0x0
    • CPU part  : 0xd03
    • CPU revision  : 4
  • Total cluster horse power: 184 64Bits cpu with SIMD, 138GB/s of main memory bandwidth for non cached data access as big data, estimated L1 bandwidth: 2TB/s, 92Gbps external bandwidth, 92GB of DDR3, 2944GB for storage.

Take care to well grounding all the hardware! Electric power grounding

Typically a third conductor, called ground (or "safety ground") (U.S.) or protective earth (UK, Europe, IEC), is used as a protection against electric shock, and ordinarily only carries significant current when there is a circuit fault. Several different earthing systems are in use.

I have used odroid c2 power connector of rev0.2 20160226, but too hard to sold. Hole to small to stack 15 odroid. Then I have used the traditional power connector, all is connected on big wire with correct copper diameter to support 20A. (See the photo above)

Video: https://www.youtube.com/watch?v=DZXUCmBf_bg

Photo

Bi portable datacenter, Red and Blue
Running ARM cluster at 1000Mbps/1Gbps
Cloud of Odroid C2 ARMv8 Cortex A53
Power cord for High density cluster
Custom power via Odroid c2 power connector rev0.2 20160226
Running dual datacenter cluster
Solar conception for datacenter with common thing in bolivia
Solar conception for datacenter with high efficiency
Reverse proxy ipv4 to ipv6
How cable the ovh ip to you home server into your local datacenter. After you attach the virtual machine in same way on too brige