- 1 Target
- 2 Topologie and installation
- 3 Income route
- 4 Backup
- 5 Hot switching
- 6 Routing and bandwidth
- 7 Links
- 8 More advanced need
- 9 Datacenter in bolivia
- 10 MACCHIATObin ClearCloud 8K
- 11 Photo
Have fix url or access, included with change of internet provider, or have multiple internet connexion. This include for other service than http, like: mail (pop, smtp), irc, xmpp, ... Have access to site web via the normal web thanks to inproxy (XXXXXX.b32.i2p.us), and less sensible to dns or routing security leak.
Have decentralised service, with minimum latency, when you can't/wan't dynamic dns and ipv6. Or multiple same service on same port, on you network. You can route 2 server on port 80 without change on internet port or have reverse proxy (haproxy and i2p).
Topologie and installation
On toc and utilite:
*nat :PREROUTING ACCEPT [966:55985] :INPUT ACCEPT [32:2024] :OUTPUT ACCEPT [19:1212] :POSTROUTING ACCEPT [0:0] -A POSTROUTING -o ethToInternet -j MASQUERADE COMMIT *filter :INPUT ACCEPT [112:7798] :FORWARD DROP [1:60] :OUTPUT ACCEPT [0:0] -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.142.0/24 ! -d 192.168.142.0/24 -i ethLocalNetwork -j ACCEPT # for utilite only -A FORWARD -s 192.168.0.0/24 -d 172.16.0.0/16 -i ethToInternet -j ACCEPT COMMIT
route add -net 172.16.0.0 netmask 255.255.0.0 gw 192.168.0.2 ethToInternet
On utilite, deny the access to 192.168.X.X, only forward to internet. But from 192.168.142.X will access to 192.168.0.X, 172.16.X.X and internet.
The reason to have an vanilla kernel:
- Have the last fix for the kernel, most of ARM vendor ship an kernel and only update their part, mean forget all the security fixes since it have snapshoted their version
- Have access to patch as vserver, Grsec, ... the repatched ARM kernel is mostly not capable of this and their new driver not ship the corresponding part. In case of Grsec, some drivers have memory problem reveled with grsec and then not boot
- Kernel 4.1+ is highly recommander for LXC
The modern kernel:
- Support better the ARM arch, for example by converting more assembly code to C code to be more same and have same behavoiur on all platform
- Better ARMv8
- More lock less code to greatly improve the performance on hardware without barrier/lock hardware (mean: ARMv6, ARMv7, ARMv8, support is included into: ARMv8.1+)
- More code optimisation to have less system time
To route the income connexion, for http I use haproxy for normal ip trafic. But I route via i2p too for secure and more general routing (http, MMORPG server, ...).
The backup is snapshoted by btrfs, rsync to be exported, and keep as snapshot to have backup diff as high efficienty and compressed backup
Hot switching is allowed by lxc 1.1 to migrate transparently the vm from one hardware to other. LXC 2 give:
- More consistent user experience between the various LXC tools.
- Improved checkpoint/restore support.
- Complete rework of our CGroup handling code, including support for the CGroup namespace.
- A massive amount of bugfixes.
- Place geographically where you have the best bandwith for datacenter and customer
- Easy close datacenter node
- Equilibrate the load on the hardware
- Nearest database to their site
Routing and bandwidth
Standard level 2 switch is used, not administrable, then how do you protect? By the virtualisation layer, it force the ip and mac address, route de packet, do some firewall filter.
The over sized network
It's transparent seam less bandwidth. The final server customer don't need care of the server bandwidth because the top level network is over sized.
The customer don't think before buy. Mean if the need it's just have load balancing, it take 2 server anywhere, producing over consuming of top level network as the backbone. Same for server for http + php and another to database.
Advantage: No over loaded top level network. Effective for high availability strategies.
Disavantage: More cost for the customer (more expensive hardware and over usage of bandwidth). Tend to random place of server, mean higher latency.
The similar network
You have the plain bandwidth in local, and slow on server into another rack. The final customer need think where place it.
The customer need studie few bit the server topologie in case of multiple dependent server.
Advantage: Lower price, more accessible technologies
Disavantage: Lower bandwidth between server of different place. Need compression and take care for high availability strategies.
More advanced need
More advanced need for complex structure.
This is the result of overall cluster temperature of each node after 2h at idle (minor burning time) at 14C ambiant temperature without cooling.
The power consumption on 5V line is here with kernel 3.14.77+ and connected with 1000Mbps is:
- With governor performance at idle: 0.32A
- With governor performance at 100% of CPU: 0.67A
- With governor ondemand at idle: 0.29A
- With governor ondemand at 100% of CPU: 0.65A
- With governor powersave at idle: 0.29A
- With governor powersave at 100% of CPU: 0.30A
- With governor powersave at 100% of network: 0.315A
Datacenter in bolivia
More info on: Datacenter in bolivia
I have done the electrical power bus, it's 2*23 Odroid C2, 1000Mbps. I used 46 MicroSD of 64GB. Power used (mesured at the 220V in Seasonic serie G power supply): 56W, disconnected power supply: 7W. 49W at 100Mbps. Peak: 111W When all is working and used, I thinks buy 1000 Odroid C2 or Odroid XU5 (if is released). I really need vanilla linux kernel support (for headless server: Ethernet, MMC, ...), this allow usage of BTRFS to pass from 20MB/s read and 4MB/s write to 100/MBs read and 20MB/s write for my common case, mean average 5x boost without speak about space improvements.
- Other hardware used: TL-SG1024D (24 1000Mbps port switch not administrable), Industrial power supply Ebest CFS-200-5: OUTPUT: 5V 40A. Custom network wire to have the correct length and cat 5e due to sort distance.
- OS: Gentoo ARMv8, Official Ubuntu 16.04 kernel (3.14LTS), hypervisor: Lxc + LXCFS (need more isolation because remain see the ressources usage of other VM), not vserver or openvz because ARM provide only vanilla kernel
- Hardware by node: Amlogic S905 (4*Cortex A53 at 1.5GHz with 512KB of L2), memory: Samsung K4B4G1646D: 2GByte DDR3 32bit RAM, Ethernet: Realtek RTL8211F 1000Mbps
- From /dev/cpuinfo (Note: NO AES/SHA1/SHA2 for crypto acceleration):
- Features : fp asimd crc32
- CPU implementer : 0x41
- CPU architecture: 8
- CPU variant : 0x0
- CPU part : 0xd03
- CPU revision : 4
- Total cluster horse power: 184 64Bits cpu with SIMD, 138GB/s of main memory bandwidth for non cached data access as big data, estimated L1 bandwidth: 2TB/s, 92Gbps external bandwidth, 92GB of DDR3, 2944GB for storage.
Take care to well grounding all the hardware! Electric power grounding
Typically a third conductor, called ground (or "safety ground") (U.S.) or protective earth (UK, Europe, IEC), is used as a protection against electric shock, and ordinarily only carries significant current when there is a circuit fault. Several different earthing systems are in use.
I have used odroid c2 power connector of rev0.2 20160226, but too hard to sold. Hole to small to stack 15 odroid. Then I have used the traditional power connector, all is connected on big wire with correct copper diameter to support 20A. (See the photo above)
MACCHIATObin ClearCloud 8K
- 12V 0.46A (5.52W) at idle with SFP+ up
- performance non constant with CPU governor ondemand, but no power consumption change