mirror of
https://github.com/nikdoof/website-hugo.git
synced 2025-12-13 23:52:23 +00:00
Blog update for large storage
Added aimage, bimage, bgallery and fimage shortcodes
This commit is contained in:
engineershamrock
@@ -18,17 +18,14 @@ As it turns out this is a little more complicated than I expected, even when you
|
||||
|
||||
# The Chassis
|
||||
|
||||
|
||||
I have several of these from an older project
|
||||
{{< bgallery width="90" >}}
|
||||
{{< fimage src="images/IMG_2991.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2992.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2993.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
{{< gallery >}}
|
||||
{{< image src="images/IMG_2991.jpg" width="200x" title="Top">}}
|
||||
{{< image src="images/IMG_2992.jpg" width="200x" title="Front">}}
|
||||
{{< image src="images/IMG_2993.jpg" width="200x" title="Back">}}
|
||||
{{< /gallery >}}
|
||||
|
||||
These are 4RU in height, wide and deep enough to sit on a 19" rack shelf and cabinet and are quite old now, >10 years. They were bought for a project which did not get much passed the
|
||||
development stage.
|
||||
@@ -39,129 +36,123 @@ I have put them on ebay on more than one occasion but never got the price I thou
|
||||
|
||||
If you open the large cover you find it has a location for 3 1/2 45 disks, see the link above, it goes into plenty of detail.
|
||||
|
||||
{{< image src="images/IMG_2994.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bimage src="images/IMG_2994.jpg" width="500x" height="400x" title="another" zone="60">}}
|
||||
|
||||
|
||||
The drives sit in the grid and the vibration of the disks spinning is dampened by some rubber bands, you can see a couple in the picture.
|
||||
|
||||
The drives are hot swappable and plug directly into a PCB, of which there are 9, each controlling 5 drives. The controller cards are actually just SATA port expanders, and in comparison to many port expanders work relatively well.
|
||||
|
||||
{{< image src="images/IMG_2996.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< image src="images/IMG_2997.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bgallery width="80" >}}
|
||||
{{< fimage src="images/IMG_2996.jpg" width="400" height="300" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2997.jpg" width="400" height="300" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
Unfortunately these can not be re-used in this project as we need the locations they are in to put new disks, and more of them. One thing these do provide are the screw locations for the underneath. To be discussed more later.
|
||||
|
||||
{{< image src="images/IMG_2998.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
## Starting from empty
|
||||
|
||||
I stripped out all the port expanders and all wiring to go with them and you can see it here, so just an empty chassis
|
||||
|
||||
{{< bimage src="images/IMG_2998.jpg" width="500x" height="400x" title="another" zone="60">}}
|
||||
|
||||
I left the nylon posts in place as they are threaded and allow the nylon screws, they were holding in the port exanpders, to be used.
|
||||
|
||||
If you do have one of these chassis and need the nylon posts they are
|
||||
If you do have one of these chassis and need the nylon posts and the screws, both are still available to buy
|
||||
|
||||
{{< image src="images/IMG_2999.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bgallery width="60" >}}
|
||||
{{< fimage src="images/IMG_2999.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_3001.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
The screws are
|
||||
|
||||
{{< image src="images/IMG_3001.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
Both of these are still available to buy.
|
||||
|
||||
{{< image src="images/s-l1600.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
## What components do I have
|
||||
|
||||
I am going to reuse the motherboard and as many components, fans, cables etc as possible. The motherboard is a supermicro X9SAE-V.
|
||||
|
||||
{{< image src="images/IMG_3010.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bgallery width="60" >}}
|
||||
{{< fimage src="images/s-l1600.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_3010.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
It has 6 PCI slots , on board HDMI and dual GigE, it is old, but will at least get things going.
|
||||
|
||||
After boot it shows the CPU is an old i5
|
||||
It has 6 PCI slots , on board HDMI and dual GigE, it is old, but will at least get things going. After boot it shows the CPU is an old i5
|
||||
|
||||
This is not going to set the world on fire with CPU capabilities but should be enough to make it work without spending too much money. This is after all a project rather than anything to be used in production (yet!)
|
||||
|
||||
So at this point we have a working motherboard, a case, locations for a PSU and drive bay location.
|
||||
|
||||
{{< image src="images/IMG_3009.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
We also have 3 of these
|
||||
|
||||
{{< bimage src="images/IMG_3009.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
Nothing amazing, but simple 4 port SATA controllers.
|
||||
|
||||
# Comparing drive sizes
|
||||
|
||||
Next I started to look at the size of the drives. I wanted to move away from spinny disks, power, noise, heat the primary factors but also fully loaded is very heavy!
|
||||
Next I started to look at the size of the drives. I wanted to move away from spinny disks, power, noise, heat the primary factors but also fully loaded is very heavy! So to keep things compact SSD form factor 2 1/2 is the way to go.Let's compare the sizes
|
||||
|
||||
So to keep things compact SSD form factor 2 1/2 is the way to go.
|
||||
|
||||
{{< image src="images/IMG_3005.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< image src="images/IMG_3004.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
Let's compare the sizes
|
||||
|
||||
{{< image src="images/IMG_2497.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bgallery width="60" >}}
|
||||
{{< fimage src="images/IMG_3005.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_3004.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
I created a 3d printed flat platform that the drives could stand on, just to test out how many seemed reasonable
|
||||
|
||||
{{< bimage src="images/IMG_2497.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
# Cabling
|
||||
|
||||
At this point I realized the cabling would not fit under the platform, so any new location for the drives would either need to be custom with driver chip (like the previous items) or just connectors with drivers elsewhere, but also with the SSDs being smaller (hence shorter) there is now more room to move things upwards, as it turns out about 35mms of room.
|
||||
At this point I realized the cabling would not fit under the platform, so any new location for the drives would either need to be custom with driver chip (like the previous items) or just connectors with drivers elsewhere, but also with the SSDs being smaller (hence shorter) there is now more room to move things upwards, as it turns out about 35mms of room. I then 3d printed a mask, that would allow me to use the nylong screws into the positions in the chassis but also allow me to add additional holes for anything I wanted to attach, so removing the need to match holes but allow me to extend upwards
|
||||
|
||||
{{< image src="images/IMG_3013.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< image src="images/IMG_3014.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
I then 3d printed a mask, that would allow me to use the nylong screws into the positions in the chassis but also allow me to add additional holes for anything I wanted to attach, so removing the need to match holes but allow me to extend upwards
|
||||
{{< bgallery width="60" >}}
|
||||
{{< fimage src="images/IMG_3013.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_3014.jpg" width="300" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
|
||||
# Drive capacity
|
||||
|
||||
{{< image src="images/IMG_3011.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
Now to get more drives into the chassis, I initially went for 10 drives , instead of 5 per location. This then required the use of port expanders, so I bought a few of these.
|
||||
|
||||
As it turns they are little rubbish. Not because they do not work, or that they are insanely slow but if you buy them from anywhere the heatsink is stuck in place with some kind of double sided heat resistance tape, which transfers little heat to the heatsink. This causes them to restart regularly when in use, rendering them next to useless.
|
||||
{{< bimage src="images/IMG_3011.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
{{< image src="images/IMG_3012.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
As it turns they are little rubbish. Not because they do not work, or that they are insanely slow but if you buy them from anywhere the heatsink is stuck in place with some kind of double sided heat resistance tape, which transfers little heat to the heatsink. This causes them to restart regularly when in use, rendering them next to useless. You can remove the heatsink, applying heat ( seems strange ) and it will come off
|
||||
|
||||
If you remove the heatsink, apply heat and it will come off
|
||||
{{< bimage src="images/IMG_3012.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
Then if you put it back with heat transfer epoxy, it will stop restarting, but remains incredibly slow!
|
||||
|
||||
But I did push on and get it going
|
||||
# Drive Connectors
|
||||
|
||||
{{< image src="images/IMG_2719.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
But I did push on and get it going and I got some custom PCBs made from PCBWay. Nothing clever, just locators for the connectors , nothing active. Then then came up with a design to have the port expanders underneath, so two per location, giving 10 drives.
|
||||
|
||||
I got some custom PCBs made from PCBWay
|
||||
{{< bgallery width="80" >}}
|
||||
{{< fimage src="images/IMG_2719.jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2575.jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2721.jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
{{< image src="images/IMG_2575.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
Nothing clever, just locators for the connectors , nothing active. Then then came up with a design to have the port expanders underneath, so two per location, giving 10 drives.
|
||||
|
||||
{{< image src="images/IMG_2721.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
and then with the PCB in place As you can see, it kind of fits ?
|
||||
|
||||
{{< image src="images/IMG_2752.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
So in my wisdom I thought I could fit more connectors on the PCB, so I went for 14 drives per PCB, two rows of 7, I would have two port expanders underneath and using some method not determined find a location for a 3rd one. If I have 9 slots, 14 x 9 = 126 , so I get to my goal with a few to spare!
|
||||
|
||||
So in my wisdom I thought I could fit more connectors on the PCB, so I went for 14 drives per PCB
|
||||
{{< bimage src="images/IMG_2752.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
Two rows of 7, I would have two port expanders underneath and using some method not determined find a location for a 3rd one. If I have 9 slots, 14 x 9 = 126 , so I get to my goal with a few to spare!
|
||||
So a completed PCB, it is a very simple design, two locations to power the drives and then pass through connectors for SATA
|
||||
|
||||
{{< image src="images/IMG_2780 (1).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< image src="images/IMG_2779 (1).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bgallery width="60" >}}
|
||||
{{< fimage src="images/IMG_2780 (1).jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2779 (1).jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
So a completed PCB
|
||||
|
||||
It is a very simple design, two locations to power the drives and then pass through connectors for SATA
|
||||
|
||||
{{< image src="images/IMG_2762 (1).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
Then I wired it up with the SATA cables I had
|
||||
|
||||
It looks pretty, and did work, 20 drives and a boot drive, so 21 drives.
|
||||
|
||||
{{< bimage src="images/IMG_2762 (1).jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
At this point, which was quite a bit of work, I realized with much distress, it was slower than a snail, at one point I was getting an underwhelming 10Mb/s !!!
|
||||
|
||||
# The current status
|
||||
@@ -184,20 +175,17 @@ I now need to find a solution to control a large number of disks, without breaki
|
||||
|
||||
If you look at this motherboard it is 2 x 1x and 1 x 2x and 3 x 4x PCIE interfacers so options are limited on the number and type of interface cards.
|
||||
|
||||
There are multiple SATA cards that provide good 8x SATA interfaces such as this [StarTech](https://www.startech.com/en-gb/cards-adapters/8p6g-pcie-sata-card)
|
||||
|
||||
it's a fast card, does the job well, so with 3 of these in I get 24 ports. Alas not getting me close to the 120 I wanted, so no real use.
|
||||
There are multiple SATA cards that provide good 8x SATA interfaces such as this [StarTech](https://www.startech.com/en-gb/cards-adapters/8p6g-pcie-sata-card) it's a fast card, does the job well, so with 3 of these in I get 24 ports. Alas not getting me close to the 120 I wanted, so no real use.
|
||||
|
||||
# Time to go SAS
|
||||
|
||||
{{< image src="images/s-l1600 (1).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
SAS cards are incredibly cheap and relatively quick and compatible with SATA drives, not to be confused with SAS drives ARE NOT compatible with SATA controllers. This one is goes for about $30 and has 2 ports on board.
|
||||
|
||||
{{< image src="images/s-l1600 (2).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
To make the best use out of it you also need to expand the number of ports available, which is where this card comes in.
|
||||
|
||||
{{< bimage src="images/s-l1600 (2).jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
It is an Intel RES2SV240 RAID card. It is not really a RAID card in this case, but provides 24 ports, of which 4 are inbound and 20 outbound. This card is relatvely good as you can then use SFF8087 cables to SATA ports, keeping the cabling manageable, rather than individual SATA cables
|
||||
|
||||
So let's do the maths
|
||||
@@ -207,21 +195,21 @@ So let's do the maths
|
||||
|
||||
So three (3) LSI cards and 6 RES240 cards = 120 ports. We are there.
|
||||
|
||||
{{< image src="images/s-l1600 (2).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
|
||||
Even better the RES240 cards do not actually need to plug into the motherboard, they just need power and are fitted with an external power socket!!!
|
||||
|
||||
- This is an amazing win.
|
||||
- I started the build out as follows
|
||||
{{< bimage src="images/s-l1600 (2).jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
{{< image src="images/IMG_2790.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< image src="images/IMG_2800 (1).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< image src="images/IMG_2804.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
This is an amazing win so I started the build out as follows
|
||||
|
||||
{{< bgallery width="80" >}}
|
||||
{{< fimage src="images/IMG_2790.jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2800 (1).jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< fimage src="images/IMG_2804.jpg" width="250" height="200" float="inline-block">}}
|
||||
{{< /bgallery >}}
|
||||
|
||||
Add some drive location testing
|
||||
|
||||
{{< image src="images/IMG_2883.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bimage src="images/IMG_2883.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
It is brilliant what could go wrong!
|
||||
|
||||
@@ -239,25 +227,25 @@ Close but not yet 120 drives - ARG!!!!!
|
||||
|
||||
As I was cabling this, space for cabling was also becoming a premium, even with SFF cables. So now I changed tac on the design for the last time, and brough my expectations down a little.
|
||||
|
||||
{{< image src="images/IMG_2883.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bimage src="images/IMG_2883.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
So I did have this
|
||||
|
||||
{{< image src="images/IMG_2892.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bimage src="images/IMG_2892.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
I am now going to do this
|
||||
|
||||
{{< image src="images/IMG_2905.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bimage src="images/IMG_2905.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
Instead of using the 9 locations for drives I am going for 6. The last 3 I can use for the RES240 cards and allow some space for cabling, air flow and as it turns out some boot space as well.
|
||||
|
||||
In this picture you can see 84 drives, but I did mention 72, where did the other 12 come from ? Remember the 3 original controller cards, as it turns out they fit into the remaining PCI slots.
|
||||
|
||||
{{< image src="images/IMG_2937 (1).jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bimage src="images/IMG_2937 (1).jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
We now end up with this
|
||||
|
||||
{{< image src="images/IMG_2906.jpg" width="150x" class="is-pulled-right" height="200x" title="">}}
|
||||
{{< bimage src="images/IMG_2906.jpg" width="400x" height="300x" title="another" zone="30">}}
|
||||
|
||||
To show that it all works under Linux
|
||||
|
||||
|
||||
Reference in New Issue
Block a user