I'm on a boat - Part 3

So the last post left off here:

Just like the thermostats housing before it, there were a couple of issues.  The bolt holding the lower anode in snapped when I tried to remove it.  The irony there is that it didn't need to come out to remove the lower section of the outboard.  There were also a couple more serviceable parts that needed to be replaced once I had the lower section off, which meant waiting a week or two for more parts to arrive.  Add to that a crazy couple of weeks at work and this past weekend was the first chance I had to get Dusty back together.  When the lower section came off a few weeks ago, it looked like this:

First thing was to clean it all up.  There was a lot of sand and gunk in there, the seals on the exhaust adapter between the lower and upper section of the outboard had melted away and I needed to get that snapped bolt out to replace the rear anode (the anode was now loose and would have made a fair bit of noise vibrating around).  I took to the entire lower section with a mixture of a wire brush on a cordless drill, and a wire brush on my Dremel for some of the trickier bits.  I also cleaned up the exhaust adapter that goes between the lower and upper section of the outboard.  I don't have a shot of it before, but it was pretty bad.  Here it is cleaned up with new seals ready to go:

Once that was done I drilled out the bolt for the rear anode an installed the new one:

Next step was to rebuild the water pump.  The pump kit came with instructions which were pretty straight forward.  Nice shiny new parts:

Once that was all together and assembled on the lower section, it was ready to go back on:

This was actually quite tricky by myself (it's got a bit of weight to it).  It took me a few goes to get it all lined up, but once it was it just slotted straight into place.  I did up all the bolts and stood back to admire my greasy fingerprints:

Only thing left to do was to turn it over.  It started pretty well, with a much stronger stream of water coming from the tell tale.  A great sign the the new pump was pushing the full amount of water through.  I let it run for a good 15 minutes and could feel when the thermostats opened and hot water started to cycle through.  It ran the entire time without any cooling system issues that I could see.  Next step is to change the oil in the gear case (since its the last serviceable item I haven't done) and put Dusty back in the water for a test trip.  Finger crossed that all the maintenance for the season done.  She needs a good clean to remove all those greasy fingerprints.  I've also thought about getting a replacement decal set from eBay, but i'll wait on any cosmetic bits and pieces until I know I've worked out all the mechanical issues.

I'm on a boat - Part 2

So a few weeks ago now we took Dusty down to give him a run with his shiny new (old) VRO pump.  We put him in the water and once warmed up he was running really well.  Nice and smooth, humming along.

After about 10 minutes at a decent speed the over heat buzzer went off.  My first thought was that I hadn't fixed the wiring like I thought previously.  However when I turned around to look at the outboard I noticed the telltale was spitting a bit.  So I cut the engine to have a look.  I thought maybe I'd sucked up a plastic bag or something and there wasn't enough water getting through.  I pulled the engine up and there was nothing visible on the intake.  I dropped the engine back down, pulled the cowl off and started it again.  The buzzer was still going - and checking the wiring it was definitely one of the temp senders this time (not the low oil) that was grounding to sound the alarm.

It was at this point that the telltale got worse, not only spitting but getting to the point were it was just putting out steam - not good.

So I shut the engine off and got a tow back in from the father in law on the jet ski.

After talking about it a bit and googling around I decided to take a look at the thermostats and water pump.  I set out early one Saturday morning to get the thermostat housing off to have a look and see what was going on:

This is the housing, at the bottom of the back of the outboard.  The housing has two chambers (one for each cylinder head), each with a thermostat and a pressure release valve.  After removing the two hoses, there are just 3 bolts that need to come out.  The first two came out easily enough, but the third would not move.  It didn't help with where the bolt was - in that I was limited in what tools (and in which position) I could get them onto the bolt.  So after stripping the bolt, filing it down to a smaller size, using some heat and some WD40 it still wouldn't budge.

I ended up having to hacksaw the corner of the housing to get it off.  Not ideal - but I couldn't see any other way:

So thanks to the bolt, the housing was cactus.  That middle spacer is plastic and snapped pretty quickly.  The thermostats were both completely gone, both being stuck just slightly open.  The hoses were also stuffed.  Time to order some parts and wait by the mailbox.  In the mean time I cleaned up surface on the outboard ready for new bits with a wire wheel on my Dremel:

Fast forward a week or so and the parts arrived:

Pretty straight forward assembly.  Comparison shot:

Putting it back on wasn't too bad, just a little tricky with the springs on the release valves.  Once back on the outboard, just a matter of putting the new hoses on, and that was it:

With that sorted, I currently have the gear case off to rebuild the water pump.  I'm just waiting for the parts to arrive.  Once they do and the pump is rebuilt I'll post that up.  Hopefully then Dusty is back on the water!

I'm on a boat

So September has seen a bit of a change, in that instead of spending the little bit of time I get at the bench, it's been spent outside doing a bit of work on the boat getting it ready for a summer on the water!  The following happened over the last 3 weekends, and a little bit here and there during the weeks in between.  It was all kind of going on at the same time - but i've written it out as if it was one after the other.

We inherited the boat from my wife's granddad.  It's a fibre glass runabout with a 1985 Johnson V4 90hp oil injected two stroke.  The Johnson was one of the first models that used a Variable Rate Oiling or VRO pump.  These VRO pumps are a real sticking point on the internet and at the jetty alike.  People seem to give them a real bash for being unreliable and causing an outboard to have running problems or worse still - kill the outboard altogether (due to lack of oil).

We've only had the boat for a couple of summers, but it's been running without issue for 29 years.  I'm reliably informed my wife's granddad used the boat more than just about anything else. It's a pretty simple principal - the pump runs off pulses on the bottom of the crank case.  It mixes in petrol and a variable amount of oil based on the pulse output of the engine and pumps that mixture into the engine.  One of the best resources on VRO pumps is this link.

The pumps have changed a little over the years.  As this outboard was one of the first with a VRO - the pump has no electronics to measure and subsequently alarm when the oiling side of the pump fails.  Johnson also introduced a pulse limiter to use on the output of the crank case which runs the pump - to stop any damage to the pump from over pulsing (think back fires etc).

So with that context, I went outside to pull the cowl off the outboard and give it a run to see where things were at heading into summer.  There were 4 things that I noticed straight away:

1. The straps on the bimini covers had all broken
2. The hose from the fuel tank to the outboard had gone hard and looked like it might crack
3. The cable ties holding the remote control arm together had broken
4. The temp buzzer was on - before I even started the engine for the first time (this has been an intermittent issue since before we got the boat)

I metered the resistance on one of the temp senders while still connected (there are two) and it was pretty close to zero - even though the motor was dead cold.  Either one of the senders was dead or there was a wiring issue.  I parked this one here (knowing that the motor wasn't overheating), and figured I'd come back to it once the engine was running.  It's been one of those niggling issues that I've been meaning to fix since we got the boat.  I headed down to the shops and got some replacement  bimini straps, a new fuel hose and filled the fuel tank.

I came home and hooked up the new hose, pulled the battery off charge and connected it up.  With the muffs on and the hose running I cranked her over.  Noting that she hadn't been started in somewhere around 4 to 5 months (amazing where time goes when you become a Dad), she started pretty well after a little bit of cranking combined with some choke.

Once running she seemed ok - until I had a closer look and noticed fuel leaking from the VRO pump.  Bummer.  Most complaints about VRO pumps all revolve around the oiling part of the pump not working 100% or failing altogether.  My VRO though was leaking fuel.  It was still pumping fuel and oil just fine - with the engine running well.  Having a closer look at the VRO, it was quite obvious that the leak wasn't new, and someone had previously had a go at sealing it up:

Unfortunately the pump housing is plastic.  If it was metal you'd just be looking for a seal kit to rebuild the pump, but as you can see in the photo above the plastic housing of the pump is cracked.  So there's no chance of fixing it - someone's already tried that.  

I did a bunch of looking and ringing around.  As I mentioned this engine is almost 30 years old now and the VRO pumps have changed a little bit over time.  I needed to figure out what part number I was actually chasing.  One of the best sites I found was this one.  You can find a part on your outboard and it will show you any superseded part number when you look the part up.  Awesome.

Turns out a replacement VRO pump is somewhere around $600-$700.  No.  Not happening.  

A few places did sell replacement after market pumps that are just fuel pumps for about $180.  This seems to be what everyone who dislikes VRO's says to do.  Disconnect the oil tank and just premix the fuel with oil.  Even though this would mean a brand new pump, I didn't overly want to do this so I parked this option as a backup.

I then rang a few boat wreckers, and managed to find a replacement VRO for $165.  Much better.   However the guy at the wreckers seemed strongly against VROs in general.  His choice.  The one thing he did say was I needed to bring the old pump in to match it up.  While the various versions of the VRO pumps over the years are pretty much the same thing internally, the angle of some of the hose connectors does change depending on the outboard.  So out I went to pull the existing VRO off.  In doing so I noticed most of the connecting hoses were pretty hard and ready to be replaced.  

Off to the wreckers to get the pump, and then the shops to get some replacement fuel hose and hose clamps.  

It was pretty easy to put the new (old) pump on with some new sections of fuel hose and hose clamps.  I primed the oil and fuel lines and cranked her over again.  

She started easily and ran really well - without leaks!  Winner.

Next stop was the temperature warning buzzer.  As mentioned above it's been an intermittent issue since before we inherited the boat, it was time to figure it out and fix it.  It only came on sometimes, but the engine never overheated.  I'd played with a couple of connections that appeared to sometimes make a difference, so I just put it down to a wiring problem.  At this point I should note the electronics on this outboard are very basic.  After a bit of reading online, I learnt the following:

• The buzzer sounds when the temp circuit grounds
• There are two temp senders - one in each head
• Any VRO oil warning circuitry is connected into the temp circuit in parallel
• Lots of posts online about people finding shorts somewhere in the wiring causing the problem

With all this new knowledge in mind and a meter in hand it was time to track down the gremlin once and for all.  I started out with each temp sender individually.  Both passed.  Stupidly high resistance when cold, and it started to drop when they warmed up. 

Noting what I said earlier about my VRO pump being 1st generation and not having any electrics for warning circuitry, I was almost ready to rule this point out.  That was until I started tracing the wiring from the outboard back to the remote control where the warning buzzer is.  I noticed a wire coming in from the oil tank.  Confirming online, there is a float in the remote oil tank that earths out when the oil gets low to sound a "low oil" alarm.  I disconnected this guy from the circuit and the buzzer stopped.  Bingo.  Metering it the resistance was almost non existent.  Problem found.  

I set about pulling the oil pickup/float out of the oil tank (it's all one piece).  I started having a look to see if it was stuck/could be fixed and it basically fell apart in my hands.  Awesome.

It was a mix of plastic and metal - and it was busted.  Not just the float, but the pick up too.

I headed back online to look for a replacement pickup/float.  At least $200.  Did I just do the wrong thing getting a new VRO and not just replacing it with a straight fuel pump and premixing?

I checked eBay, and BAM.  A full (that's right full - not just the pickup) remote oil kit (old new stock), about 20 minutes for home - for $15!!!  I couldn't get down there to pick it up quick enough:

Fitting this guy up was pretty straight forward.  The hose and two wires needed to be trimmed as they were both massively too long at factory length.  Once hooked up - I filled it with some nice new marine 2 stroke oil (I primed the oil line as best I could when hooking it up) and turned the key to on - no warning buzzer!  So I started the outboard and again - no buzzer.  She runs pretty smooth. 

Buzzer issue solved.

I should mention that somewhere in there while trying to find the problem with the buzzer - I blew the main fuse on the motor.  This wouldn't have been much of an issue (after all - what are fuses for?) but the holder took a very weird size cylinder fuse that I couldn't get in the local SuperCheap or Autobarn.  That combined with the wires on either end being worse for ware, I simply bought a new standard size blade fuze holder and put it in.  Another problem (introduced while fixing the outboard) fixed.

So last problem - the remote control.  It was broken and held together with cable ties when we got it.  When I was at the wreckers, I mentioned it to the bloke there and he knew straight away what I was talking about.  The actual throttle arm is metal, so its fine, but there is a plastic piece on one side of the arm that holds the tilt switch at the top.  This piece just snaps off after a while due to it's design.  Given this - he said second hand non-broken replacements don't exist and that I'd have to get one new.  $70.  I told him I'd think about it and went home and used some duct tape:

One other part that I was keen to get was the pulse limiter that is mentioned in the external VRO link:  

"Since the VRO depends on crankcase pulses to operate, it is susceptible to backfires from a lean running cylinder or an out of tune engine. If your motor is older than a 1993, make sure it has the blue colored pulse limiter to protect the air motor and the check valves in the pump. Follow the pulse line from the VRO to the engine block and look for a hex shaped fitting threaded into the crankcase. If it has a black face on it, replace it with a blue style (OMC P/N 435009)."

No surprise that mine didn't have the blue limiter installed.  I wanted to make sure I put one of these on since I'd just spent the time and money replacing the VRO pump.  Knowing that I needed the pulse limiter and a replacement plastic arm for the remote control I got hunting online again.  I ended up going with this place based out of Sydney.  They have an online parts quoting form and once submitted Paul responded promptly to clarify my order, and that was that.  The prices with shipping were better than I got phoning around in locally too.  $54 for the pulse limiter and $32 for the plastic arm - winner!

Once the parts arrived I set about installing both.   The pulse limiter install I didn't really get a decent picture of (it's hard when your hands are covered in grease).  I did get a couple of pics of the remote control though (white part is metal, black part is plastic):

All done.  I've pumped up the tyres on the trailer and she's good to go.  We've got a long weekend coming up - so fingers crossed I should be able to get some time on the water to put the old girl through her paces.  Bring on Summer!

Commodore 128 - Part 4

This week my 1541 Ultimate arrived:

This thing is nothing short of amazing.  Sure they are not super cheap - but they are awesome, and I can only imagine the man hours put into the design of this thing.  I like supporting people that have built stuff on their own in their spare time and have come up with something great.  I've only spent half an hour or so using the 1541U and it works exceptionally well.  I know there are other things it does - I'll need to spend more time getting to know it better.  So what did I load first with this thing?  Games of course!!!!

Next on the list is the right adapter/cables so I can get some sound, and a joystick or two!

Commodore 128 - Part 3

I got my hands on a Dell 2408wfp at the end of the week.  This guy has an S-Video input, so I could remove the S-Video - VGA converter I was using in the previous post.  I plugged it all in and flicked the switch:

Much better! GO64:

Not sure I will get it much better than this - open to suggestions though.  Now that I have a working setup I can start to look at what I might run/do with this thing, I've got a few ideas.  More to come!

Commodore 128 - Part 2

tl;dr It works!  Read on for detail.

So I finally received enough bits to connect up the 128 and power it up and see if it works.  First up I needed a power supply, so I ordered one from Ray Carlsen.  Ray makes a replacement 64/128 power supply with new parts.  The design is available but as Ray mentions, sourcing all of the parts wouldn't have worked out much different to buying one from Ray.  He's a great guy who's been repairing Commodores for a long time.  Pretty impressed with the supply, and its good to know that I won't be troubleshooting any issues with the power supply side of things.  It is setup for 240v - but it still has an American plug on it.  I'm using an adapter for the moment, but once I have a setup i'm happy with i'll put an Australian plug on it:

With the power side of things covered I needed to look to video out.  The 128 has three ports for video out, the round DIN connector, the RF connector and the DB9 connector.  The round DIN connector and RF connector display video in 40 column mode, which is what the majority of software for the Commodore 64/128 uses.  You can very easily get a cable that goes from the DIN connector to S-Video - so that's what I ordered (figuring that would be better/easier than using the RF connector and tuning in a TV):  

The DB9 connector requires a little more circuitry to work with a modern display and outputs video in 80 column mode, for which there isn't a great deal of software.  So while that is something I intend on trying, I wanted/needed to look into it a bit more before ordering bits.  

Once I'd ordered the DIN to S-Video cable and was waiting for it to arrive I went looking at the TV I had to see how easy it would be to get a cable to it - and low and behold it does't have an S-Video input!  So I jumped back online and ordered an S-Video to VGA adapter.  I got the cheapest one I could find ($10AUD):

So with all the parts finally in hand I plugged it all in, said a little prayer and flicked the power switch on the 128.  

Nothing.

No power light.  No smoke.  No burning smell.  No video output.  Bummer

I'd debated taking the top off the 128 for the first power on in case something went bad but thought I'd give it a try first.  So with it failing to power on I grabbed the power cable to pull it out and as I moved it the 128 turned on!

The video output was stable but not perfect.  I tried reseating cables and some different power supplies for the S-Video to VGA adapter but there was no difference.  Most of my googling and chatting to other Commodore users pointed at the same conclusion - the $10AUD S-Video to VGA convertor wasn't the best.  I've got a plan to get a display that will just take S-Video straight in.  Will write a blog post when I have something, showing whether the video output is any better.

So with the 128 working, one of the first things I did was "GO64":

I then reset and typed "SYS 32800,123,45,6":

So both the 128 and 64 modes work in 40 column mode.  As I mentioned to go 80 column I need some circuitry to use the DB9 port, and I also intend on getting  display that can take S-Video input directly.  I'll do another post on both of those when those parts arrive, which hopefully isn't too far away.

Soldering Practice

While I have been reading and building a few things on my own, process is a little slow going due to time constraints.  It's probably going to be a little while before I have any boards in hand that are of my own design.  So to scratch my soldering itch I ordered some very cheap kits off eBay a little while ago.  No other motive in mind with these other than just a bit of fun putting them together.  There are kits available locally, but none of them are this cheap.  If they were things I might put to use I'd be happy to spend a little more, but as these will probably just got on my shelf of bits - the cheaper the better.  The first two were through hole clock kits, $5.60 and $7.70 respectively (including postage).   They actually arrived quite quickly.   Kit no. 1:

Both kits have a PDF with schematic and contraction details.  The PDF is not required for construction though thanks to the detail on the silk screen.  Kit no. 2:

Both kits are based on an Atmel AT89C2051, have a buzzer, various alarm/chime functions and buttons with a menu system for settings.  The second kit also has count down and stop watch functions.  No battery backup on either.  The first kit is 5V and comes with a USB power lead, the second is anywhere from 7V to 12V and comes with a short power lead with a plug on one end.

Both went together without issue and worked first go.  Not rocket science at all - but I still enjoyed building them both.

The other purchase I made was some SMD practice kits that were $1 each plus $0.99 postage.  I have done a little bit of SMD work now, but not a lot by any stretch.  I wanted to do a bit more SMD work and this seemed like the cheapest way to do it.  They are purely just practice boards with no function at all:

The quality of the board in these kits is not amazing - but its good enough to practice with.  I bought a few - but have only soldered one so far.  Good practice.

Commodore 128 - Part 1

I bought something off eBay about 2 weeks ago now and it arrived yesterday:

As the title suggests - a Commodore 128.  I've wanted a Commodore 64 or 128 for a long time but they always seemed quite expensive on eBay.  This one is a complete gamble.  Sold as is - untested - due to there being no power supply and video cable to test with.  It was however priced appropriately for a change and in Australia.  So I rolled the dice.  The auction included just the unit itself and a copy of the Commodore 128 System Guide:

The outside cover is a bit poor but the inside pages are like new.  Interesting British stamp printed on the front.  I also found a note inside the guide on a bit of paper about how to load a program from disk.  What was interesting about this hand written note was the paper the note was written on:

Check out that offer expiry!

I have a power supply and video cable en route which I'll put some more info up on when they arrive - as well as seeing whether the unit powers up and works or not.

The first thing I did when this guy arrived was open it up.  To my untrained eyes it looks as though it hasn't been opened since it was built in the mid 80's.  Even the heat grease on the main ICs that touch the large metal shield to act as a heat sink looked original.  Apart from the yellowing of the top half of the case it is in pretty amazing physical condition inside and out.  There was just a little bit of dust on the inside (not bad for a life of just under 30 years).  Nothing a quick clean couldn't fix:

I've got a few other things I'd like to buy/try and build for this guy but am holding off doing anything else until the power supply and video cable arrive to see whether this bad boy works or not.  Fingers crossed!

New Bench Supply

I finally pulled the trigger and bought a bench power supply.  Up until this point I have just been using the ATX breakout board.  I've been looking around for a while, trying to way up price vs features and finally settled on the Siglent SPD3303D.

The supply has 2 fully adjustable channels that can work in independent, series and parallel modes.  The 3rd channel switches between 2.5v/3.3v/5v at 3A.  The display only covers the first 2 channels, the 3rd channel just has the LED displaying CV/CC mode.  The 2 adjustable channels also have a timer function which can be programmed either on the supply itself or via USB using the included software (just like other features).

I settled on this guy because it ticked all my boxes - and physically was the right size for my shelf as well.

There is quite a lengthy thread on this particular supply here on the EEVBlog forum.  This thread mentions two issues found with the supply - namely:

1. Spikes / overshoots on CH3 during power on / power off of the entire supply
2. Spikes when the output of CH3 when a load is removed

If you read the entire thread on the EEVBlog forum you'll notice that both Siglent and the Australian supplier mention that there was a revision of the hardware late last year (2013) which fixed these issues.  As posted by Siglent Support the hardware version number has not changed. The supplier confirmed that the version I had ordered was this latest revision. 

So the first thing I did was some testing when the supply showed up.  The results I got show that there is some level of fix that has been applied to the 3rd channel.  Cold start of the whole unit with a load resistor on the 3rd channel:

There is still a spike there.  Removing the load from the 3rd channel:

Again still a small spike.

Both of these results are less that the original issues picked up with the hardware revision from last year.  And they only occur on the 3rd channel - which I personally don't have a huge issue with.  The first two channels don't have any of these issues. 

I also wanted to note that the service I received from TRIO Test & Measurement was awesome.  If you are in Australia check them out!

Getting started with CPLDs

Just like playing with micros I wanted to get my hands on and play with come CPLDs so I went hunting around for some tutorials.  The one I settled on to start with was this one.  It uses a CPLD from the Altera MAX II family.  The chip itself is not overly important but the branding on the chip is - as this is basically an exercise in learning how to use a the associated vendors software.  I found the same boards used in the tutorial here, ordered and waited for delivery.

Once the boards turned up I opened one up and noticed that the board was not very clean.  There was a lot of extra flux and other gunk on the board.  It worked - it just wasn't clean, so I thought I'd have  go at cleaning it.

After googling around I found a lot of people using a lot of different ways of cleaning their boards.  At lot of people talked about just washing it with some warm soapy water, or even putting it through the dishwasher.  I found this completely weird, I've never thought water and electronics should mix.  Given these boards were so cheap though I thought it would be better to try it with one of these cheap dev boards and kill them than try it for the first time with something else.

So I washed it with some warm soapy water and a toothbrush, and it came up really well.  To make sure it dried out properly I thought I'd put it in the over for a minute.  The oven was off - after just having cooked lunch.  The board was in there for 5 minutes max - this being the result:

The board was dry, but notice the melted power connector and power button below it!  I hooked up my bench supply and shorted the switch and the board still worked!  So I headed down to my local Jaycar store, but unfortunately they had no stock of the DC barrel jacks or the non-momentary switches in the foot print I needed.  So I headed home and got on eBay and ordered some replacements.  While a lot cheaper, I also then had to wait another couple of weeks for the bits to arrive from overseas.  In the meantime I carefully removed the broken components:

The parts finally arrived (I also finally got the lamp for the bench that I'd been chasing):

Now that the board was clean (and complete) I cracked on with the Hackshed tutorials.  Starting with the "Hello world!" of electronics - blinking an LED:

The tutorials are a great way to get up and running with the Altera software very quickly.  I wouldn't say the software is complicated - its just not very intuitive for someone who has't used it before.  Bil Herd over at Hack a Day has started a series on Programable Logic which is also well worth checking out (and not just because it's Bil Herd doing it).

As Bil would say "Keep Hacking!".

Bare Bones ATMega168

I've been playing quite a bit with Arduino based stuff over the last while.  It's an awesome way to get started without a huge amount of low level knowledge of the micro required to get an LED to blink.  I wanted to try some bare bones bits though with some ATMega's as well.   I found a few different tutorials online that explain the gap that the Arduino bootloader fills.  One of the ones I liked the most is the Hack a Day series on AVR programming.

I ordered a few ATMega168's from Core Electronics.  I've been buying a bit from them lately.  They are here in Australia, with pretty good prices and very fast shipping.

As for the AVR programming hardware I decided to use my BusPirate (fully supported by AVRDude).
So I read through the tutorials and got to the section in Tutorial 02 where it's time to connect up and program the ATMega168.  Dangerous Prototypes have a full page on using the BusPirate as an AVR programmer.

There were two bits that I had to change to get the setup to work for me with my BusPirate.  Firstly I had to remove the connections to 5v and GND for pins AVCC (20) and AGND (22) to program the ATMega168 and have it work once programmed.  The other change was that I had to make sure the GND from the BusPirate was connected for it to program the micro.  I was initially trying to use an external 5v source with the GND and 5v probes from the BusPirate completely disconnected, but the programming failed consistently.  As soon as I connected the GND from the BusPirate everything worked flawlessly.  Once programmed I could just use my external 5v supply and watch the LED blink!

Before programming the ATMega168 I verified the connections were correct using this command to see if avrdude could read the device signature:

avrdude -c buspirate -P /dev/tty.usbserial-AXXXXXXX -p m168 -v

Once that successfully worked (that was the command I was using to track down the two issues mentioned previously) I used this command to actually program the ATMega168:

avrdude -c buspirate -P /dev/tty.usbserial-AXXXXXXX -p m168 -U flash:w:main.hex

Once the ATMega168 was programmed giving it 5v made the LED blink - SUCCESS!!!

I then went on and walked through part 3 and 4.  I managed to write most of the code talked about in part 4 with only a few pointers required.

Most of the content in these AVR tutorials was a lot easier to digest after doing the edx course that I previously mentioned.  

Ubuntu, MDADM, RAID5, LVM and a Few Things In-Between

Ok, so I've been a bit slack while on leave and let Scotty down by not posting up my speedometer. I swear I will, however, today is more about server administration than an electronic project.

The reason I am writing about this is that it took me a while and a lot of reading to get everything I needed to grow, repair, destroy and create (in that order oddly enough) a RAID5 software array with and overlaying Logical Volume.

A while back I purchased a HP micro server and with the assistance of a mate got it up and running with Ubuntu 12.04 LTS. The OS runs of a USB key inside the server, while the RAID5 array sits in the bay drives. I needed the help originally as I had no real idea about LVM ‘s (or for that matter any real working knowledge of a RAID) – so to begin with I had four 2TB hard disks, which gave me about 5.4TB of usable space on the array.

As with all things, I 'had' to tinker with my Micro Server. The plan (as suggested by said friend), install two more 2TB hard disks in the optical bay slot of the server and increase the RAID5. After all, the whole point of the LVM is to allow the administrator to easily increase/modify disk space.

So I purchased two new hard drives, a power cable splitter, an esata to sata cable as well as a straight sata cable (all 3Gb/s). Finally, I purchased a Nexus twin hard drive mount. The nexus is an awesome little bit of kit that mounts two 3.5” hard drives on top of each other and pushes out vibration dampeners to the side so that the drives mount into the 5.25” optical bay. 

So now the fun part – I will actually quote direct commands here as this was all done in CLI, thus I can state these all work in Ubuntu 12.04 LTS, having said that if you've stumbled upon this, read the whole bloody thing - for the sake of your own sanity. I will headline each actual task as best I can in the order they're listed above.

Growing the Array

This should be simple – and it is. It must be done a disk at a time but overall it’s very easy. I do not have hot-plug enabled (I have no idea why I don’t, I just decided I didn't want it, so didn't enable it in my BIOS). If you do have hot-plug then no need to power down for you.

So I had to power down the server, physically connect one of the new disks and power the server back up. All going well there should be a lovely new hard drive in the system. You should be able to spot your drive allocation by using:

sudo cat /proc/diskstats

If you don’t know, you should also find out the allocation of your existing array:

sudo cat /proc/mdstat

So the result for mdstat is that my array iss md0 (I knew this already, but for the sake of someone reading who doesn't know that's how it's done). Diskstats told me that sde was my new disk.

First things first, let’s partition the new disk:

sudo fdisk /dev/sde

n – for a new partition

p – for a primary partition (followed by default boundaries)

t – to change partition type

1 – to select partition to change (you should only have one partition in this example)

fd – as the partition type (Linux RAID auto detect)

p – to view what your about to write to the disk matches the above

w – to write it to the disk!

Congratulations you've now partitioned the disk. Let’s add it to the existing array:

sudo mdadm --add /dev/md0 /dev/sde1

So now your array begins to grow. It can be a long process, an array the size of mine took around 24 – 30 hours (I can’t tell you exactly as the speed fluctuates and I didn't hang around waiting for it), later I’ll explain a couple of little tricks to speed this process up, some are common, some are not. 

Now this is where my problems began, I didn't really read the output of ‘p’ in fdisk and hadn't partitioned the disk properly. This wasn't a major deal, I just now had sda1, sdb1, sdc1, sdd1 and sde. My OCD kicked in here – I want them all to be partitioned the same blah blah blah (it honestly doesn't matter, it really is just internal aesthetics but oh well). 

Removing a Device from the Array

Now I’m not actually shrinking my array here, I just wanted to remove my disk and re-partition it before re-adding it to the array. Should be simple right? 

It’s slow, it’s painful and here is where it all came crashing down. In saying that, the commands below are correct, it was just the idiot telling the computer what to do that should of double checked his own work. 

First I failed the disk out and removed it:

sudo mdadm --manage /dev/md0 --fail /dev/sde

sudo mdadm --manage /dev/md0 --remove /dev/sde

Now at this stage I rebooted, re-partitioned my disk (as previously done) and re-added it:

sudo mdadm --manage /dev/md0 --add /dev/sde1

This was my undoing as the system re-allocated the disk names on reboot and I didn't check – I (somehow) partitioned an existing, active RAID disk, destroying the superblock in the process. So now I have a RAID5 with two failed disks. What do we know about RAID5 kids – two failed disks means an unrecoverable array - $#&*&!

After some googling and some luck (and through the lack of detail here you can probably tell I’m not exactly sure how), I managed to get the RAID to commence recovering. Recovery takes a lot longer than creating or growing, it’s a lot more work for the drives, as in this case, the missing disk was effectively from the middle of the array, needless to say the heads were spinning feverishly. 

The original estimate on the recovery was 4.2 days (7000 odd minutes) and it came pretty close to this. When it finished I was disappointed to see it didn't save everything, the damage was done and I had lost some data. Thankfully I keep a back-up of my data, the most recent being a month earlier, by combining what I could save from the array and comparing it to my backup I found I was missing around 18GB, out of 2.1TB of data. 

This I can live with as it is not mission critical stuff and easily replaced. So now what to do, the array is repaired but clearly not in a great state – I WANT A FRESH START!

Destroying a MDADM Array and LV

First thing first, this is how to destroy a RAID5 array, so be sure you want to do it – there is no coming back from this once it’s done!

In my case, my array is mounted on boot through fstab so I had to disable it in fstab and reboot (I find it easier this way as there aren't remnants hiding in a system process stopping the array from being shutdown). So rebooting my server without fstab mounting the array and binding my network share to it meant that I could deactivate everything.

First we need to deactivate the Logical Volume and Volume Group:

sudo lvdisplay – get the name of the volume group (Array in this instance).

sudo lvchange –a n Array – this deactivates the Volume Group

sudo lvremove Array – removes Volume Group

An example of lvdisplay from a previous setup:

sudo vgdisplay – get the name of the volume group (RAID5 in this instance).

sudo vgchange –a n RAID5 – this deactivates the Volume Group

sudo vgremove RAID5 – removes Volume Group

So now we need to stop the array:

sudo mdadm --stop /dev/md0

Zero the superblocks:

sudo mdadm --zero-superblock /dev/sda1 

(repeat for all disks, one at a time sda1 – sdx)

Remove the array:

sudo mdadm --remove /dev/md0

Finally, I did this to make sure the partition table was gone too:

sudo shred –n 1 /dev/sda  (repeat for all disks, one at a time sda – sdx)

It should be noted I’d let the above run for five minutes or so and then cancel it (Ctrl – C), we only need to destroy the data on the front of the disk so we can create our new array without mdadm complaining (it’ll destroy the rest of the data anyway, also shredding even a single pass on a 2TB disk will take a while and there’s no real need unless you want to be painfully security conscious).

Creating a RAID5 Software Array and LV

So now it’s time to make our new array (thank Christ!), this is not as slow as recovering or even growing an existing array. All up creating the (now) 10TB array (9.3TB usable), took around six hours. Creating the LVM over the top is less than fifteen minutes.

So first, let’s partition our disks (properly this time – take it from me, check your work):

sudo fdisk /dev/sda

n – for a new partition

p – for a primary partition (followed by default boundaries)

t – to change partition type

1 – to select partition to change (you should only have one partition in this example)

fd – as the partition type (Linux RAID auto detect)

p – to view what your about to write to the disk matches the above

w – to write it to the disk!

Complete the above for all disks to be used in the array, in my case it was sda through sdf.

Now let’s create our new array:

sudo mdadm --create /dev/md0 --level=5 –n 6 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

This is the long part as now the system will start creating the array. Level as referenced above refers to the RAID type, 'n' is the number of devices (six hard disks). A spare drive can also be added to assist in automating things when there's a failure, though this does not interest me. 

Fast forward six hours and my array is complete (at this point I was also quite drunk after six cans of Guinness). Let’s update the mdadm configuration file and propagate our new array through the system before we do anything (kernel included):

sudo mdadm --detail –-scan 

We need the UUID of our mdadm array (don’t confuse this with the UUID obtained later) and a few other details that were just spat out at you (see below for what you need). Right now though we need to edit the mdadm.conf file (it’s located at /etc/mdadm/mdadm.conf). Below ‘#definitions of existing arrays’, add the following from the above output (delete any existing entry here if you removed your old array too):

ARRAY /dev/md0 metadata=x.x name=xxxx:x UUID=xxx:xxx:xxx:xxx

Where the x’s are the data spat out from the above command (you’re probably wise to copy and paste the output TBH).

Propagate the changes through the system:

sudo update-initframs -u

The above is especially important if you've removed a previous array and are replacing it. Anyway time to get our LVM on.

Let’s create the physical volume:

sudo pvcreate /dev/md0

   

Now the Volume Group:

sudo vgcreate RAID5 /dev/md0

RAID5 is the name I chose for my Volume Group, I kept it the same as the one my friend helped me make originally so it was easier for me to follow.

To create the Logical Volume we need to know the total size of our Volume Group:

sudo vgdisplay RAID5 | grep “Total PE”

The result in my case was ‘2384496’. Now let’s make the LV:

sudo lvcreate –l 2384496 –m Array RAID5 /dev/md0

Where ‘Array’ is the name of my LV and ‘RAID5’ the name of my previously created Volume Group.

Finally, we need a usable file system, in my case ext4:

sudo lvdisplay – get the full path of our LV

sudo mkfs.ext4 /dev/RAID5/Array

A short time later the process will complete and you’ll have a software RAID5 with a ext4 Logical Volume.

I added mine back to fstab so it would auto mount, first you need the UUID of the array:

sudo blkid

Here we’re looking for the UUID of our new array /dev/mapper/RAI5-Array in my case (note the VG and LV name use here, that’s how you’ll know you've got the right one).

Finally, in fstab itself add a line similar to the below (you’ll have to be root to edit fstab BTW):

UUID=xxx-xxx-xxx-xxx-xxx /my/mount-point ext4 defaults  1    1   

Where the UUID is the UUID from blkid, /my/mount-point is where you are going to mount your new LVM (an example might be /opt/myarray), ext4 is the file system in use and the rest you don’t need to worry about unless you’re an advanced user (in which case you’re probably not reading this). The spacing between the mount point, ext4, default values etc should be a TAB spacing.

Reboot and you’re done.

Speeding up Growth, Recovery and Creation

If you've kept reading to this point good for you! Here’s some tips for getting better disk speeds when dealing with growing, recovering or creating an array. Some of these can be altered while the process is going, however things like bit mapping cannot and must be done beforehand. There’s some pros and cons and I’ll touch on them a little where applicable.

To give some ballpark figures from my system (these will be different based on the machine used):

Growing the array, 15 – 20 MB/s

Recovering the array, 5 – 7 MB/S peaking briefly at times to 20MB/s

Creating the array, 100MB/s (this dropped to 60MB/s at the end)

To monitor the systems progress during any of the above:

cat /proc/mdstat

or

sudo mdadm --detail /dev/md0

From here on in I did have some issues with ‘sudo’. From what I've been told it sounds like a re-direct issue, I couldn't execute these commands (permission denied error). Logging in as root however fixed this (sudo su). Just something to keep in mind.

Minimum and maximum disk speeds (can be changed during a running process):

sudo echo 20000 > /proc/sys/dev/raid/speed_limit_min

sudo echo 200000 > /proc/sys/dev/raid/speed_limit_max

This sets the minimum speed to 20MB/s and the maximum to 200MB/s. From experience this doesn't mean the system will hold these speeds if it can’t, so adjust the according to what you’re seeing from cat /proc/mdstat (use some common sense basically).

Stripe buffer caching (can be changed during a running process):

This is a bit like reading ahead, it accepts values from 16 to 32768 (32MB). There were a lot of warnings about this when I was reading about it, basically setting the value too high had caused swap storms and the like - too much memory (and/or virtual memory) had been used for the process is the short version. 

I took into account the age of the posts (they were reasonably old) and monitored the usage closely. I set the stripe buffer to 32MB and noticed an increase around the fifteen percent mark in RAM usage. I have 8GB of RAM and a 3.7GB swap partition, at this point I decided I was safe to leave it there. The default value is 256, obviously the reference to md0 below relates directly to the fact md0 is my array:

sudo echo 32768 > /sys/block/md0/md/stripe_cache_size

Bit-mapping (can only be enabled prior to the process commencing):

Now I have to admit I have no experience with this one, but if it prevents you sitting there for 4.2 days then it’s worth it. Bit-mapping reportedly improves recovery times after crashing as well as growth times, it does however gradually degrade your arrays performance if you do not disable it at the completion of the process (it should also be noted this is only relevant for certain RAID configurations of which RAID5 is one).

Enabled:

sudo mdadm --grow --bitmap=internal /dev/md0

Disabled:

sudo mdadm --grow --bitmap=none /dev/md0

Conclusion

Well that’s it folks. My system is now back up and running as it was before I broke it (ya!) and I hope to never have to do this again. Hopefully this saves someone from having as many browser windows open as I did when trying to learn all this stuff.

Here’s a picture of my hacked up server with its two 3.5” drives secreted in the optical bay slot – enjoy!