I faced a similar situation this past summer. Our beloved German Shepard, Brandy, whom was 7 years old suddenly get very sick, for about 2 weeks we battled with what we thought was worms, then a minor infection. They day after the 4th of July she suddenly just would not move, eat, or do anything. We rushed her to the vet and he determined that she had a tumor and it had ruptured and there was nothing we could do. We had to put her down. It was honestly the most awful thing I've ever had to do ( I still cry about it today.) This dog was my first child in my eyes.
Coming home to my children without her was the 2nd worst thing I've ever experienced. My daughter, who was only 1.5, didn't know much else besides, "Where's Brandy?" but my son, who is 4, and was extremely attached to Brandy had more questions. At the time we opted for a vague but slight truth of, "Brandy had a very bad bellyache, the doctor couldn't make her feel better and she is not coming home." It worked well, not many questions about where she had gone, but just mentions that my son missed her and wanted her back.
We have a new dog now and it seems to help. She is nowhere as good as a dog as what Brandy was, but the mentions of Brandy have decreased to almost none. It pains me to see her forgotten but I know she's in a better place now and I know my son has been able to process the loss with the new dog easier.

We have lost a dog--but I explained that she was sick and very old and went to heaven.  However, this was just weeks before my Grandmother passed away from a very brave, but difficult battle with cancer--so we were preparing my boys for what was to come.  So--I think it is different--we had already explained that sometimes dr's can't fix boo-boo's and they have to go to heaven.  My oldest son knows this--and asks about my Grandma and says she is an angel watching over us.  It is definitely a hard thing to talk about.  We believe in Heaven, so it makes things a little bit easier!  I however, know there are several good books out there for young children on the loss of a pet!!  Might be worth a look!

Our labrador died this past November. It was very difficult for all of us, she had been in our family for 7 years. We weren't sure if it was the rabies shot (she had gotten it the previous day and really wasn't herself at all) or if it was a seizure. She had them occasionally. I think it was a seizure that caused her to have a stroke but if it was related to the shot, I can't say. She died after the kids were in bed and we called a sitter to stay with them so we could bury her. We just decided that honesty was the best policy. My son won't remember her, he was only 18 months old. But my daughter is 3 and she's a very mature 3. We told her that Kisses (the dog) had gotten sick during the night and she died. We told her that it was a sickness in the dog's head and we weren't able to fix it. We also assured her that it was something that only dogs get, not people. Of course, that was a little lie, since we all know that people do have seizures and strokes that kill them. But we didn't want her to worry about "getting sick" too and we can explain that there are differences when she's older. She was very upset about it and kept asking for us to take the dog to the doctor to get better. It took a while for her to understand that the dog wasn't coming home. We also had to reassure her that it wasn't her fault, that it wasn't something that she did. She kept saying "maybe if I had petted her more...". It really was just as heartbreaking to see her struggle to understand as it was to lose the dog. But we didn't want to tell her that Kisses went to a farm to live because then she would be hurt that the dog didn't want to live with her anymore and wouldn't understand why we couldn't go visit. It's been a few months now and she still talks about how her dog died and how that makes her feel. But she understands that death means not coming back. I'm not really sure if she entirely understands the actual concept of death but as far as a 3 year old goes, that's about all the understanding they need. 

I think you handled it well. :) Much better than I would have. I don't know how I'd explain death to my kids.

Last Spring  we had to put our old boy down... he was 13, and a wonderful dog.
We had gotten a new puppy in January, and our house was very busy.. and we would talk to our son about how Yeager was getting very old, and very tired. He was at daycare one day when the dog got really sick (I will spare details) and I had to call my husband home from work. He took the dog to the vet, and we let him go. It was a very sad day.. i still miss that dog.. he was goofy, but a Dachshund/Black lab cross and such a sweetheart.
Our son (who was 3 and half at the time) asked where Yeager was when he came home, and I decided to be honest with him. I told him that Yeager had been very old, and very sick, and tired. And daddy took him to the vet who helped him go to doggy heaven. And there he would be young and happy again.
For us, it was the best answer we could have given him.

I'll admit that I don't have kids, but I've thought about this very thing often.

My opinion is that you should always tell your kids the truth. They have to live in the real world. Not being acquainted with reality isn't doing them any favors.

My father-in-law passed away this past January.  All three of our kids (ages 3, 5, &7) attended the wake and the funeral.  Our oldest was the only one who truly understood and felt sorrow.  Our 5 year old was curious, but not emotional.  And our youngest, I think felt a little closure being able to physically see where his papa was.  If you ask him, to this day, he'll tell you he's "sleeping in his bed"..  Our oldest will tell you he's up in heaven, giving the other angels hair cuts (he owned a barbershop).  Our middle child will tell you he's underground now, in his bed.  And the youngest will say he's sleeping.  Death is a part of life.  There's no need to shelter kids from it because honestly they can only grasp so much of it anyways, and I think that the bit that they do grasp at a young age will help them understand more clearly as they get older.

Back in the day kids read the Wizard of Oz with all the mutilation the movie left out, and creepy-ass fairy tales. 

I don't think any age is too young. If the child is old enough to miss the pet, or person, they can process the facts if given to them age appropriately.