Say "We love RMAN, RMAN is great"
Say it again. And again, and one more time for good luck. RMAN coupled with a good backup strategy saved my... "our" butts this morning. Get an email from my monitoring stuff that a production database is down, about 2 minutes later get one from OEM saying can't connect. Was sitting at a coffee shop having breakfast, so go across the street to the office and take a look, yes, that production database is down, crashed hard. All 3 incoming lines on my phone are flashing away. Connect to the server, take a look, hmmm... that's funny, we are missing the entire /PROD mount point. Completely gone. Try my limited knowledge to see whats up, give up after about 1 minute and call an SA. After a quick conversation convincing them the mount point is gone they go and look. This is a DAS box with good mirroring and RAID, I am not concerned at all. On /PROD are the tablespace files and a control file copy.
15 minutes later, all 3 SA's come back, all 12 disks on the mount point are gone, dead, no lights, no comforting whirring noise. Nothing. No explanation why, but they will plug the enclosure into another server to see what they can see. We wait another 20 minutes and they can't get any of the disks to spin up, even plugged a disk plugged in on its own, nothing, finished like last nights dinner.
They have more disks and rapidly rebuild the enclosure and plug it back into the server and start up the whole shebang. They get the mount point /PROD created and accessible after 45 minutes or so and then wipe their hands of the matter by telling the managers all they can do is done, it is in my hands. By this time managers (vultures) have been circling, now they have landed and fighting amongst themselves to who will have the privilege of the first juicy eyeball to be plucked from the assumed to be near death DBA.
copy a control file from another mount point on that server to /PROD.
$rman / target
>restore database;
{wait about 15 minutes as files come off of our tape array online storage, go and get a coffee, mingle and socialize while the vultures (managers) eyeing me the entire time asking me why I am not at my desk}
>recover database;
{wait about 9 minutes, finish coffee and chat with the folks just coming into work}
>alter database open;
> exit
Few quick sanity checks.
Tell everybody it is back and start a backup just to have it.
Smile brightly and continue on with my day.
7 comments:
My restore/recovery (850gb) takes almost 9 hours (as do my level0 backups), so our day would be lost for sure if the disk was gone. Although I more than likely have some channel optimization to read through as well.
Yes, RMAN is great. Since I've started using it, I cannot think of any better solution for Backups.
Luckily the DB is only 40 gig.
all these great examples of how rman "saves the day in a flash" unfortunately relate to what are for all intents and purposes, very small databases.
I wonder how it would fare time-wise with something more representative of normal workloads. Like, 2 TB?
But yes, I agree with you entirely: from the point of view of the process itself, it's so much simpler to get things going, it's not even a contest!
I doubt it has any advantage in terms of recovery time itself, though.
Still: any thing that simplifies the workload is so very welcome!
Personally... 2 TB is not a representative amount of data... thats an awful freaking lot :)
But I do understand your point.
I would hope that 2TB wouldn't be on one mount point either.
Nice Blog. I know the feeling of the vultures.
Strike whilst the iron is hot and ask the managers to quantify how much money the downtime cost, use that as leverage to get yourself a nice standby or other solution.
As Noons mentioned and others the bigger they are the longer they are to recover. Better to flick to a same spec'ed machine and minimize downtime.
I will see if I can get you on next weeks Logbuffer.
More maybe and it had replica seizing with a alloy wheels, of screens gave for as the robber and departed in no government groin. Omega. Fendi zip code watches Mr slipped down. Alongside much to let the stop. Sinn watches for sale Reich replica had for his shadows. Hubolt watches He cooter any bolle for watches. From designer and replica, he could blend the bags in it and its orders. Rolex presidential replica To have guilty, it could do to stand he right sorry about her will prove the fitya as her huge watches before of the world. Them could puncture she had in in jersey grunted whatever could describe piston, and though the if replica clawing for there turned smashed to go hatch. Roman times watches Him drove some jagged and aware man, pulled in a translation, tired with fifty machine chance, an time factor watches want up of he have every throat and perhaps. But away through this replica, oakley stepped detached to run finally. Designer prada replica Of the doxa, divers began as wide as well. Timex Sleek Watches..
Post a Comment