|
Dear Race/Network Subscribers:
Believe me the last thing we wish to be doing is sending out a letter of apology. Not because we are not genuinely apologetic, but because it means there was a failure at some level that we need to address to you.
Last night was the perfect storm. We had slight picture issues nights 1 and 2 of the 360 Nationals due to electrical interference/surge. This was finally diagnosed back to a short in a common (shared) cable. This is the reason why when we swapped out cabling for new it did not resolve it initially, but for Saturday night we were crisp for the picture. To show people what we are doing, we have put out free previews during the Hot Lap sessions and we felt this was important to do, especially when there was a slight picture issue the night before. Then we put a screenshot out of the video with all the cars in the infield – well that picture was worth a 1000 words. Actually it was worth 1000s and 1000s of people.
We knew all along the 360 Nationals were going to be a popular event, but the traffic to the site was overwhelming during last night’s free preview. Our server logs show over 8000 requests and with this all pretty much happening at the exact same time, the database server locked up processing all the simultaneous requests.
This did not actually affect the video stream as that is served elsewhere with different servers. But when we asked people to refresh the page due to the free preview ending, that caused all to lose backend access with the database server locked up – even though the stream was still running.
We knew what happened and called our data center to do a manually reboot of the database server figuring that reboot would bring everything back within 10 minutes. Of course that did not happen as the lock up of the database caused errors and a full disk repair had to be performed. What we then did since the stream was running and the index page was still reachable is that we emailed all that contacted us and told them to go to our index page and we put the stream out there. The backend was back up in time for 410 A main and the B and A 360 mains, if people did not go to the index page.
Now that we have been forthright with everything that occurred, we also know that means nothing to the paying customer. We just felt all deserved a true, honest explanation of what happened.
What will be done… 2 options for those tuned in last night:
1. Refund or
2. Free credit to tonight’s Tournament of Champions PLUS Free credit to complete 4-Day On-Demand rebroadcast
(* for those who purchased the 4 day package we will work out a credit/refund that will be emailed out separately)
Please email us the option you will be choosing, the quicker the better if you are going to opt for tonight’s broadcast. Refunds will be issued next week. Please send all correspondence to race/network and not Knoxville Raceway.
Sincerely,
Ian Marvuglio
President – co-Founder
Race/Network
PS For those asking why I wasn’t on the boards as the trouble hit, that is because I was knee deep in server issues with my partner and I was in 2 phone conversations, 2 chat sessions and txt messaging over trying to get the servers up again + tons of customer emails. I apologize I was unable to break away and hit all the boards.
|
