RDnet Hosting System Status - 5/9/2011
All services are up and running, with no issues to report at this time…
telfair.vsh.dc - Outage (4/30/10)
telfair.vsh.dc - Outage (4/30/10)
—————————————————————————-
Users Affected:
All clients hosted on server Telfair (telfair.vsh.dc)
Subject:
RDnet Hosting - Outage
Status:
Incident reported via system monitoring process. (1:08 pm)
Managed Services Team Received. (1:09pm)
Incident assigned Level 1 (1:09pm)
Under investigation. (1:10pm)
Service Restored (1:12pm)
Incident resolved (1:12pm)
Update posted (1:18pm)
What will/has happened:
Server Telfair just alerted to a down status. We will have more information shortly…
Additional Information:
N/A
Update:
After further investigation, it was determined that the web server (httpd) service failed unexpectedly. It restarted automatically, and all websites came back up successfully.
—————————————————————————-
telfair.vsh.dc - HTTP, IMAP, SMTP, PING Downtime (9/04/10)
—————————————————————————-
Users Affected:
All clients hosted on server Telfair (telfair.vsh.dc)
Subject:
RDnet Hosting - HTTP, IMAP, SMTP, PING downtime
Status:
Incident reported via system monitoring process. (6:12 pm)
Managed Services Team Received. (6:15pm)
Incident assigned Level 1 (6:15pm)
Under investigation. (6:15pm)
Incident assigned Level 2 (6:45pm)
Update posted. (6:45pm)
Incident assigned Level 3 (7:00pm)
Update posted. (7:16pm)
Update posted. (8:09pm)
Update posted. (8:19pm)
Incident assigned Level 1 (8:22pm)
Incident resolved (8:22pm)
Update posted (8:48pm)
What will/has happened:
We have been made aware that our server Telfair is experiencing some downtime with the following processes: HTTP, IMAP, SMTP, PING. Our team, and our data center are currently investigating.
Additional Information:
N/A
Update 1:
We have been informed that there are several more servers that do not belong to us which are having these issues right now, leading us to believe that the current problem is on the datacenter side.
Update 2:
Our datacenter has confirmed that while this is not a complete outage, several of their customers are experiencing issues at this time. That includes us.
Update 3:
Our datacenter is saying that the issue is related to faulty router interconnectivity on part of their network, and an early ETA on resolving the problem completely is approximately 30 minutes. They have all of their engineers on deck working at this time. Hardware is being swapped out, with more updates to follow.
Update 4:
Our monitoring service has informed us that all systems are now back online. We will continue to monitor closely to confirm.
Update 5:
Official “post mortem” from our datacenter: “The degradation this evening was caused when one of the cards in one of our core routers died. This particular card handles connectivity between several of our core routers. The loss of this card and, by extension, the loss of normal levels of bandwidth across our core, caused certain customers to lose their ability to see both inbound and outbound paths to and from all Internet destinations. This problem did not affect all paths, but enough that many customers noticed degradation. To permanently fix this problem, our engineers have moved all interconnects off of this card, and permanently removed the card from our network.”
—————————————————————————-
telfair.vsh.dc - Email Connection Issues (8/18/10)
—————————————————————————-
Users Affected:
All clients hosted on server Telfair (telfair.vsh.dc)
Subject:
RDnet Hosting - email connection issues
Status:
Incident reported via client. (6:45 am)
Managed Services Team Received. (6:45am)
Incident assigned Level 1 (6:45am)
Under investigation. (7:00am)
Update posted (7:10am)
Update posted (12:00pm)
Incident assigned Level 2 (12:45pm)
Incident resolved (1:05pm)
Report Finalized - AR (5:56pm)
What will/has happened:
We have just been made aware of an issue in which clients trying to use webmail or a third party email program to retrieve email are finding their connection to the server is terminated or refused. All email processes are running at this time and did not go down, and it is believed that the issue may be related to SSL. We are currently investigating.
Additional Information:
N/A
UPDATE 1 (8/18/10 - 7:10am):
At this time, webmail is still accessible. We are still conducting an investigation in regards to the disconnection our clients are experiencing when using a third party email program to retrieve messages. - posted by Managed Support Services
UPDATE 2 (8/18/10 - 12:00pm):
At this time, we are experiencing a problem with Comcast in which we are unable to access external websites. This has delayed our investigation of the email disconnection issue until Comcast can resolve this new problem. - posted by Managed Support Services
UPDATE 3 (8/18/10 - 1:05pm):
At this time, all problems connecting to the email have been fixed. The main culprit was a corrupted .ca bundled on one of our SSL certificates. Webmail was back up shortly after the original incident was reported, and clients experiencing the disconnection issue were advised to use webmail until we could resolve the SSL issue.
—————————————————————————-
telfair.vsh.dc - Brief MySQL/FTP Downtime (8/16/10)
—————————————————————————-
Users Affected:
All clients hosted on server Telfair (telfair.vsh.dc)
Subject:
RDnet Hosting - Brief MySQL/FTP downtime today.
Reason:
Security updates needed to be applied.
What will/has happened:
New security rules were applied to the MySQL and FTP processes on Telfair and this required a reboot of those services. Downtime was around a minute for these processes, and did not impact anything else. (such as normal web traffic)
Implementation Date:
N/A
Additional Information:
N/A
—————————————————————————-
