Webserver C

SummaryWebserver is currently offline
ImpactDegraded Service
Current statusProblem identified
Open timeJune 26, 2018, 7:46PM UTC (GMT +00:00)

History

June 26, 2018, 7:47PM UTC[Problem identified] Hard drive failure detected. Replacement drives ordered.
June 26, 2018, 7:46PM UTC[Investigating] Currently investigating

Email System

SummaryHard drive failure
ImpactDowntime
Current statusResolved
Open timeMay 3, 2017, 2:56PM UTC (GMT +00:00)
Close time26 June 2018 19:46

History

June 26, 2018, 7:46PM UTC[Resolved] Hardware replaced
May 3, 2017, 2:57PM UTC[Problem identified] We've identified the bad drive. Purchasing a new one now
May 3, 2017, 2:56PM UTC[Investigating]

Server A

SummaryScheduled reboot
ImpactPlanned Maintenance
Current statusResolved
Open timeMay 3, 2017, 2:55PM UTC (GMT +00:00)
Close time03 May 2017 14:57

History

May 3, 2017, 2:57PM UTC[Resolved] reboot complete
May 3, 2017, 2:55PM UTC[Monitoring]

Webserver C

SummaryThe webserver is serving pages slowly
ImpactDegraded Service
Current statusResolved
Open timeMay 3, 2017, 1:54PM UTC (GMT +00:00)
Close time26 June 2018 19:45

History

June 26, 2018, 7:45PM UTC[Resolved] Maintenance complete
May 3, 2017, 2:02PM UTC[Problem identified] Looks like a memory issue
May 3, 2017, 1:54PM UTC[Investigating]

Voicemail

SummaryVoicemail system is offline
ImpactDowntime
Current statusResolved
Open timeMay 11, 2016, 1:50PM UTC (GMT +00:00)
Close time03 May 2017 13:53

History

May 3, 2017, 1:53PM UTC[Resolved]
May 11, 2016, 1:51PM UTC[Problem identified] There appears to be a failed hard disk
May 11, 2016, 1:50PM UTC[Investigating] We are currently looking into this issue

Server A

SummaryPower supply failure
ImpactDowntime
Current statusResolved
Open timeMay 13, 2015, 12:07AM UTC (GMT +00:00)
Close time11 May 2016 13:50

History

May 11, 2016, 1:50PM UTC[Resolved] Resolved
May 13, 2015, 12:46AM UTC[Problem identified] Power supply is installed
May 13, 2015, 12:43AM UTC[Problem identified] We've received the new power supply
May 13, 2015, 12:27AM UTC[Problem identified] New power supply ordered
May 13, 2015, 12:07AM UTC[Problem identified] Server need's it's power supply replaced.

Server A

SummaryPower supply failure
ImpactDowntime
Current statusResolved
Open timeMay 13, 2015, 12:07AM UTC (GMT +00:00)
Close time13 May 2015 00:25

History

May 13, 2015, 12:25AM UTC[Resolved] One of the two failed power supplies have been replaced.
May 13, 2015, 12:07AM UTC[Problem identified] Server need's it's power supply replaced.

Voicemail

SummaryVoicemail offline
ImpactDowntime
Current statusResolved
Open timeMay 12, 2015, 3:26AM UTC (GMT +00:00)
Close time12 May 2015 03:44

History

May 12, 2015, 3:44AM UTC[Resolved] Server is back online
May 12, 2015, 3:38AM UTC[Monitoring] Power supply replaced
May 12, 2015, 3:37AM UTC[Problem identified] New power supply ordered
May 12, 2015, 3:32AM UTC[Problem identified] Power supply is dead
May 12, 2015, 3:26AM UTC[Investigating] Voicemail is currently offline

Webserver C

SummaryHigh packet loss
ImpactDegraded Service
Current statusResolved
Open timeMay 11, 2015, 11:35PM UTC (GMT +00:00)
Close time12 May 2015 03:21

History

May 12, 2015, 3:21AM UTC[Resolved] Issue resolved
May 12, 2015, 3:17AM UTC[Monitoring] Server is online and appears functional
May 12, 2015, 3:13AM UTC[Problem identified] Server is still booting up
May 12, 2015, 2:55AM UTC[Problem identified] Server is powered on, testing network connectivity.
May 12, 2015, 2:54AM UTC[Problem identified] Powering on the server now
May 12, 2015, 2:49AM UTC[Problem identified] Server has been connected to new switch
May 12, 2015, 2:44AM UTC[Problem identified] Switch wiring completed
May 12, 2015, 1:49AM UTC[Problem identified] Still connecting the new switch
May 12, 2015, 1:17AM UTC[Problem identified] half way done with cabling the new switch.
May 12, 2015, 1:08AM UTC[Problem identified] The switch has been racked, we are now connecting the cables
May 12, 2015, 1:06AM UTC[Problem identified] We are racking the new switch now
May 12, 2015, 1:03AM UTC[Problem identified] A new switch has been located.
May 12, 2015, 12:59AM UTC[Problem identified] All switchports are full. We are provisioning a new switch now.
May 12, 2015, 12:26AM UTC[Problem identified] We've located an open network port
May 12, 2015, 12:09AM UTC[Problem identified] Attempting to locate another open switchport
May 11, 2015, 11:42PM UTC[Problem identified] Appears to be a bad network switchport
May 11, 2015, 11:35PM UTC[Investigating] This server is experiencing high packet loss - network performance is degraded. Our technicians are currently working on it.

Server A

SummaryServer Offline
ImpactDowntime
Current statusResolved
Open timeMay 4, 2015, 6:50PM UTC (GMT +00:00)
Close time04 May 2015 19:34

History

May 4, 2015, 7:34PM UTC[Resolved] Power supply has been replaced. Server is online again.
May 4, 2015, 7:13PM UTC[Problem identified] employees have been dispatched to pickup the power supply.
May 4, 2015, 7:08PM UTC[Problem identified] We've located a power supply at a remote site that can be used as a replacement.
May 4, 2015, 7:03PM UTC[Problem identified] Power supply appears to be dead. Looking for replacement.
May 4, 2015, 6:50PM UTC[Investigating] The server has powered down again

Server A

SummaryHardware Failure
ImpactDowntime
Current statusResolved
Open timeMay 4, 2015, 6:40PM UTC (GMT +00:00)
Close time04 May 2015 18:46

History

May 4, 2015, 6:46PM UTC[Resolved] Power cable has been replaced. The server is back online.
May 4, 2015, 6:40PM UTC[Investigating] This server shutdown unexpectedly, and will not power on.

Webserver D

SummaryServer blocking outbound connections
ImpactDegraded Service
Current statusResolved
Open timeMarch 29, 2015, 1:02AM UTC (GMT +00:00)
Close time09 April 2015 02:04

History

April 9, 2015, 2:04AM UTC[Resolved] Problem resolved.
March 29, 2015, 1:02AM UTC[Investigating] User reports coming in

Voicemail

SummaryVoicemail offline
ImpactDowntime
Current statusResolved
Open timeMarch 15, 2015, 3:10AM UTC (GMT +00:00)
Close time29 March 2015 01:01

History

March 29, 2015, 1:01AM UTC[Resolved] Hardware replaced, issues resolved
March 15, 2015, 3:13AM UTC[Problem identified] Appears to be a hard disk failure
March 15, 2015, 3:10AM UTC[Investigating] Unsure at this time

Server B

SummaryWebserver slowness
ImpactDegraded Service
Current statusResolved
Open timeMarch 15, 2015, 1:45AM UTC (GMT +00:00)
Close time19 March 2015 01:58

History

March 19, 2015, 1:58AM UTC[Resolved] Hard disk has been replaced
March 15, 2015, 1:45AM UTC[Investigating] compaints from users that the website is not responsive

Server B

SummaryServer offline
ImpactDowntime
Current status
Open timeMarch 15, 2015, 1:21AM UTC (GMT +00:00)

History

Server A

SummaryHardware upgrade
ImpactPlanned Maintenance
Current statusResolved
Open timeMarch 2, 2015, 9:40PM UTC (GMT +00:00)
Close time02 March 2015 21:42

History

March 2, 2015, 9:42PM UTC[Resolved] issue resolved
March 2, 2015, 9:41PM UTC[Monitoring] one more update
March 2, 2015, 9:41PM UTC[Problem identified] another update
March 2, 2015, 9:40PM UTC[Investigating] Test of the system

Webserver C

SummaryNot reachable on network
ImpactDowntime
Current statusResolved
Open timeFebruary 18, 2015, 1:51AM UTC (GMT +00:00)
Close time18 February 2015 02:29

History

February 18, 2015, 2:29AM UTC[Resolved] Problem resolved
February 18, 2015, 2:26AM UTC[Problem identified] Problem has been isolated
February 18, 2015, 1:51AM UTC[Investigating] Another test

Telephone System

SummaryMemory usage 100%
ImpactDegraded Service
Current statusResolved
Open timeFebruary 18, 2015, 1:50AM UTC (GMT +00:00)
Close time02 March 2015 21:43

History

March 2, 2015, 9:43PM UTC[Resolved] service restored
February 18, 2015, 1:50AM UTC[Investigating] this is a test

Server B

SummaryServer down
ImpactDowntime
Current statusResolved
Open timeFebruary 10, 2015, 2:39AM UTC (GMT +00:00)
Close time11 February 2015 22:31

History

February 11, 2015, 10:31PM UTC[Resolved] All fixed
February 10, 2015, 2:39AM UTC[Investigating] Testing the outage

Server B

SummaryServer down
ImpactDowntime
Current statusResolved
Open timeFebruary 10, 2015, 2:06AM UTC (GMT +00:00)
Close time10 February 2015 02:07

History

February 10, 2015, 2:07AM UTC[Resolved] more testing
February 10, 2015, 2:07AM UTC[Monitoring] more testing
February 10, 2015, 2:07AM UTC[Problem identified] more testing
February 10, 2015, 2:06AM UTC[Investigating] more testing
February 10, 2015, 2:06AM UTC[Problem identified] more testing

Server B

SummaryFirewall blocking external requests
ImpactDegraded Service
Current statusMonitoring
Open timeFebruary 10, 2015, 2:01AM UTC (GMT +00:00)

History

February 10, 2015, 2:22AM UTC[Monitoring] lots more testing
February 10, 2015, 2:18AM UTC[Monitoring] more testing
February 10, 2015, 2:18AM UTC[Problem identified] more testing
February 10, 2015, 2:14AM UTC[Problem identified]
February 10, 2015, 2:14AM UTC[Monitoring] more testing
February 10, 2015, 2:08AM UTC[Monitoring] more testing
February 10, 2015, 2:05AM UTC[Monitoring] more testing
February 10, 2015, 2:05AM UTC[Monitoring] more testing
February 10, 2015, 2:03AM UTC[Monitoring] more testing
February 10, 2015, 2:02AM UTC[Monitoring] Test of the monitoring
February 10, 2015, 2:02AM UTC[Problem identified] Test of the problem identified
February 10, 2015, 2:01AM UTC[Investigating] This is a test

Webserver D

SummaryOffline
ImpactDowntime
Current statusResolved
Open timeFebruary 10, 2015, 1:17AM UTC (GMT +00:00)
Close time18 February 2015 01:50

History

February 18, 2015, 1:50AM UTC[Resolved] Finally fixed everything
February 10, 2015, 2:32AM UTC[Problem identified] more testing
February 10, 2015, 1:17AM UTC[Investigating] Memory error

Server A

SummaryOffline
ImpactDowntime
Current statusResolved
Open timeFebruary 7, 2015, 4:09AM UTC (GMT +00:00)
Close time10 February 2015 02:32

History

February 10, 2015, 2:32AM UTC[Resolved] more testing
February 10, 2015, 2:32AM UTC[Monitoring] more testing
February 10, 2015, 2:26AM UTC[Monitoring] getting there
February 10, 2015, 2:25AM UTC[Problem identified] getting there
February 10, 2015, 2:25AM UTC[Investigating] getting there
February 10, 2015, 1:28AM UTC[Monitoring] Hard disk successfully replaced
February 7, 2015, 4:09AM UTC[Problem identified] Hard disk failure

Voicemail

SummaryOffline
ImpactDowntime
Current statusResolved
Open timeFebruary 4, 2015, 3:39AM UTC (GMT +00:00)
Close time07 February 2015 04:22

History

February 7, 2015, 4:22AM UTC[Resolved] Voicemail Restored
February 4, 2015, 3:39AM UTC[Investigating] Voicemail is currently down

Server B

SummaryOffline
ImpactDowntime
Current statusResolved
Open timeFebruary 3, 2015, 2:42AM UTC (GMT +00:00)
Close time10 February 2015 02:01

History

February 10, 2015, 2:01AM UTC[Resolved] i think it's fixed
February 10, 2015, 2:00AM UTC[Monitoring] still checking
February 10, 2015, 1:59AM UTC[Monitoring] Power restored
February 10, 2015, 1:58AM UTC[Problem identified] Searching for the correct power plug
February 3, 2015, 2:44AM UTC[Problem identified] I think we need to power it on again
February 3, 2015, 2:42AM UTC[Investigating] I tripped over the power cord on this server too

Server A

SummaryOffline
ImpactDowntime
Current statusResolved
Open timeFebruary 3, 2015, 2:39AM UTC (GMT +00:00)
Close time03 February 2015 02:45

History

February 3, 2015, 2:45AM UTC[Resolved] Powered on!
February 3, 2015, 2:43AM UTC[Monitoring] I've plugged it back in again
February 3, 2015, 2:39AM UTC[Investigating] I tripped over the power cord

Server A

SummaryOffline
ImpactDowntime
Current statusResolved
Open timeFebruary 3, 2015, 1:14AM UTC (GMT +00:00)
Close time03 February 2015 02:27

History

February 3, 2015, 2:27AM UTC[Resolved] Everything looks good. Setting back to green
February 3, 2015, 1:15AM UTC[Monitoring] We've plugged it back in, and it appears online
February 3, 2015, 1:15AM UTC[Problem identified] It appears that the UPS has died
February 3, 2015, 1:14AM UTC[Investigating] Server appears offline

Telephone System

SummaryMemory usage 100%
ImpactDegraded Service
Current statusResolved
Open timeFebruary 18, 2015, 1:50AM UTC (GMT +00:00)
Close time02 March 2015 21:43

History

March 2, 2015, 9:43PM UTC[Resolved] service restored
February 18, 2015, 1:50AM UTC[Investigating] this is a test

Email System

SummaryHard drive failure
ImpactDowntime
Current statusResolved
Open timeMay 3, 2017, 2:56PM UTC (GMT +00:00)
Close time26 June 2018 19:46

History

June 26, 2018, 7:46PM UTC[Resolved] Hardware replaced
May 3, 2017, 2:57PM UTC[Problem identified] We've identified the bad drive. Purchasing a new one now
May 3, 2017, 2:56PM UTC[Investigating]