tag:bidtheatre.statuspage.io,2005:/historyBidTheatre Status - Incident History2024-03-29T04:54:49+01:00BidTheatretag:bidtheatre.statuspage.io,2005:Incident/101321272022-06-02T16:02:58+02:002022-06-02T16:02:58+02:00Administration Interface unavailable & Bidding halted<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>16:02</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>15:35</var> CEST</small><br><strong>Identified</strong> - The issue has been identified as being related to a major outage at our database hosting provider in the Netherlands. They are working on the issue.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>15:29</var> CEST</small><br><strong>Investigating</strong> - You may have noticed, that the BidTheatre DSP administration interface is currently unavailable. We are working to restore service, and will keep you posted. <br /><br />As a precautionary measure, bidding has been halted. Adserving is operational.</p>tag:bidtheatre.statuspage.io,2005:Incident/58406222020-12-19T12:27:37+01:002020-12-19T12:27:37+01:00Bidding halted & Admin interface unavailable<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>12:27</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>11:56</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>11:08</var> CET</small><br><strong>Investigating</strong> - We are experiencing a system issue that affects bidding. Our platform team is working with the highest effort to restore services, and we will keep you posted.</p>tag:bidtheatre.statuspage.io,2005:Incident/39007412020-04-17T15:55:32+02:002020-04-17T15:55:32+02:00Server Maintenance - Administration Interface unavailable<p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>15:55</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>15:45</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>15:27</var> CEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:bidtheatre.statuspage.io,2005:Incident/38988062020-04-17T09:38:06+02:002020-04-17T09:38:06+02:00Bidding halted, Administration interface down<p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>09:38</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>08:37</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>07:49</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:bidtheatre.statuspage.io,2005:Incident/38751602020-04-13T15:34:04+02:002020-04-13T15:34:04+02:00Statistics lag, bidding halted<p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>15:34</var> CEST</small><br><strong>Resolved</strong> - Dear client,<br /><br />a system incident has impacted BidTheatre's platform and our services. <br /><br />Background:<br />Starting on the evening of Saturday 11th if April at 22 CET, our adserving cluster was able to only partially release collected statistics on ads bought and served to our data backend. This was not detected and alerted upon by our platform monitoring, and the issue was not detected until 11 CET Sunday 12th of April, after which an investigation into the problem ensued. <br /><br />As of 13 CET Monday 13th of April, fixes have been put in production and operation is restored to normal. We are currently closely monitoring, and will follow up with more information if needed.<br /><br />Cause:<br />The problem was caused by a misconfiguration that limited the capacity of our database backend to receive data. Combined with recent changes to our database tables, we hit a capacity limitation that was not foreseen.<br /><br />Effect:<br />All missing data has been restored from a backup system. However, some campaigns have been paced poorly during the time of the incident, particularly during the morning of April 12th. We will reach out to you if your campaigns have been impacted.<br /><br />Resolution:<br />A number of measures have been taken to mitigate the risk of similar future incidents:<br />1) Application configuration has been altered to improve the throughput to our database backend.<br />2) Database tables have been reconfigured to improve the capacity to store data.<br />3) An alerting system is currently being implemented to notify our platform operations team if a similar incident would occur again.<br /><br />Timeline:<br />Incident started: Sat 11 Apr @ approx 22 CET<br />Incident detected: Sun 12 Apr @ approx 11 CET<br />Incident fully resolved: Mon 13 Apr @ approx 13 CET<br /><br />Although we are doing everything we can to keep the BidTheatre platform robust and available 24/7, incidents unfortunately occur. Our job is to make them as few and far between as possible. We hope for the continued trust in delivering your marketing campaigns, thank you for your business!<br /><br />Kind regards,<br />Marcus Johansson, CEO</p><p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>11:15</var> CEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:bidtheatre.statuspage.io,2005:Incident/35895422020-02-20T16:05:31+01:002020-02-20T16:05:31+01:00Bidding halted, Administration interface down<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>16:05</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>15:37</var> CET</small><br><strong>Investigating</strong> - We are experiencing a system issue that affects bidding and the administration interface. Our platform team is working with the highest effort to restore services, and we will keep you posted.</p>tag:bidtheatre.statuspage.io,2005:Incident/35757532020-02-17T10:59:58+01:002020-02-17T10:59:58+01:00System issue<p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>10:59</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>10:56</var> CET</small><br><strong>Investigating</strong> - We are experiencing a system issue that affects bidding and that makes the BidTheatre DSP administration interface unavailable. Our platform team is working with the highest effort to restore services, and we will keep you posted.</p>tag:bidtheatre.statuspage.io,2005:Incident/34212072020-01-13T09:27:43+01:002020-01-13T09:27:43+01:00Bidding halted<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>09:27</var> CET</small><br><strong>Resolved</strong> - All system are back up and operational. Root cause of the problem was an issue at one of our hosting providers.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>08:35</var> CET</small><br><strong>Investigating</strong> - You may have noticed, that the BidTheatre DSP administration interface is currently unavailable. In addition to this bidding is halted. <br /><br />Our platform team is working with the highest effort to restore services, and we will keep you posted.</p>tag:bidtheatre.statuspage.io,2005:Incident/31773512019-11-04T12:30:24+01:002019-11-04T12:30:24+01:00Bidding halted due to statistics lag<p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>12:30</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:55</var> CET</small><br><strong>Identified</strong> - An internal check has indicated that hourly statistics has not been computed within standard time limits. As a precaution, bidding has been halted. We are investigating the issue and will update shortly.</p>tag:bidtheatre.statuspage.io,2005:Incident/31673252019-11-01T10:32:24+01:002019-11-01T10:32:24+01:00Bidding halted<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>10:32</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>03:28</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>03:25</var> CET</small><br><strong>Investigating</strong> - One of our hosting partners are experiencing a system issue that affects our bidding, API and administration interface. Our platform team is working with the highest effort to restore services, and we will keep you posted.</p>tag:bidtheatre.statuspage.io,2005:Incident/30542992019-10-01T07:37:54+02:002019-10-01T07:37:54+02:00Bidding halted<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>07:37</var> CEST</small><br><strong>Resolved</strong> - An incident at our hosting provider caused the CDN storage that stores creative to come offline. Incident is now resolved.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>00:00</var> CEST</small><br><strong>Investigating</strong> - We are experiencing a system issue that affects bidding. Our platform team is working with the highest effort to restore services, and we will keep you posted.</p>tag:bidtheatre.statuspage.io,2005:Incident/29009032019-08-22T20:33:19+02:002019-08-22T20:33:19+02:00Hosting provider network issues.<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>20:33</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>20:10</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>15:59</var> CEST</small><br><strong>Investigating</strong> - Our hosting provider are currently experiencing network problems that affect parts of our systems.</p>tag:bidtheatre.statuspage.io,2005:Incident/26091512019-06-26T10:25:46+02:002019-06-26T10:25:46+02:00Bidding halted due to statistics lag<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>10:25</var> CEST</small><br><strong>Resolved</strong> - A system incident occurred on the evening of the 22nd of June at approximately 22:45. A regular maintenance job failed, and bidding was halted as a precaution. On call support staff was alerted, but a human error caused this to go un-noted until approximately 11:30 on the 23 of June. The maintenance job was restarted, and bidding resumed at approximately 12:15.</p>tag:bidtheatre.statuspage.io,2005:Incident/22011972019-02-11T10:51:59+01:002019-02-13T11:59:11+01:00Administration Interface Unavailable<p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>10:51</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>09:42</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:bidtheatre.statuspage.io,2005:Incident/21982932019-02-08T15:00:27+01:002019-02-08T15:00:27+01:00Planned maintenance: Restart of administrator interface<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:30</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:17</var> CET</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:17</var> CET</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:14</var> CET</small><br><strong>Scheduled</strong> - We will be restarting the Administrator interface, which will cause some down-time.</p>tag:bidtheatre.statuspage.io,2005:Incident/21956702019-02-07T08:28:55+01:002019-02-07T08:28:55+01:00Administration interface unavailable<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>08:28</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>08:16</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:bidtheatre.statuspage.io,2005:Incident/21895572019-02-04T23:00:50+01:002019-02-04T23:00:50+01:00Planned maintenance: Restart of Windows servers<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>23:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>22:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>14:17</var> CET</small><br><strong>Scheduled</strong> - A restart of Windows servers due to patching is planned. Administration interface and API will be degraded for a short period of time.</p>tag:bidtheatre.statuspage.io,2005:Incident/21616542019-01-22T11:51:22+01:002019-01-22T11:51:22+01:00Administration Interface unavailable and Bidding halted<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>11:51</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>11:50</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>11:05</var> CET</small><br><strong>Investigating</strong> - You may have noticed, that the BidTheatre DSP administration interface is currently unavailable. In addition to this all bidding is halted.<br /><br />Our platform team is working with the highest effort to restore services, and we will keep you posted.<br /><br />Please note that adserving is unaffected and fully operations.</p>tag:bidtheatre.statuspage.io,2005:Incident/21606932019-01-21T19:05:17+01:002019-01-21T19:05:17+01:00Incident Report 18-19 Jan<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>19:05</var> CET</small><br><strong>Resolved</strong> - Dear client,<br /><br />On Fri 18 Jan we had a severe system incident that affected BidTheatre services. This incident report details what occurred, and what measures have been taken to guard against similar futures incidents.<br /><br />Summary:<br />On the afternoon of Friday the 18th, an intrusion was made on our database backend, consisting of a master / slave replica set. The replicated setup allows for fast failover, should something happen to the master database. However, the intrusion rendered both master and replica unusable, and we had to resort to off-site backups to restore the database service. All-in-all, this caused most services to be unavailable during Friday evening to mid-day Saturday.<br /><br />Timeline:<br />Fri 18 12:14 - Monitoring indicates that standby replica database is unaccessible<br />Fri 18 15:00 - DB replica is established to have been intruded upon<br />Fri 18 16:00 - Intrusion is deemed to likely have been possible through a weakness in the Microsoft RDP protocol for remote access<br />Fri 18 17:30 - Clean system setup of slave is initiated<br />Fri 18 18:15 - Unplanned maintenance of master db is initiated to secure it from the same vulnerability as the master<br />Fri 18 18:20 - An incident issue to inform clients is created<br />Fri 18 18:55 - Master db is established to have been intruded upon. Clean system setup of server is initiated<br />Sat 19 09:50 - Backup restored on master db<br />Sat 19 18:00 - Full replication of database master - slave is restored<br /><br />Impact:<br />Bidding service: Halted from Fri 18:15 to Sat 19 13:10<br />API service: Unavailable from Fri 18:15 to 10:55<br />Administration UI: Unavailable from Fri 18:15 to 10:55<br />Adserving service: Operational<br /><br />Data loss from approx 16:14 (time of last backup) to 18:20 on Fri 18 (time for bidding off). Please consult with our platform team at operations@bidtheatre.com if you have experienced data loss, and we will try to remedy this to the best of our ability.<br /><br />Measurements:<br />The intrusion was facilitated due to a weakness in the Microsoft RDP protocol used for remote management of servers. We have since the attack closed RDP access, and will resort to other ways to control the servers remotely.<br /><br />Server patching with latest software updates has been upgraded to be done within maximum three days instead of weekly.<br /><br />Comments:<br />We are deeply sorry for this incident, an outage of this sort is simply unacceptable. We have made it a first priority to employ state-of-the-art security measurements such as two factor authentication, encryption, anti-virus software, along with documented security policies and other procedures to enforce security. Nevertheless, the work to guard Internet services against malicious entities is an ongoing and never ending one. <br /><br />We are committed towards offering an uninterrupted, always-on 24/7 service of programmatic advertising, and I sincerely hope that we will have your continued trust in doing so.<br /><br />Best Regards<br />Marcus Johansson, Founder & CEO BidTheatre</p>tag:bidtheatre.statuspage.io,2005:Incident/21568902019-01-20T15:04:57+01:002019-01-22T01:37:56+01:00Bidding halted, administration interface unavailable<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>15:04</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>13:22</var> CET</small><br><strong>Monitoring</strong> - All systems are back online.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>19:01</var> CET</small><br><strong>Identified</strong> - We are experiencing a system issue that affects bidding. Our platform team is working with the highest effort to restore services, and we will keep you posted.</p>tag:bidtheatre.statuspage.io,2005:Incident/20965932018-12-13T12:30:46+01:002018-12-13T12:30:46+01:00Planned maintenance: Database restart<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>12:30</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>11:30</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>11:08</var> CET</small><br><strong>Scheduled</strong> - A restart of a database is planned. Bidding will be stopped during this period.</p>tag:bidtheatre.statuspage.io,2005:Incident/20325822018-11-13T12:25:33+01:002018-11-13T12:25:33+01:00Bidding halted due to statistics lag<p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>12:25</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>09:33</var> CET</small><br><strong>Monitoring</strong> - An internal check has indicated that hourly statistics has not been computed within standard time limits. As a precaution, bidding has been halted. We are investigating the issue and will update shortly.</p>tag:bidtheatre.statuspage.io,2005:Incident/20328282018-11-13T11:59:48+01:002018-11-13T11:59:48+01:00Administration interface offline<p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>11:59</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>11:43</var> CET</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>11:43</var> CET</small><br><strong>Identified</strong> - Administration interface is currently offline due to a database restart, we expect it to be back online with 30 minutes.</p>tag:bidtheatre.statuspage.io,2005:Incident/19956402018-10-26T15:23:19+02:002018-10-26T15:23:19+02:00Bid strategies paused and re-started<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>15:23</var> CEST</small><br><strong>Resolved</strong> - Dear client,<br /><br />an incident just occured where campaigns where temporarily halted due to a mistake in a planned maintenance activity. This caused campaigns to be halted for circa 25 minutes. The problem is resolved, but you will see logs about campaigns stopping and starting again. These can be ignored.<br /><br />We're sorry for this inconvenience and the halted bidding.<br /><br />BidTheatre Team</p>tag:bidtheatre.statuspage.io,2005:Incident/19208032018-09-20T21:35:47+02:002018-09-20T21:35:47+02:00Bidding halted due to statistics lag<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>21:35</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>20:30</var> CEST</small><br><strong>Identified</strong> - An internal check has indicated that hourly statistics has not been computed within standard time limits. As a precaution, bidding has been halted. We are investigating the issue and will update shortly.</p>