tag:status.ticketmatic.com,2005:/historyTicketmatic Status - Incident History2024-03-19T09:48:23+01:00Ticketmatictag:status.ticketmatic.com,2005:Incident/150826052022-12-13T15:45:58+01:002022-12-14T10:30:28+01:00Service problems due to high load<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>15:45</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>14:33</var> CET</small><br><strong>Monitoring</strong> - The load is under control and we are monitoring the situation.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>13:45</var> CET</small><br><strong>Update</strong> - We managed to bring the load on the system under control. We will now gradually increase the rate limiters to normal levels.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>13:19</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>13:19</var> CET</small><br><strong>Investigating</strong> - We are experiencing service problems due to very high load. We are investigating the issue.</p>tag:status.ticketmatic.com,2005:Incident/34981942020-01-31T09:34:50+01:002020-01-31T09:34:50+01:00Order crashes<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>09:34</var> CET</small><br><strong>Resolved</strong> - After deploy of a new version, we noticed crashes in the Order app. We quickly reverted to the previous version, everything should be fully functional again.</p>tag:status.ticketmatic.com,2005:Incident/31521542019-10-28T14:48:09+01:002019-10-28T14:48:09+01:00Issues with Mollie PSP payments<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>14:48</var> CET</small><br><strong>Resolved</strong> - Mollie reports the incident as resolved</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>14:29</var> CET</small><br><strong>Identified</strong> - PSP Mollie currently has problems on its platform that causes problems with processing payments. More info and status updates on their status page: https://status.mollie.com/</p>tag:status.ticketmatic.com,2005:Incident/30764552019-10-07T14:05:42+02:002019-10-07T14:05:42+02:00Temporary issue during deployment<p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>14:05</var> CEST</small><br><strong>Resolved</strong> - There was a small issue during a deployment around 13:50 CET that might have caused some crashes. It was resolved after a few minutes.</p>tag:status.ticketmatic.com,2005:Incident/21917022019-02-05T20:54:24+01:002019-02-05T20:54:24+01:00Web errors<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>20:54</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>15:39</var> CET</small><br><strong>Monitoring</strong> - We have resolved the issue, error levels are back to normal.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>14:29</var> CET</small><br><strong>Investigating</strong> - We see elevated error rates and are investigating</p>tag:status.ticketmatic.com,2005:Incident/19321522018-09-26T12:53:33+02:002018-09-26T12:53:33+02:00Ticket generation issues<p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>12:53</var> CEST</small><br><strong>Resolved</strong> - The problems at AWS are resolved and ticket generation is fully functional again. Admins may still receive some error messages by e-mail that were delayed in the queue, these may be ignored.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>10:49</var> CEST</small><br><strong>Update</strong> - AWS confirms that they are currently experiencing issues that are effecting EC2 and S3 in the eu-west-1 region. They are busy investigating this issue.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>10:39</var> CEST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>10:38</var> CEST</small><br><strong>Identified</strong> - Due to network problems at our hosting provider AWS, there are issues with generating tickets. We are looking for a solution.</p>tag:status.ticketmatic.com,2005:Incident/17666452018-06-12T19:12:27+02:002018-06-12T19:12:27+02:00Possible disruption of credit card payments<p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>19:12</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>18:22</var> CEST</small><br><strong>Monitoring</strong> - Credit card payments should be functional again. As displayed on Mollie status page: "We've identified the problem and are seeing no more failures on credit card payments. We are monitoring the situation and will resolve this incident once we're certain it will not reoccur. "</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>16:16</var> CEST</small><br><strong>Investigating</strong> - Payment processor Mollie indicates possible problems with credit card payment processing since 15:36. You can monitor https://status.mollie.com/ for up to date information</p>tag:status.ticketmatic.com,2005:Incident/16996702018-04-21T17:09:23+02:002018-04-21T17:09:23+02:00Server problem: ticketmatic backoffice unavailable<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>17:09</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>16:52</var> CEST</small><br><strong>Update</strong> - All services back online.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>16:50</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>16:49</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.ticketmatic.com,2005:Incident/16405672018-03-14T10:49:44+01:002018-03-14T10:49:44+01:00Printclient issues<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>10:49</var> CET</small><br><strong>Resolved</strong> - The fix is rolled out and printing using the printclient should work again. Refresh the application if necessary.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>10:30</var> CET</small><br><strong>Identified</strong> - We found the origin of the problem and have a fix ready. We will roll this out soon.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>10:25</var> CET</small><br><strong>Investigating</strong> - We are getting reports on printclient issues and are investigating. In the mean time, if you cannot print using the printclient, use order preview as a workaround to print tickets.</p>tag:status.ticketmatic.com,2005:Incident/15590592018-01-09T01:56:56+01:002018-01-09T01:56:56+01:00Database maintenance<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>01:56</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>00:57</var> CET</small><br><strong>Identified</strong> - We are performing scheduled database maintenance that may cause downtime.</p>tag:status.ticketmatic.com,2005:Incident/15403842017-12-18T18:31:52+01:002017-12-18T18:31:52+01:00Faulty deployment<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>18:31</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>17:19</var> CET</small><br><strong>Monitoring</strong> - Everything should be back to normal and all service should be restored.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>16:56</var> CET</small><br><strong>Identified</strong> - We identified the cause of the problem and are working on a fix</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>16:48</var> CET</small><br><strong>Monitoring</strong> - Everything should be back to normal and all service should be restored.
<br />
<br />We're keeping a close eye on things.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>16:38</var> CET</small><br><strong>Investigating</strong> - A faulty deployment is causing requests to time out. We are actively investigating the problem.</p>tag:status.ticketmatic.com,2005:Incident/15031942017-11-14T14:59:53+01:002017-11-14T14:59:53+01:00Contact detail problem<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>14:59</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>14:59</var> CET</small><br><strong>Monitoring</strong> - The problem is fixed!</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>14:52</var> CET</small><br><strong>Identified</strong> - We are experiencing problems on the contact detail page. A fix is being rolled out now and will be available very soon.</p>tag:status.ticketmatic.com,2005:Incident/14116672017-10-13T12:20:22+02:002017-10-13T12:20:22+02:00Error when creating new event and selling seated tickets in backoffice<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>12:20</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>10:47</var> CEST</small><br><strong>Monitoring</strong> - A fix is deployed that solves these problems. If necessary, refresh the application.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:46</var> CEST</small><br><strong>Update</strong> - The problem with selling tickets from the Backoffice only occur when you use Dutch as interface language. While we are working on a final fix, you can use this workaround to be able to sell tickets through the Backoffice:
<br />
<br />- as user go to 'Mijn Ticketmatic ID' pagina (on top right, under your user name)
<br />- change your interface language to 'English' and Save
<br />- refresh the application (close browser and reopen application)</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:23</var> CEST</small><br><strong>Update</strong> - There are also related problems with selling tickets for seated events from the Backoffice. Websales is not affected. We are working on a fix.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:11</var> CEST</small><br><strong>Identified</strong> - We see errors when creating new events from the Backoffice. The reason is related to a problem with the latest deployment of the app this morning. We are now rolling out a new version that should fix this.</p>tag:status.ticketmatic.com,2005:Incident/13639032017-09-18T13:27:41+02:002017-09-18T13:27:41+02:00504 errors while using backoffice<p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>13:27</var> CEST</small><br><strong>Resolved</strong> - This incident is resolved.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>12:19</var> CEST</small><br><strong>Monitoring</strong> - The problem seems to originate from a networking problem on one of the underlying worker nodes. We restarted the faulty node and this seems to resolve the problem. We keep monitoring for now.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>12:03</var> CEST</small><br><strong>Investigating</strong> - We are investigating reports of users getting 504 errors while using the application.</p>tag:status.ticketmatic.com,2005:Incident/12844182017-07-05T11:13:53+02:002017-07-05T11:13:53+02:00Web sales down for some accounts<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>11:13</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>10:21</var> CEST</small><br><strong>Monitoring</strong> - Rollback is completed, functionality is restored. We keep monitoring this.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>10:15</var> CEST</small><br><strong>Identified</strong> - Due to a failed deployment, web sales is down for some accounts. We are rolling back to the previous version.</p>tag:status.ticketmatic.com,2005:Incident/11244222017-02-23T17:05:02+01:002017-02-23T17:05:02+01:00[TM2 only] Backoffice login problem<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>17:05</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>01:23</var> CET</small><br><strong>Monitoring</strong> - We finished setting up new temporary infrastructure and backoffice login is again possible for all accounts. Make sure to use the launcher to login. We will keep on working to restore the original servers and bring these back in operation once this can be safely done.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>19:09</var> CET</small><br><strong>Identified</strong> - 7 Ticketmatic Windows servers are under heavy attack, making the Backoffice unusable. We are both mitigating the attack and setting up new infrastructure. Be sure to use Ticketmatic Launcher to login.
<br />Databases and webservers are unaffected. Online sales is unaffected.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>14:59</var> CET</small><br><strong>Investigating</strong> - There are problems logging in to the backoffice. We are currently investigating.
<br />Online ticket sales is unaffected.</p>tag:status.ticketmatic.com,2005:Incident/10181442016-12-11T21:15:58+01:002016-12-11T21:15:58+01:00Increased error rates<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>21:15</var> CET</small><br><strong>Resolved</strong> - Everything is back to normal.
<br />
<br />This disruption was caused by a recent change in our web socket connection handling code.
<br />A combination of factors led to a serious overload on one of our services, which in turn caused some connections to time out.
<br />
<br />We'll be updating our automated QA processes to ensure that this won't happen again in the future.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>20:04</var> CET</small><br><strong>Monitoring</strong> - A fix has been deployed, service should no longer be disrupted.
<br />We're keeping a close eye on things but there shouldn't be any impact anymore.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>19:09</var> CET</small><br><strong>Investigating</strong> - We're observing increased error rates due to a disruption in one of our services. A small minority of our users will be impacted and encounter an error dialog. A simple reload of the web page should fix things and allow you to continue.
<br />
<br />A fix is being worked on, apologies for the inconvenience.</p>tag:status.ticketmatic.com,2005:Incident/9429472016-10-08T08:23:26+02:002018-08-03T19:32:02+02:00Possible service disruption<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>08:23</var> CEST</small><br><strong>Resolved</strong> - The service seems fully operational again, we keep following up</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>08:15</var> CEST</small><br><strong>Identified</strong> - An internal service failed over and was put in read-only mode, we are restoring the service now.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>08:07</var> CEST</small><br><strong>Investigating</strong> - There is a possible service disruption causing part of the internal api calls to fail. Investigating.</p>tag:status.ticketmatic.com,2005:Incident/5373882016-06-21T10:34:44+02:002016-06-21T10:34:44+02:00datacenter disruption: some accounts unavailable<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>10:34</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>10:28</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>09:39</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.ticketmatic.com,2005:Incident/5080332016-05-18T23:27:07+02:002016-05-18T23:27:07+02:00Service disruption<p><small>May <var data-var='date'>18</var>, <var data-var='time'>23:27</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>21:02</var> CEST</small><br><strong>Monitoring</strong> - The issue is resolved for now, we keep monitoring</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>20:24</var> CEST</small><br><strong>Update</strong> - The cause for the disruption is identified and caused by excessive amount of database connections. The disruption is limited to a single account</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>20:18</var> CEST</small><br><strong>Investigating</strong> - We experience a service disruption at this moment. We are further investigating</p>tag:status.ticketmatic.com,2005:Incident/5054752016-05-16T13:58:00+02:002016-05-16T14:53:53+02:00temporary malfunction of one webinstance<p><small>May <var data-var='date'>16</var>, <var data-var='time'>13:58</var> CEST</small><br><strong>Resolved</strong> - might have impacted some ticketmatic accounts</p>tag:status.ticketmatic.com,2005:Incident/4956172016-05-07T10:35:22+02:002016-05-07T10:35:22+02:00possible login problems with iOS scanning app<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>10:35</var> CEST</small><br><strong>Resolved</strong> - fixed version of iOS scanning app available in App Store</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>18:30</var> CEST</small><br><strong>Identified</strong> - The latest release of the iOS scanning app can malfunction, preventing login. A fixed version is being published on the App Store. Previous versions work normally.</p>tag:status.ticketmatic.com,2005:Incident/3508062015-11-05T11:32:25+01:002015-11-05T11:32:25+01:00some servers unavailable<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>11:32</var> CET</small><br><strong>Resolved</strong> - router disruption was cause, problem solved</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>11:21</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.ticketmatic.com,2005:Incident/3386602015-10-17T00:24:46+02:002015-10-17T00:24:46+02:00some servers unreachable<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>00:24</var> CEST</small><br><strong>Resolved</strong> - router problem solved, all accounts reachable</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>00:04</var> CEST</small><br><strong>Investigating</strong> - several servers cannot be reached, affecting some customers</p>tag:status.ticketmatic.com,2005:Incident/3062132015-09-01T20:36:00+02:002015-09-02T10:13:01+02:00Accounts partially unavailable<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>20:36</var> CEST</small><br><strong>Resolved</strong> - Our upstream provider confirms the cause was a problem with router infrastructure in the datacenter. All systems are operational for now but we keep monitoring the situation.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>20:35</var> CEST</small><br><strong>Update</strong> - Everything looks ok now.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>20:28</var> CEST</small><br><strong>Monitoring</strong> - Some accounts were unavailable for a short time. The problem was situated at a network provider upstream. Everything seems back to normal now, we keep on monitoring the situation</p>