Does anyone inside Twitter know who the executive was who pulled the trigger on this decision? There's no way this person knew the gravity of accepting the risk for blowing this many high-profile customers out of the water all at once.
I'm pretty accustomed to executives making dreadful decisions without the approval/acceptance of other stakeholders within said person's firm. I'd rather know who made this error and not interact with that specific person's department rather than stop doing business (e.g. large ad-buys) with Twitter as a whole.
I don't want this to be a witch-hunt so much as I want the person to just come forward and own the decision, because unless they have an exceptionally good reason for it, it comes off as absurdly high-risk to both Twitter-the-business as well as to all the clients who've likely written serious penalties into their contracts for events like this, which again brings that business risk back full-circle to... Twitter.
I work at twitter and wasn’t able to find out immediately, but have been collecting responses like these + the article and communicating with someone who knows what to do. I only found out about this over twitter and I am personally deeply frustrated and working to understand whatever led to this.
To be honest, this reads less like an attempt to integrate anti-harassment measures and more like an attempt to completely destroy a business trying to offer them. It further solidifies Twitter management as complicit in aiding and abetting the harassers by using its capital to eliminate a threat from the market. After the speed with which Twitter went after the ICE employees list, the evidence is clear that Twitter only has a problem tackling harassment when it's against marginalized populations.
That's a pretty extreme leap when virtually no information is available yet. The HN guidelines ask you to "assume good faith" for a reason: people are all too ready to take such charges as givens and then pile more on top of them. It's a behavior that harms the container here, and it's not hard to wait until actual evidence appears.
That's fair, I will suspend judgement until further evidence appears, but the well of beneficial doubt is running low. I think a long past history of suspicious behavior is a good reason to update our assumptions.
I don't disagree. But the real reason for having an "assume good faith" rule is what it does to this community when people don't. Therefore it needs practicing even when it feels undeserved.
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
It's talking about how we all talk to each other. I think it's main purpose is not to lessen criticisms for corporations so much as to keep the discussion constructive.
You're probably not going to win an argument with a moderator by explaining to them the subtle logical error in the interpretation and purpose of their own guidelines.
Yet now we're failing to assume good faith in our moderators, by assuming that they will fail to recognize a conflict of interest arising from questioning the extent and source of their authority, which they have reason to see maximized.
The parent comment actually seemed spot on to me. The rule is to assume good faith of your interlocutors. The parent failed to assume good faith of a billion dollar corporation. I think it's reasonable to push back against being asked to tone down the latter, while honoring the rule for the former.
I think dang explained it pretty clearly in the comment. It is bad for the forum when, given some very limited set of facts, you offer the most heated, ragefest pile-on inducing explanation or theory. It doesn't require some talmudic reading of the guidelines which are, after all, guidelines and not every single thing is explicitly spelled out. 'Avoid taking discussions into flamey or ragey directions' is advice the mods dole out to some person or another nearly daily.
Hi. Former Twitter anti-abuse employee here. If I had to sum up Twitter's executive leadership, it would be with Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity."
Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way. They mainly care about it as a PR problem, so they'll do the minimum necessary to get out of trouble. But that's not because they hate anybody; it's because as a company they're not very good at getting anything at all done. They succeeded because they created a platform with a network effect; in that situation, competence is secondary.
My guess is that Smyte was failing as a business and looked around for an acquihire. I'd further guess that they had some connection at Twitter, and that getting a few engineers with relevant experience at a low price seemed like a good deal. So I'd suspect that nobody at Twitter cared either way what happened to Smyte customers.
>Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.
I don't like the implication here (i.e. white men don't care about harassment/abuse and they are also stupid.) I would wager that most anyone of any color or gender or religion in any high position of authority at any company cares first and foremost about the financial viability of the company they have been charged with running.
Many (most) "white men" do in fact care about this stuff and aren't bigots/sexists/racists. You're naive and part of the problem if you think this is a "white male" issue and not a much more complex problem. In fact, you're exhibiting the bias you attribute to these managers.
I think it's fair to say that people who are not white males (such as myself) are more likely to be targeted by a wide variety of harassment, and that without their views it may be much easier to miss the issue. As a white male I don't get targeted by misogyny or racism (at least not anywhere near the scale that others have). Having people who have experienced that around to help guide the decisions for how to deal with it- and how many resources to devote to it, since most business decisions come down to money.
Also, we can all have biases without it meaning people are being racist. My parents like rock music, so I'm biased towards listening to that over country. I went to certain schools and that colored my world view. I once heard bias described as a heuristic the brain uses to make quick decisions- something that was extremely useful for primitive humans but which needs to be actively balanced in our complex society. A huge part of that for decision makers is exposing themselves to other viewpoints as a way to collect information they themselves wouldn't have gotten. There's a reason why companies have used things like focus groups and surveys to understand their customers- having more people who understand those customers in leadership at companies is an even better way to do it.
The abuse is disproportionately targeted at people who aren't white men, and is is disproportionately performed by white men.
This is nothing particularly new. It's the history of America, which was founded by and for white men, and whose story can be told as white men slowly and reluctantly giving up their power.
If "many (most) 'white men' do in fact care", history doesn't really demonstrate that. Not, at least, in terms of doing anything about it. This is not particularly shocking; people care more about problems they experience, and less about problems that happen to benefit them.
We need only look at how the majority of white men supported a blatant racist and sexist for president. And who continue to support him despite (or because) that has only become more clear.
Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.
To the folks inclined to keep piling on to this comment, I think a more charitable reading is that "The leadership consists of very privileged people who likely haven't dealt firsthand with the kind of harassment to which some demographics are routinely subjected. They appear to just not really get it at times."
I think that's a very realistic assessment of how some things remain intractable. People in power have blind spots because it isn't something they have to personally deal with. So they end up doing things that are counterproductive or seriously problematic without really meaning to.
>The leadership consists of very privileged people
Same problem. Who's to say they are "privileged" as opposed to being hard working / talented people? You're pushing the same narrative; white men in positions of power are privileged and don't really deserve what they have. Nonsense. Sometimes that's true, sometimes it's not. Either way it's a useless, tribal way of thinking that does more harm than good.
It's completely fair to comment on this because it is the basis of the GP's argument.
I don't read the parent comment as claiming that the people in power are not hard-working or talented, nor that they don't care. Rather, I read it as asserting that many white men have had the privilege of not having to deal with discrimination and harassment, and that it doesn't come to mind as easily as it does for underrepresented individuals.
Just like a developer is going to spot defects in applications more than a non-technical user because they are exposed to defects more often and understand where things will fail, people who have lived discrimination will be more likely to see it where it happens, and understand where to look to find it.
I imagine the leadership of a company like Twitter make really good money, have substantial power and don't have the problems of your average working stiff, regardless of skin color, gender etc.
It's sort of tautalogical to say "People in power don't live like the rest of the world." And this fact has been a problem in terms of creating policies that work well for the masses since time immemorial.
It's one thing to say that people in positions of power, on average, have not directly experienced the harassment or abuse that minorities or people belonging to marginalized groups have. That's fair and may account for some level of ignorance on their part. It is another to say that they _do not care about these things_ specifically because they are a member of a certain class/race/gender/whatever.
It's an important distinction as it sets the parameters by which we engage in conversation and debate. If you begin the conversation by accusing the other side of being inherently bigoted then there is no civil or useful exchange of ideas to be had. It puts the other side in a defensive posture from the get go and only leads to more tribal group-think.
And you refusing to accept my reframing as valid intentionally digs this hole deeper. This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something.
Both sides need to give a little here if we aren't going to remain stuck on such topics. And expecting all that giving to come from women, people of color, and other underprivileged groups is merely entrenching white male privilege in a really ugly, toxic manner.
>And you refusing to accept my reframing as valid intentionally digs this hole deeper.
Because I don't believe it is valid. Class/color/gender are not the only influencers on someone's decision making process, yet you two seem to think that's all we need to consider.
Where is the evidence that these people don't care about abuse and harassment because of their class and color? I see a pretty damning statement which includes zero evidence, just what someone believes to be true for <reasons>.
So no, I won't accept your premise or the GP's until I see evidence. If there is no evidence then fine, you and anyone else are free to believe whatever you like, but don't pretend it's a logically defensible position.
>This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something. Both sides need to give a little here if we aren't going to remain stuck on such topics.
Not every complaint _is_ valid, and we have no idea what the GP's motivations were. I'm not going to give any ground. You and the GP are taking a complex issue and boiling it down to something absurdly simple. All that I'm saying is that it's not simple and, if we're ever to have a reasoned debate, it needs to be predicated on facts, not "feelings". If you're going to attribute someone's actions to their gender and race then you had better be able to back it up.
It's telling that you believe _I'm_ the one doing harm when I have made no accusations at all and the GP was inarguably expressing a racist view.
It was not in fact a racist view. It's a view about power dynamics in a system that is racist and sexist.
If you really can't get over your feelings of personal insult and your need to defend white men, then please go back and substitute "part of the socially dominant racial group" for "white" and "part of the socially dominant gender group" for "men". In US history, those terms are equivalent. If you don't like that, talk to the founding fathers who set it up that way.
The reason it's material here is that a large portion of Twitter abuse is related to this. You can look at famous incidents, for example. Leslie Jones received a giant wave of racist, sexist abuse: https://www.thecut.com/2016/08/a-timeline-of-leslie-joness-h...
Note also your standard here. You, a white man, refuse to accept that race or gender could be relevant to a social problem until other people take the time to spoon-feed you. Why isn't it your job to know something about American history? Why isn't it your job to learn about the problem of online abuse before deciding that the privilege afforded to white men couldn't possibly be the relevant? Answer: because you were raised to believe that whiteness and maleness were the default and were good. So you choose that as your supposedly neutral stance, and require others to prove you wrong.
The true neutral stance would either be "I don't know about this problem and would like to learn" or "let's assume that online culture has some relationship to the existing offline social dynamic".
The problem is not the genetic characteristics of whiteness or maleness. It's that they grew up in a society that favors whiteness and maleness, and is much more accepting of the abuse of people who aren't so favored. If Twitter had started in China, where the Han are favored over ethnic minority groups like Tibetans and the Uyghur, it would be the same dynamic but with different ethnic groups.
Of course, tech is disproportionately white and male, so it's entirely unsurprising that HN readers have risen up en masse to defend the honor of white men. It exactly proves the point. White men will say that fighting abuse is important. But when they have to show their real priorities by how they allocate time and effort, actually understanding the abuse problem and how it disproportionately effects minorities comes well after defending their own interests.
(And: Yes yes, not all white men. Generalizations are general statements that may or may not apply to an individual.)
It's late. I'm tired. I left a couple of replies earlier, then deleted them. This seems like a just no win situation to me.
I'm not a feminist. I'm a former homemaker that self proclaimed feminists routinely crap on. I come to Hacker News because I need a place where I can engage in meaty intellectual discussion so I don't lose my marbles. That is my only agenda for being here. However, dealing with sexual politics is not something I can entirely avoid what with posting as openly female.
Suffice it to say that I don't really want to be associated with this toxic attitude and I am kind of wishing I had not said anything at all. I am starting to feel like "The only winning move is not to play" and I failed to make that move.
While that's true, you could at least tip your hat to Jack Dorsey's $3bn net worth, the insanely high executive and engineering salaries in San Francisco, and the effect that being so insanely rich must have on the complacency and sheltered lives of everyone at Twitter. It's a little bit worrying that you think race and sex are greater assets than class (= "network" and "culture fit") and money.
There is also an interesting 'get off my property' aspect to this acquisition. It may indicate that there is a desire within Twitter the company for complete control over Twitter the social media platform. Because they can, Twitter will always reign over the mirage of an ecosystem built on their social media platform.
I think to a great extent that can be (just about) assumed about anyone in a leadership role at a company the size of Twitter. Regardless of anything else.
Implying that white males don't care is not helpful and is both racist and sexist. I'm a white male and I care about abuse and all kinds of other things. Maybe you should do some reading on unconscious bias.
Another plausible scenario is the acquisition terms required Smyte to wind down the service over the past 6 months or build a replacement to fulfill contracts. Smyte didn't for whatever reason (either through negligence or negligence+hope to continue operations in a new capacity) and the deadline came to hand over the servers in the agreed condition (no externally accessible APIs available).
Twitter could have stepped in and halted things, but that would have required Smyte to have acknowledged breaching the contract and forfeiting $$.
This is all guess of what seems to me more likely than any Twitter exec pulling any triggers like this. Of course, it's still a screwup by Twitter to have not been tuned in and aware that fingers would point to them.
But that's a very different kind of mistake than "I have an idea, let's hit the power button at 6:30. Team: you have 30 minutes to let all our customers know."
You never, ever agree to a change in your business before the acquisition is complete. The chances that an acquisition won't go through are just too high, and it could leave you completely f'ed over.
Yeah, but if you're less experienced, or if you're more devoted to your potential profit than to your customers, you might agree to the term, and then realize what you just said, leaving you with the (immature but perhaps "seemed reasonable") situation we see here - doing nothing until the acquisition is complete and then immediately abiding by the terms.
To a large degree, the responsibility is also on Smyte, right? Selling to a big corp is one thing, and once you are no longer owner of a company you've washed your hands of future decisions. But at the time when they agreed to the sale they were still owners of the company and their hands had not been washed and they could have and arguably should have stipulated some conditions for how the service would be wound down following the sale.
Litigation aside, the founders of Smyte seem pretty culpable here. I would be wary of using another service that they created, given a demonstrated willingness to turn it off without any notice.
"We couldn't operate their business and continue collecting data from their customers, while continuing to meet our own high standards as a global company."
My hunch as an outsider is that Smyte wasn't GDPR compliant. Their leadership knew it, they knew they couldn't easily become so (for instance, they may have been using Kafka in a way where compaction wouldn't help, and didn't want to build an encryption-based monstrosity [0]), realized that they wouldn't increase in value as a business due to that risk, took an acquihire for cheap in order to give their employees a decent landing and give a return to their investors, and couldn't tell anyone about these plans in advance because it might jeopardize the transaction.
EDIT: They were indeed using Kafka per [1], and due to the strict latency requirements on their business, that may have ruled out the type of encryption scheme in [0].
I've seen lawyers advice that [0] doesnt do the trick anyway, ie deleting the key is not enough to comply.
GDPR does seem like the most likely culprit here, if they've been planning an aquihire for a while it wouldn't have been worth implementing GDPR and Twitter likely made it a condition that the service is shut down before the acquisition completes
I'm quite surprised none of the cloud vendors were interested/could offer more than Twitter for this team though, seems like a logical service to add.
That's a good thought. The only good reason to shut them down so abruptly is if they posed an existential threat to Twitter itself. The threat of GDPR fines or sanctions might have done it.
I still don't get why they couldn't just come out and say that. Was incompetence and unprofessionalism really a better look then a understandable engineering challenge? Or was this another case of "being honest and doing the right thing would have created legal liability so we opted not to do it"?
Presumably because Twitter was interested in the project, but didn't need the actual implementation immediately. Get the team on the cheap for now, and then just have them rebuild it in-house in a compliant way.
This is one of those instances in which I think employees (developers) and companies working with Twitter need to have a "never forget" attitude. I actually know someone personally at Twitter corp dev (the group that usually manages M&A activity) and I hope that individual was not involved here, as it would be pretty out of character.
Why would Twitter care? They make money from ads, right? APIs and legacy deals are going to be time consuming and boring for them. Stuff like filtering hate speech etc is something which can be done globally and in a way which doesn't require highly paid employees, APIs etc. Hasn't Twitter repeatedly show that third parties aren't really that interesting to them?
All companies have suppliers and partners, including Twitter. Getting a reputation for this kind of toxic behaviour harms their ability to build relationships with them, as nobody will feel they can trust them.
It could also hamper their ability to make further acquisitions. In future, a startup being linked with a possible Twitter acquisition is going to cause its customers to panic and start moving elsewhere. That'll make startups more reluctant to engage, and may turn some off entirely, if they're not comfortable with the prospect of totally screwing over the customers who've supported them during their early stages.
A startup's customers panicking? Not sure that's on twitter's radar.
I think potential for reputation damage, on an activity they don't do very often, and where they will justify it, will be weighed up against the financial benefit now, and disregarded.
I think it's possible to overstate the relevance and importance of people typing furiously on internet forums about things like this. Twitter is what it is - most people aren't using APIs, third party clients etc and will never notice this.
I never said Twitter cares about startups' customers panicking, but it cares about being able to make strategically useful acquisitions, like Smyte. And acting like this makes it harder.
This isn't about people typing furiously on internet forums. This is about business risk, as weighed up by people whose job it is to care. Acquisitions take time and negotiation. Just because a startup is talking to Twitter, doesn't mean the acquisition will definitely happen. But if customers get wind of it, then they may start looking to move elsewhere, anyway, instead of waking up to find services they rely on have been shutdown without warning. If you're the customer of a startup Twitter is negotiating to acquire, then you're negligent not to reduce your reliance on them. And if you're the CEO of that startup, then you have to consider the risk of customer flight (and your own comfort with the thought of screwing them over) vs. the likelihood of the acquisition happening.
if the Smyte shutdown limits the opportunity of current and future Twitter associated startups, whose business model is to help companies safely navigate the social media landscape, then that sounds like a win-win for Twitter
They can shrug it off, still those companies that had contracts with Smyte are not one person businesses on the other side of the world. Those Twitter people are going to have lunch or dinner with other people and get to talk about this issue for a while. Not very pleasant, even for employees totally unrelated with this case like the one who answered the very first comment in this page (sorry.)
We had 20 minutes notice, and then everyone was kicked out of the Slack support channel and API responses simply died. What the actual fuck. They have mobile client SDKs out in the wild that are now just eating up battery life as they retry an impossible query forever.
That sucks. We (Sift Science) have been building something similar for 7 years and aren’t going anywhere. If we can help, please ping me - jason at siftscience dot com
Nothing specific against Sift Science whatsoever in my comment, but many, many times this sentiment has been conveyed by companies and it rings hollow IMO. There are a lot of different ways a company can change or disappear at some unforeseen point in the future, and claims that "you can trust us to be here forever" do not carry much weight for the large group of experienced users, devs, management, etc who have been burned multiple times.
I think the parent commenter is just saying that it's not something you can actually believe, just like a company saying they'll never sell your data.
Sure, that's the intention but when someone comes around waving a checkbook, they can in turn buy you and shut you down, sell your customers' info, etc. Sure, its not YOU allowing that to happen but by ceding control you essentially allowed that to happen.
"You can trust us to be here (until enough money comes our way)" is just how it is for most businesses like this. An unfortunate, and in my opinion largely unresolvable, side effect of an ecosystem where specific useful tools are hard enough to create and manage that full companies have to be formed around them.
A third party code escrow account with instructions on how to build the service and a contract that specifies conditions when a customer gets access to that escrow.
I've been on both the customer and service end of such agreements.
Is there a popular provider for doing this? It sounds like a lot of work to find a solution in which both parties trust (both technically versed and reliability wise). A common notary probably wouldn't cut it, right?
There's a bunch of companies doing it, some dedicated, some as part of larger related offerings. E.g. I know NCC Group (large IT security company) does offer it, TÜV (which does all kinds of certification and compliance work), ...
A guarantee is only as good as the guarantor. A guarantor can be a company providing a service, as well as all sorts of assorted guarantees about said service.
In time, the situation can change. The product can get sold. Cash can be spent. The guarantor can no longer make good on its guarantees.
You're back to square one where the guarantees are as worthless as the original service.
You need 3rd party backing (insurance) in such situations, but that costs money. This money is a cost which makes competing against unbacked entities tougher.
In most cases, you can not have 100% foolproof guarantees of anything. The closest I can think of is governments standing behind their banking institutions. Even there though, governments have defaulted on their guarantees.
The world is not a stable, perfect, and cut-and-dry place as many would like to believe. It is dynamic, and ultimately backed by trust.
It's much harder to mitigate this kind of risk in the world of services as against purchased software.
The problem has always been present especially when larger slower moving companies buy from smaller, riskier, companies.
In the days of software, code escrow was possible to mitigate some of this kind of risk. That's still got it's costs but can be an effective hedge against a supplier going bust.
Do you have a high level overview of what I’d need to do to get started? Are there any specific areas you specialise in as opposed to general users and what information will I need to share with you?
Shockingly bad. I wonder if they shitcanned the whole technical team and they turned the lights off on the way out. Not that it’s really much better but at least it wouldn’t make sense.
I actually ripped out Smyte's Android SDK because of how bad it was and replaced it with my own implementation against their API which has a hard limit on retries. But I imagine that other customers were using their SDK and still have live apps out there using it... not good :/
Oh please. Don't pretend that battery destroying, relentlessly network hogging apps are a good thing.
No app should retry an impossible query forever. This is not the same as some sort of laughable abstinence-only birth control policy, where you get to roll your eyes, and say "yeah right. good luck with that."
What's more, an app that hits the network of an uncontrollable third party, should always check with a controlled system periodically, to verify that the third party shall continue to respond predictably, as a sanity check.
And beyond even that, if the sanity check within your own domain, for your own app that you created, and are responsible for, is unresponsive, it really should cripple the app's connections to things you don't control.
Finally, as one last safety measure, if your status/activation end point dies, and your native apps go dormant, you should expect to be able to re-activate them with a number of channels, including emails, push notifications and new releases that up-version the app.
If you build your kingdom within a walled garden to begin with, and rent all your tools of the trade from an external platform, expect to behave as a subsistence farmer.
It's worse for companies that have shipped mobile apps with the API embedded in the client. Web apps can at least try an alternative or disable the affected features until they work around this mess..
This confirms my belief that I don't want to ship anything that uses directly a third party API. Instead always go through your own server which then calls the API so at least you have a chance to replace that.
That works for a lot of cases, but for many tracking APIs you want the request to come from the end user, so the tracking service can get their IP and network metadata. That would certainly apply to a fraud detection API.
Tbh that has always frustrated me though. I should be able to pipe those requests and just add an X-Forwarded-For header.
But it will have the cost of 2x API bandwidth (vs 0) and another point of failure that you are responsible for. A DNS CNAME probably wouldn't work if you had to go over SSL. Maybe a 30x redirector (still have the SPoF, but simpler and much less bandwidth)? Except in the past, I've found that most clients react poorly to redirects for anything other than GETs.
Is there any indirection that would avoid having to walk the traffic over my own network?
The way I would implement it is as a “proxy” on my own network, eg trackingservice.myapp.com, that is just a web server listening and forwarding requests (technically a MITM). Override the request method of the client SDK to hit my endpoint instead of the tracking service.
Obviously you then pay the bandwidth, but it’s likely negligible compared to your app traffic. As a bonus you’ll probably circumvent adblockers.
If you want to avoid traffic forwarding, but keep flexibility over the endpoint, you can override the init of the sdk to first query your server for which endpoint to use. That way if the third party service goes down, you just need to change the config on the server.
Damn right I'm the man in the middle if I'm the SSL-offloader. It actually strengthens security because there's one less party to trust for the client.
When I was developing mobile apps, this is what we did. Too many times something changes and trying to update the code in the mobile app is way more of a pain than updating the server.
I wonder how much Twitter is going to pay on top of the acquisition cost of what I can only imagine constitutes breach of contract with all of these companies. (If it doesn't, why even bother having a contract with a duration?)
Kind of depends on the terms of the deal, namely whether it's an asset deal as opposed to a stock deal. In the latter Twitter would assume liability for contracts. In the former they just acquire IP and hire away the employees.
Acquirers usually prefer asset deals for this reason as it allows them to leave unpleasant liabilities behind. There's generally some sort of shell left that goes through wind down but it has few/no assets attached. In this case you can try to go after the original investors including founders or other shareholders (the IRS may do this if there tax issues) but since there's nobody home you are less likely to get much out of a favorable judgement.
This seems like the kind of thing a legal system should be designed to help defend the market against. It's a net negative to the economy as a whole to let torpedoes like this fly without consequence; and evidently, a totally Laissez-faire market will engineer ways to insulate the perpetrators against the damages rather than the insulate economy as a whole against the splash.
On the flip side asset deals also let acquisitions proceed that would otherwise be blocked due to uncertainty about liability, which is another way of saying that the acquirer can't properly assess value of the investment. I have been through this process and have some experience with the type of issues that can arise--in our case they had nothing to do with screwing customers. It's possible Smyte would have just disappeared anyway.
Moreover recovering damages in this kind of case is time-consuming, doubtful, and too late to fix anything. (Like closing the door after the horse is gone.)
What's left is reputation--the names of the founders of Smyte are listed at https://www.crunchbase.com/organization/smyte. I would have some pretty hard questions if they ever showed up in a deal or looking for work.
p.s., According to the Techcrunch page "Smyte stops bad actors on marketplaces and social networks." You can't make this stuff up.
You can write just about any words you like into a contract. But that doesn't mean they are enforceable and doesn't mean a legislature or court can't add or modify clauses. Then there's the whole doctrine of equity thing over the top of it. In law there are always a whole bunch of fine-grained questions that can change the whole outcome.
I expect one or more lawsuits to emerge, maybe as a class action. It might not come to trial and simply wash up as a settlement.
Both parties presumably had lawyers writing ass-covering contracts, so who loses? This isn't the usual "Wells Fargo screws you over then forces arbitration where you lose again" imbalance, there's no particular reason to think Smyte could have swung a favorable contract with e.g. ZenDesk.
It really depends on who was negotiating, whose paper was used to write the contract, and how much each side wanted the deal. Legacy enterprises like Disney often force vendors to use the legacy company's paper, which case substantial protections would be written in or the deal would not happen.
Self-service web services on the other hand generally come with take-it-or-leave-it conditions of use. When's the last time you redlined Github terms of use when starting a project? ;)
More companies than that, and large ones. These are just the ones that went public... some of them don't want you to know their trust/safety/moderation features just went dark. Smyte just ruined a lot of people's day(s).
Twitter owns Smyte. When you buy a company, you acquire all of its responsibilities. There is no longer a Smyte to point the finger at. It's all Twitter.
Legally, the onus is on Twitter for pulling the plug on an API that was being used actively without much notice.
I’m actually surprised that it was done in this manner, as many startups have a “business continuity” section in whatever contracts are drawn up with customers, detailing the specific steps and timeframe of retiring a particular service. I would have thought this is standard practice for a SaaS company.
My point was that we shouldn't point the finger at Smyte because that lets the blame drop on the floor when the "Smyte" brand evaporates.
Twitter owns this now and the way to ensure it's their reputation that is affected over the long term is by correctly calling this a "Twitter" failure.
What about blaming the companies who decided to depend upon a service that they have no control over? I find arguing against using third party APIs to save a few days of development a constantly losing battle, with it being presented as a benefit with no costs or risks. Do those pushing/agreeing with such a view not share some blame?
> What about blaming the companies who decided to depend upon a service that they have no control over?
Unless you're advocating against any kind of *aaS, this is kind of a silly argument.
A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
Obviously when you make your product/company depend upon these things, you're taking a risk that they could go away, but you protect yourself against that by having robust contracts.
Outages are different, but for a significant outage you expect to see some kind of RFO and explanation of how they're going to mitigate it.
Deliberately withdrawing service for all customers with 30 minutes notice? That's entirely the fault of whomever is in charge at Smyte.
Even 30 days, while it'd probably ruin some existing plans, would at least allow people to have a chance to migrate without things just breaking.
>A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
For the cases I was involved in, there were options of buying the service but hosting it internally with some update pipeline. So even if the updates were suddenly turned off, you have a far longer time to swap to a different system. Generally deploying these aren't to much time, but what is really the issue to me isn't that the options that were chosen were the ones which were chosen, but that the process of choosing them did not evaluate the risks of creating such a strong external dependency at all. Its one thing if it is a risk taken with full knowledge of the potential costs and benefits, it is another when people do it because they think there is no possible downside.
No website or app should ever call out to any third party API
Definitely. Your website and app should only call your APIs. Your API can then call the third party API from the server. It's a lot easier to change something on the server than having to update the app.
I wanted to argue with you for posting this, but you're right, this is a smart way of handling API calls.
It's not really addressing the whole issue of depending on third party APIs if your API is calling on them, but you are technically correct, which I've heard is the best kind of correct.
Per the article, they did at least have long-term contracts locked in with Smyte. That's not the same as control over the feature, obviously, but plenty of companies do business on the strength of contracts instead of vertically integrating their entire product.
Obviously we don't have many details yet, but I think there's a real difference between simply using a third party API that works for the moment, and buying a service from a third party. If Twitter/Smyte simply broke contract with all their existing customers, that's very much their responsibility.
One way that's sometimes done is that you require them to submit an independent respectable audit of their financials, or in case of confidentiality issues, not submit the actual financials but a statement from a respectable auditor that they have verified the suppliers financials and their cashflow/"runway"/etc satisfies the criteria that you required.
I'm not denying that what they've done is really bad, but who puts a 3rd party API in a critical path without having any fallback (alternative service, reasonable timeouts, etc.) for when it goes down (as almost any service will eventually do)?
People who have multi-year contracts with those API providers, probably with an SLA detailed in the contract. Especially since this API call would have to be on the critical path; you don't want people signing up when the anti-spam service isn't running.
2. If it's that critical, you have redundancy (an alternative provider, or a naive implementation that maybe causes more false positives but that will cover you during an outage)
I might be wrong, but I'm getting the sense that you don't have much actual experience.
I've been a senior developer in an organization and known that a solution isn't perfectly robust. Clearly I want to fix it because I (like most other developers) like building robust solutions, but I have six weeks of tasks to finish in two weeks, and that isn't one of them. What do I do?
You might say "do it anyways" and, if so, that's a junior developer attitude. I'd strongly urge you not to do that as shipping big projects is really more about choreography than engineering.
Or, you might say "fight for permission" and that's not a bad idea, as long as you accept that you'll likely lose.
You'll likely lose for business reasons. On some level, we all know that a SAAS on a critical path is a bad idea, but we do math. Our SLAs are not 100%, but we can accept a few hours of downtime a year. And, the probability of a service you've seemingly vetted being acquired and shutting down this quickly is low. Is the probability of a shut down like this high enough to justify adding and testing redundancy?
The math usually works out in favour of bug fixes and new features > redundancy.
In the first place, it was an awful business decision to take the route of trusting a SaaS provider with such an important task.
It is extremely irresponsible to forward sensitive user data and site interactions to a third party, even if it is in the name of spam/abuse/scam filtering. That its implementation brings down production websites in the case of service failure only adds insult to injury, originally caused by plain-awful product design.
If there's one thing that you should be doing in-house when you run a large online community of privacy-conscious users, it's the kind of service that Smyte provided. If sound business reasoning had prevailed, npm would have suffered no downtime.
It's easy to say that now that everything has gone to hell, but in business, we usually have to make decisions with imperfect knowledge.
I don't agree that it's irresponsible to forward sensitive data to third parties. Rather, I believe that intelligent companies have intelligent policies around sharing data with third parties. Some third parties provide an amazing service that would be extremely expensive for companies to mimic in-house. And, you cannot convince me that the probability of a well vetted service shutting down as fast as Smyte did is high enough to justify bringing this in-house when you have other things to work on.
No, but if I were npm, I'd have an SLA guaranteeing three-and-a-half or four nines of uptime (which is far from unreasonable) and just accept that there might be times that signup is not accessible because that'd only cost me a couple hours of signups a year.
And no, I wouldn't have an alternative provider. It would not make sense to, if I'm paying a company to provide a service. Why should I pay another to do nothing?
This whole thread is moral posturing about how unprofessional Twitter/Smyte have been, but if we want to talk about professionalism we should look at ourselves.
If you upload to AWS S3 you put in a fallback to another service? And any other AWS services you use? That's a lot of fall backs. Is the second fallback good enough? Should we add a third? In any practical sense development would grind to a halt. Twitter/Smyte are totally in the wrong for not giving a reasonable notice. I think whoever made this decision should actually be removed from their post and the servers should powered back on again with an apology to all of their clients. If they still want to shut it down later then they should provide a reasonable time frame.
Maybe I'm in the minority, but I take an interest in the latest software/tech/ideas for dealing with the hard problem of moderation, and I had never heard of Smyte, until now (which made me think TC was being literal with its "Twitter 'smytes' customers" headline).
Apparently, they were a dominant player in this space, such that its shutdown impacted such a wide array of significant-sized companies, from ZenDesk to TaskRabbit to Meetup. Even more surprising is that they were part of YC (W15), and yet from what I can tell, have only had one significant mention on HN from 3 years ago (65 upvotes, 11 comments)
I would've guessed that Smyte would've been the kind of Silicon Valley business that was made for media hype (what with all the attention/criticism given to fake news and Google/FB/Twitter moderation). Then again, it does seem that content moderation -- important as it may be -- isn't a field as sexy as AI/deep-learning and other forms of automation.
They were also noteworthy for their founder Pete Hunt, who is largely credited with instigating the open sourcing of React. (As I recall, it was internal Facebook tech that he extracted into a separate library when he moved from FB to Instagram after that merger.)
Given that these businesses had contracts with Smyte, how is this not a flagrant violation of the contracts? Surely Twitter took on Smyte's contractual obligations when they acquired the startup?
In a case like this, does that mean that Smyte (the company, even if they no longer own the trademark) still exists and is liable for all the penalties? Would they usually ask for the cash to pay for them as part of the asset purchase?
In a case like this what's left other than an empty sack? Unless you're able to pierce the corporate veil there isn't anyone/anything with assets left to be made whole by (e.g. sue).
IANAL but I was a plaintiff in a similar case. We were made whole in a settlement but before that, the judge tore the defendants a new one over this tactic leading to a bitter dispute between the acquierer and shareholders (or the board of officers, don't remember) over who was liable for breach of contract. Eventually it came out that the auditors had done the math and included cost of litigation/liabilities on these contracts into their valuations, basically admitting that they knew about the obligations beforehand, so the acquierer ended up settling.
At the end of the day, judges (and appellate panels) have to interpret the law and they don't function like automatons. They take into account the spirit of laws/contracts as well as the letter and most have enough common sense to get missed off at these naive tactics.
The argument that penalties should be paid out of the purchase price would probably hold water. When Smyte entered into those contracts they took on some duty to honor them and to suddenly dishonor them and divert all the assets out of the company so that penalties wouldn't ve avoided would be constructive fraud.
Well, usually they are using the money to wind down.
(All of this is a good reason i hate LLC's. They aren't necessary anymore to actually do risky innovation in 90%+ of cases. In a non-LLC, they shareholders would be liable, and then you'd still have someone to go after)
This is the thing that occurred to me. Surely those contracts don't have a clause that says "Smyte can terminate all services with no notice"? If so, surely someone spotted it by now? Or is this normal in service contracts in the US?
I guess the liability is covered by the usual "no liability outside provision of the service" clause - so you can't claim that you were damaged by their lack of service. But I can see a lawyer making a convincing case that the liability should rest with Smyte. IANAL, obviously, but it seems like the sort of thing they love to fight about ;)
Perhaps the default terms were Smyte-friendly, but I can't imagine that some of these big companies (with savvy lawyers) would sign an agreement with a change-of-control provision that would permit this.
There might be a carveout for consequential damages (which would be huge here), but this seems like it will trigger a number of sizable lawsuits against Smyte's new owner.
So... what was the Service Level Agreement? and the Terms Of Service?
People get all hyped by -aaS stuff, but services do go down, and if you aren't the one controlling when and how they will go up, then you better have a contract with the Service Provider on some terms you like.
Even then, prepare for when they go down, because they will sooner or later, for shorter or longer periods of time.
I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
I find it silly to rely on some service, any external service
This is always trotted out and it's, by this point, a completely content-free thing to say. Short of removing yourself from the economy and society in general to live off potatoes grown in your own dung, you're going to rely on some external services and the fact that, for the most part, the parties you do business with will behave responsibly. When they behave very irresponsibly, unpleasant things happen and people quite reasonably complain.
Exactly. You want reliable CAPTCHAs? CDN service? Malware detection? Customer support software? Website uptime monitoring? Accounting software? Analytics? Email delivery? Good luck building and running all that yourself.
I don't know why you're getting downvoted because you're absolutely right. Perhaps most of the people in this thread haven't managed servers for more than a decade - because it used to be (and still is in some organisations) the norm to self-host half the stuff that was listed.
* Malware detection?
Isn't that pretty must how most malware detection works? Sure they use the cloud to pull periodic updates but the software itself is self hosted
* Customer support software?
I've lost count of the number of solutions available to self host. Both free and commercial.
* Website uptime monitoring?
While there is a strong argument for hosting that in the cloud, there are umpteen solutions for website monitoring.
* Accounting software?
I didn't even realise running accounting software as a service was now a thing!
* Email delivery?
Fair enough, self hosting email can be a massive headache. Less so if you run Microsoft Exchange (one of the few things Windows actually makes easier than it's Linux / UNIX counterparts). You can even go for a hybrid approach and self host the mail server but have a mail proxy between your server and the outside world - so you benefit from spam detection as a service while still hosting your own mail.
I'm not going to argue that one is better than the other when it comes to SaaS vs self hosting. But people in here seem to have very short memories when arguing that self-hosting isn't at all viable. In fact I have personally supported all of the above in self hosted versions in the last couple of years.
I don't think anyone is disputing that it's possible, but rather how cost-effective and time-efficient it is [1].
I also have the experience of managing many of these services, and I therefore know just how much more productive you can be when you don't need to do it yourself.
Of course there are trade-offs; business is nothing if not a game of tradeoffs. We're just talking about what is optimal.
[1] Not to mention how much more stable and secure; Self-hosting can involve huge security and reliability risks.
I don't believe what is optimal can be generalised like that. "Optimal" is going to be specific to the business in question. Which is why I'm a firm believer of leaving all the options on the table.
> Self-hosting can involve huge security and reliability risks.
In fairness so can the cloud. I personally don't see difference as 'risk' but more 'responsibility'. Some of the places I've worked have been PCI DSS and Gambling Commission audited so we had that responsibility already. Self hosting a few other resources didn't really add much extra in terms of our security responsibilities. But the case would be very different of lots of other different types of businesses.
> It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.
Even that strikes me as a mischaracterization, or, at best, taking just one statement out of context.
It was an assertion about the folly of relinquishing control, not being prepared for that lack of control, and thereby subsequently suffering a prod outage.
hosted where? On a VPS, in a data center, or in your closet? All of these depend on third party services and good luck keeping them up and running as well as someone who makes just that ONE little thing their entire reason for existing.
Yes, any of those sounds like a good enough plan B. You don't need it to be perfect, just good enough to avoid getting stranded in a pinch. Then you can start shopping around for someone who does that one little thing much better than you do. That way it no longer is a show stopper and rather just an optimization problem.
I don't agree with the parent either (I'd rather use a SaaS if it makes sense) its also not as easy as you make it sound.
For starters, most smallish companies I know of use housing for their servers. So, they actually do own the hard metal servers and have full access to them. Both over SSH and on the console for debugging purposes. The housing provider generally replaces HDDs in case of disk failures. naturally, you're still dependent on electricity and internet, but even that can be mitigated by spreading your servers across several locations.
All of that is obviously possible with AWS, GCP or other VPS hosts, but doing it yourself is not that much harder if you've got an experienced ops person that wants to bother with all of that.
but i repeat... Its often not worth the time investment to get that system properly set up. Just renting VMs or going straight into dockerized microservices is an increasingly better alternative
No. Whether it's potatoes or servers, having a single point of failure with no plan B in place, is silly.
Your local restaurant going out of business is no reason for you to starve to death, you just go to another one, or to a grocery store, or indeed grow your own food if you're in some post-apocalyptic world or got no money... but somehow a service provider going out of business is reason enough for your production to stop? No, just no.
Indeed, it's spurred a discussion of an over-simplification of the issues.
I don't think anyone is suggesting never to outsource, or even that the answer to "build vs. buy" is always "build".
The point is not having a critical dependency on a single vendor.
I don't think anyone is even saying "never", either.
The question is one of risk.
The risk of your tax accountant "going down" (or being crooked or something) is sufficiently low, and the cost of making that redundant is sufficiently high, that I don't think anybody does that. This could even apply to general accounting software, though one might argue that's not critical in the same way and/or more easily replaceable.
There's a sibling sub-thread that essentially implies, if not outright states that there's no choice but to "buy" in "build vs. buy". Although it may be true that building is no longer a practical choice due to time/cost, under certain circustances, it by no means eliminates the risk if there's only one vendor.
It's also a decision that can end up having tremendous costs at scale, if not re-evaluated, especially with even a soft version of vendor lock-in, like AWS.
That's not what the comment I replied to said, though. It just said it's 'silly'. If they had said something specific about, let's say, npm and made some reasonable and knowledgable comment or even speculation about how in this particular case they'd mis-assessed risk, that would be a pretty interesting comment. As it is, it's just a clone of the same tedious comment we get in droves in every thread about some multi-party outage. They're generic and free of insight.
It doesn't really change the quality of the comment. There is nothing inherently and outright wrong with such a dependence, whether it causes an outage in prod or not. In some cases it matters (maybe you're making a nuclear reactor) in many, it's a completely sensible tradeoff. Reflexively saying 'zomg you depend on something else and this is bad' every single time something is affected by a problem in another thing is 100% uninteresting. Including or not including 'in prod' or 'in bed' doesn't make it any more interesting or clever. It's just lazy grousing.
> There is nothing inherently and outright wrong with such a dependence
This is yet another strawman. The original commenter found it "silly", which, I agree, was a poor choice of wording.
Perhaps "overly risky" would have been better. I don't know. I just take the most charitable reading, per the guidelines.
It also comes after an exhortation to have a contingency plan, so it's at least implied that the "silliness" isn't inherent merely to the dependence but to the dependence without such a plan.
> in many, it's a completely sensible tradeoff
Is the tradeoff actually considered, though? Or is there just an automatic "we're not a nuclear reactor" decision process?
> is 100% uninteresting
Were that true, I doubt there would be replies. This is not one of those traditionally emotional/political issues.
> Including or not including 'in prod'
I (again, charitably) read "in prod" as a metaphor for "business critical". I'm sure we can all come up with examples, even in Internet companies, where one does not necessarily mean the other (or vice-versa). In the instant case, it seems to apply, so it made sense as shorthand.
> It's just lazy grousing.
I would agree if there were no built-in suggestion on how to avoid the problem at all, but there were.
Just because the point has been made before doesn't make it any less valid, until it has been refuted.
Only if 'strawman' means 'thing I disagree with'. The comment says such a dependence is bad and how you have to prepare for it with SLAs or whatever. This isn't universally true at all.
I doubt there would be replies.
A lot of really boring, trite things generate piles of replies. That's why they should be avoided.
Just because the point has been made before doesn't make it any less valid
The stated goal of the forum is not 'an exhaustive, if repetitive collection of generically valid things'. Remember that time you wanted to print something and the thing just wouldn't print? That's bad and annoying. It's valid that it's bad and annoying. We probably don't need to talk about it in the general sense every time it happens, though.
> This is always trotted out and it's, by this point, a completely content-free thing to say.
Only if you pretend there are no companies that do in fact responsibly nail this stuff down. There's plenty. Your argument seems to be that nobody does it because it's hard.
Welcome to the world of third-party Twitter clients. Not content with screwing over businesses relying on Twitter's API, it seems, they have now set out to screw over any company relying on any other third-party API as well.
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
There are pretty solid SLA's for that. We provide some services as kind-of-saas but the SLAs are watertight.
Usually in those kind of sla's you can't sell the company (or move assets) without guarantee of delivery of the contract. In some cases they (the contract holders) even own you if you screw up and go under.
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
I'm curious; what would you have these companies do in such a scenario? If 100% uptime is impossible then what you seem to be suggesting is that people should never use third party services or if they do use them they must not be in a critical path.
Are you suggesting everyone should do everything in house?
This was in order of preference. If you can, use free software that you can fix and support yourself if need be. If not, buy the software and run it yourself, so you can at least control its operation. The last option is to by SaaS, and if you do, you need to have contingency plans in case it goes away. You should take this risk seriously, and factor it in to your decision when buying SaaS, and invest the engineering resources to make sure you aren't overly dependent on it.
For any critical service, they should have an alternate provider ready at all times so that they could instantly switch to it. Whether it's another external provider or in house, is not really relevant, although having a bare bones in house alternative for the critical parts is a good idea.
What I'm suggesting is that people should use external services for their lower costs, better performance, or extra features, not to 100% depend on any of them.
No. Smyte wasn't so business critical that they should consider a failure in the service a reason to stop the site. Netflix had an example where if thier authentication service was down, they didn't stop people from watching videos.
There is a popular C# package called Polly that has all sorts of fault tolerant strategies.
I feel like there must be more to this story. Maybe they immediately saw security problems, or legal issues. Not saying it's going to exonerate Twitter (maybe the truth is even worse than it sounds), but this is such an obvious PR mess that I can't see them doing it for no reason.
I'm calling it now: B2B startups will soon have to sign poison pill contracts that specifically detail what happens if they're bought out. This might ruin the "buy them for their people and discard the business" method of hiring - or at least it will if the customers have any sense.
I'm baffled if this wasn't already the case. Surely the entire point of signing a long-term contract (as opposed to just paying by the 'unit' used) is to protect against exactly this kind of outcome? I know it doesn't always happen, but it seems like most rapid "hire and shutter" moves hit either consumer-facing services or B2B stuff like analytics that isn't production critical.
I had assumed Twitter was going to be paying out a bunch of breach-of-contract penalties, because this is basically what contracts are for. Is there a standard way around this?
I don't know how it works in Silicon valley, but finance companies I have worked for have pretty much always required that source code be turned over if the vendor folds when doing business with small vendors.
It's amateur hour if your contracts do not already contain such clauses. For anything business critical that's the norm. Change of control clauses are super common and tend to trigger penalties, escrow arrangements, contract voidance and other niceties.
This is incredibly fragile ecosystem if one obscure API getting shutdown causes so much damage.
This could be prevented by a simple periodic check that would determine if API is available and some fail-safe alternative to remain online.
Instead this is like a house of cards that breaks with any of the cards fold.
I'm not excusing the abrupt shutdown, its obviously a wrong way to end service, but being prepared for it is much better.
This is a hard call when the system is question is your anti-abuse provider. If you "fail open" when they're down, you risk allowing a flood of bad users during outages. Depending on your business, that may be much worse than having an outage yourself.
You can turn on something like manual verification of new users, an alternative security service, or just temporary halt new account registration. All of which don't result in system wide failure.
Even a few abusive users is a small and temporary cost to absorb vs complete outage.
I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way.
What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.
What you're suggesting is something for large companies. Most of these companies affected probably don't have time and resources to develop and pay for alternatives to every single third party service they require. And why would they? They have contracts you'd expect to be honored.
Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.
When I work on systems, I ask, "how does it scale and how does it fail?" Everything I work on considers handling of failure cases, especially 3rd party APIs. I'm surprised by how many people assume that the network will just work and their provider will be available.
I think everyone is missing that this is the canary in the coalmine for how the tech giants are going to behave going forward. Hell, Twitter isn't even that big!
Every tech company big enough probably sees the same writing on the wall the rest of us do that it's time to press their advantages. That means shutting everyone else out.
We probably all have mission-critical parts of our infrastructure that aren't core competencies. We probably can't afford for them to go away tomorrow, but now it's time to expect and plan for it.
While this should be upsetting, and I'm sure it was to the people who were affected, I'm actually happy to see it. I'm of the mindset that this will have an overall positive effect on the state of the Web. Why? Because users and devs both need to remember that companies are not friends, and contracts are only as enforceable as the courts will allow.
Imagine if Google decided that tomorrow at 6AM PT they were pulling the plug on YouTube embedding, or GMail / GDrive integrations. The court's going to say, "They put in the contract that they can change the terms of service at any time, and they're not liable for outages or lack of service. And you signed it. Sucks for you."
Then again, I'm just anti-corp in general and this gives me better reason to avoid Twitter.
There's a difference between Google turning off a feature and Google cutting off a whole service. This is the kind of thing negotiated with contracts. Lawyers on both sides. If you build your business around another company, you protect yourself. Twitter with surely get sued for this, if for no reason other than failure to provide a service that they promised to customers in writing. Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.
> Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.
This will be how it turns out for a few big players with deep enough pockets. Surely though this is hurting somebody's business that can't afford the lawsuit -- potentially to the point of putting them out of business. That's good for Twitter (and especially Twitter, as precarious as they are).
We've all joked for years about how FAANG pay their engineers so much mostly to keep other companies from being able to hire great engineers. We've gotten the big three slapped for colluding against their employees on wages. Why do you think they would engage in some anti-competitive business practices and not others?
Couldn't you file a amicus curiae, advice the court and state the company has harmed multiple people which are unable to defend due to cost, wait for the case to settle with the big guys and then sue with reference to the original case?
I'm not sure how good apple is, but when they buy software available on Windows that version usually gets killed. (See Logic music production, killed 3 months after purchasing, but presumably the old installs still worked.).
Even Halo was demoed on mac before MS bought it (The mac version never came out)
Doubtful. For most long-term corporate contracts there is no "we might cease operations at any point and leave you hanging before end of the contract".
No company lawyer I know would willingly put their signature on such a contract unless the other side is using a free tier. If someone actually signed 3 year contract with such a clause, congrats, you managed to throw a lot of money into a black hole.
For a free product, sure. But when you're signing a multi-year contract where money is changing hands, no way in hell would any customer accept that kind of clause in the contract.
What other unpleasant things would you wish on people as a way to educate the public about the realities of contract law? Your internet service just deciding to cut you off with no notice? Water? Electricity? Ambulance or firefighters not showing up when you need them? That might help cull the hopelessly naive from the herd and have an overall positive effect on the state of the Web.
I’m aware of your sarcasm but will reply at face value:
1) If your business depends on Internet access you shouldn’t pay for a residential plan. You need an SLA.
2) Natural disasters happen. Not having emergency drinking water is very naïve.
3) If your business depends on electricity you should also get business rate with SLA and maybe have a backup generator or two. Heck, I have a UPS just to not lose the 15 minutes of work that’s not automatically backed up.
4) Yes, get a weapon (preferably a gun) and at least know enough first-aid to be able to stop a bleeding. Keep an aspirin on hand in case of a heart attack. Know your emergency exits and evacuation plan. Make sure you have the mindset to leave all your possessions behind in a burning inferno.
All of the above used to be common sense before technology made us think we are immortal and entertainment distracted us from real world.
I'm not sure that's a reply to something I said. The poster thinks a crappy thing happening to someone is good for the web because, I dunno, it's some sort of teachable moment. That's pretty silly, to put it politely. I'm not wishing you an earthquake so that you'll learn basic disaster preparedness. And if you did, fates forbid, end up in an earthquake and were short on water, I don't think I or any sane person is going to be happy about it.
I wouldn't be happy about any loss of life, you're right. However, has the Smyte situation led to anyone coming to physical harm or put in mortal danger? If it has, I'm sorry, I didn't see anything about that.
What I've seen was a select group of companies and their users who've been inconvenienced by the abrupt shutdown of a service; and whose unpleasant experience should be used to teach every user - this can happen to you. I would not be at all happy if, say, the servers for medical tracking software used in hospitals around the world went down the same way; or if suddenly air traffic control software that every airport uses goes offline.
This particular situation doesn't cause potential physical harm to a human being. It's not actually an emergency situation. That, to me, is a major difference.
What if it’s a little earthquake? It will make you think twice about building your home on a bad foundation.
I think the issue here, other than earthquakes being potentially very deadly, which makes this comparison a little absurd, is that they are in a sense unavoidable.
On the other hand you can avoid using SaaS.
For example, there’s enough market demand for Github to provide on-premises secure hosting. Clearly then, if SaaS as a whole is seen as fundamentally unreliable it will create more market demand for similar on-premises, or open source, or otherwise “buy not rent” model services, which I see as the motivation for OP’s comment.
I mean, we can stretch these analogies like pieces of tasteless gum all day. I don't think being happy at the misfortune of others is a particularly constrictive attitude nor do I think Twitter buying and then apparently shutting down some service with 20 minutes notice is good for the web, teaches anyone anything or is in any way a desirable outcome. I don't understand how any reasonable person would.
That's really amazing - I am having great difficulty imagining a situation in which doing this would make sense. Just for starters, even if you completely discount goodwill, ethics or even PR, I'm guessing Twitter just bought a bunch of lawsuits. This just seems nuts.
Twitter seem to have an institutional blindess towards the importance of APIs. Their execs need to be sat down and have it drummed into them that breeding this much bad “karma” (for lack of a better term) is going to come back and bite them in the ass.
> Twitter seem to have an institutional blindness towards the importance of APIs.
Which is especially weird for Twitter. Their official iOS client was a third-party API client they bought. Third-parties created @replies, #hashtags, etc. If anyone should recognize the important of third-parties using APIs, it should be Twitter... yet they've been probably the most abusive of the large APIs.
This is the same company with NIH Syndrome severe enough to in-house their own woefully-under-performing message queue.
Twice. And they blamed Rails after the first go of it.
They had contracts that they took on when they acquired the company. Twitter are now legally liable for these contracts, I won't be at all surprised if someone sues them. And frankly, they would be in the right.
What an absolute debacle. This doesn't bode well for the entire Twitter platform.
I went through an acquisition of an unsuccessful company. My employer created a shell which is what was acquired. The old corporation wasn't acquired and existed as an independent entity to wind down contracts / deal with the acquisition hold-back, etc.
If eg smyte was out of money, this is plausible. And all of a sudden no-one was there to run the old corp.
crunchbase says they raised a $4m A in Mar 2017, but linkedin says they had 23 employees. They could burn that in 1.25 years with 23 employees and have hit a cash crunch.
Don't know to which jurisdiction smyte belongs, but in many countries creating figureheads/companies, dumping them, firing all employees and then buying the leftovers and rehiring people is considered the same a acquiring the company. E.g. finding ways around laws does not make them invalid and (sometimes, not always) you have to follow them anyway.
But if it is all within the states, it could be just that.
> If eg smyte was out of money, this is plausible. And all of a sudden no-one was there to run the old corp.
Sound more like a tech/knowledge acquisition to me.
4M$ for 23 employees only works if you have proper financial management though.
That is not a good structure for the acquisition, especially if you are on the hook for the shell. Quite often you have all of the responsibility to deal with shit hitting the fan but no resources or personnel to actually do it.
> Because the reverse triangular merger retains the seller entity and its business contracts, the reverse triangular merger is used more often than the triangular merger.
I was going to say "I wish that I could do that!" But then realized that bankruptcy comes pretty close.
But why would a judge not go, "No, Twitter, you clearly acquired the company. Just because you broke it into pieces, and then picked up all those pieces does not mean that you didn't acquire the company."
That is (obviously) the trick: the "acquirer" doesn't acquire the shell. And therefore is on the hook for nothing. Customers have contracts with a corporation with no assets and no employees.
It doesn't work that way. You can't just transfer assets into a shell and leave the liabilities. Any lawsuits would be able to go after the assets, even if they were in a separate shell companies.
You can't transfer assets into a shell, but you can buy certain assets you want (e.g. IP) for a reasonable price even if it is much less than the liabilities, and take over the employees (which aren't "owned" by the company and thus an asset), and leave the old company to bankruptcy.
Also, it's quite plausible that Smyte had negative value before the acquisition - that it's going to be an "empty sack" not because valuable assets were transferred elsewhere (which can be reversed in courts) but because its debts already exceeded any value it had, that it was an empty sack in the first place and no debts can be paid in full in any case.
* If the assets were bought, they'd be the property of the original company, and the money would go to the company, which would therefore have money to be sued for.
* If the directors took that money out of the company, full in the knowledge that they had legal liabilities as the result of terminating service, at least in the case the UK (and I can't speak for US law), they'd be guilty of a number of crimes, as they are responsible for acting in the best interests of the business.
What I mean by "its debts already exceeded any value it had" is that it's quite plausible that any money paid for the assets immediately goes to the company's current debtors (including some investors, if convertible debt was used instead of equity) to partially cover their existing debts (which would be a fully legal and correct action by the directors), and there's nothing left to be sued for future liabilities.
There are startup acquisitions which result in lots of money being paid to investors, and there are acqui-hires which amount to a pre-bankruptcy firesale to get part of the money back, but resulting in unpaid debt and bankruptcy anyway. I don't know about Smyte's financials, but it may well be that their situation is like the latter case.
I don't see any reason to suppose that there some big bag of money (even post-acquisition) that's somehow illegally taken out of the company. It may well also be that the money paid to the company itself for acquisition is insignificant, and the rest is in form of some "hiring bonus" to make the transfer of employees work, so not something that the debtors can claim.
Obviously this is massively speculative at this point in the conversation, largely a thought experiment, but --
Any side payment to the directors and staff outside of the acquisition - such as a "hiring bonus" as you suggest, which lead to the sale of assets at a lower price than would otherwise be the case - I'm pretty certain (again, in UK law, and I'm certainly not a lawyer), that this would be seen by the courts as a bribe, ie. money paid to perform their duty to the company improperly.
In my failing startup that was acquired, all eng got a carve out. It was basically a bonus to get us to remain employees after it became obvious the company was failing and was going to be sold.
I'd imagine the legality goes like this: without that bonus, I certainly would have left. And there would have been nothing to be sold. So it was long-term better for investors.
It's only legal because there is a legitimate reason to pay it. The board members in charge have a fiduciary duty to the shareholders.
So you will see payouts and bonuses to executives to get them to stay with the sinking ship in hopes of eeking out more money before the collapse, but if you were to just pay a bonus to an executive on their way out, you would have a very hard time justifying that and could be sued for breach of your fiduciary duty.
The original company could have argued that the value received for its assets was less than its liabilities. It’s perfectly legal, but way overused in tech (where assets are overwhelmingly in the form of IP without an easily established market value).
This type of uncertainty does lead to a situation where potential customers are wary of using a startup service — I can’t build a business off your service if I don’t have confidence your company will want me as a customer in 3 years.
Ultimately I think we’re witnessing the final consolidation of the software industry. I expect the cloud/software industry of the next 30 years to start to look much like telecom over the last 30 years.
Probably worse for consumers short term, but ultimately the plumbing isn’t all that interesting and it will hopefully enable the next round of innovation.
This seems like by far the most likely case. That said, Twitter should know better than to let this happen, anyone could have predicted that the optics would place a lot of blame on them. I can't imagine Twitter couldn't support an additional 7 days after giving warning.
It's not unlikely the costs of work performed for these customers wasn't enough to keep the company running. Honestly, it looks more like "let's cut down on costs, immediately", rather than a mistake on a part of an exec.
It's very easy to have revenue but no runway. If your monthly burn is $1M and monthly revenue is $800K and you have no cash in the bank, you have no runway.
Ideally, yes. In practice, it depends on the contract. My bet on this is its provisions were pretty weak (and definitely not multi-million as it's suggested elsewhere) so Twitter's lawyers went through it and said "screw that", knowing that any lawsuit isn't going to be successful.
There are a few reasons why it's probably not entirely Smyte's fault. Given that Smyte comms dropped off over the past month, as another commenter mentioned, there might not have been anyone around to warn the users.
And if you're thinking "well they could just personally reach out to their customers if they're fired because it's the right thing to do" -- if the last-minute nature of everything is any indication, current/former employees were probably ordered not to say anything.
Twitter bought a company, then found out that is isn't production-ready because it doesn't handle customer data safely, so immediately shut down the service.
I wonder if that triggers any clawbacks in the purchase price, as Smyte was running a substandard service.
Even if the assets were acquired, I wonder why you can't sue the remaining shell and get a cut of the money that was exchanged for the assets.
I guess the real issue is that smyte's contract probably said something like (from GCP's terms of service):
> 13.1 Limitation on Indirect Liability. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NEITHER PARTY, NOR GOOGLE’S SUPPLIERS, WILL BE LIABLE UNDER THIS AGREEMENT FOR LOST REVENUES OR INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL, EXEMPLARY, OR PUNITIVE DAMAGES, EVEN IF THE PARTY KNEW OR SHOULD HAVE KNOWN THAT SUCH DAMAGES WERE POSSIBLE AND EVEN IF DIRECT DAMAGES DO NOT SATISFY A REMEDY.
> 13.2 Limitation on Amount of Liability. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NEITHER PARTY, NOR GOOGLE’S SUPPLIERS, MAY BE HELD LIABLE UNDER THIS AGREEMENT FOR MORE THAN THE AMOUNT PAID BY CUSTOMER TO GOOGLE UNDER THIS AGREEMENT DURING THE TWELVE MONTHS PRIOR TO THE EVENT GIVING RISE TO LIABILITY.
Twitter continues its insanely tone-deaf approach to dealing with third parties that interact with / rely on their services. I don’t expect anything more from them, but am surprised Smyte sold out their users just like that, as I’m sure their fate was part of the negotiations.
Given there are actual damages coming out of this to paying customers perhaps we’ll finally see them taken to task.
If I was including an anti spam/security service as part of my pipeline I probably would want my system to stop if it went down, rather than just skipping the protection. Especially as cutting me off from said service is a possible attack option for a bad actor.
I could imagine (just guessing): submit new repo -> scan with Smyte -> on success, publish repo. Maybe not handling an error properly and skipping Smyte.
This problem creeps up with external mfa/2 factor auth APIs that go down, bringing down the main login. Some choose to skip mfa if down, some don’t
I mean, sometimes handling an API failure correctly is failing over. If an API call actually matters, then when the API stops responding, throw an error. Even if you're probably catching that and giving out an error message, I'd still call that a production outage.
We have a differing opinions on what outage means then. Catching the error and showing it to the user is obviously the right thing to do, but not what I'd call an outage.
I can't say I've ever met a developer who thinks that a 100% error rate to valid requests isn't an outage. Not sure why you have such a strong view of your semantics.
Sure it is, if it affects all users at once, and makes your system unusable. If I dial a phone number and get a busy signal, that's an "error" of sorts. If every Verizon user gets that busy tone on every call, that's an outage.
There are lots of potential security issues with npm (compromised accounts, spam packages, etc). I think failing closed is the right thing to do for them. It's a temporary annoyance and loss of productivity for a day - a good time to put your feet up and relax. Failing open could lead to a costly security breach that affects many people, and could lead to npm's demise.
npm gets a lot of flak on HN, but this seems fine (or rather, it should have been fine) given that they and others apparently had multi-year contracts with Smyte. What kind of situation would you be in if Github went away, for comparison?
This seems like a clear breach of contract. I can’t imagine how anybody thought it was a good idea. Maybe it was a mistake, although that wouldn’t be a whole lot better.
After hearing so many stories like this for so many years, any time a company acquires another, it usually seems advisable to jump ship from the acquired to other vendors before things start going south.
PR will come up with all sorts of reasons said acquisition will benefit both company's customers but I've yet to see it.
Oooh.. I'm glad they screwed over well-funded companies with multi-year contracts. Now at least they're going to get the shit sued out of them unlike when they go after regular users.
How's the legal situation in America regarding breach of contract?
At least in Austria we have a law that exactly tries to prevent such things (§38 et seq. UGB): when you buy a company you enter into all of their existing (non highly personal) contracts and obligations, unless you specifically don't want that, in which case you must tell the other side and give them some time.
I am not lawyer. I don't think there's a valid loop hole that would pass a judge in a court of law. Either the contract specifies early termination damages or it was neglected.
lmao this is hilarious. Never seen something like this happen with such a large company. Let's hope the codebases out there had a simple integration with Smyte and the change to that "smyte" filter was fast. Crikey!
Hm, one of the tweets mentions a "3 year contract". A contract means two parties agree to something, with penalties if they don't do what they agree to, right? Couldn't they sue Smyte/Twitter or something?
Ah I see the plan here: Twitter knows their platform is a cesspool of abuse and harassment, but instead of fixing it they're pulling the rest of the internet down to their level so they stand out less!
That will make ALL companies think twice about trusting a startup when choosing a SaaS provider. VC got its way but at the same time decreased any future startup success potential by a huge measure.
As multi-stepped and prolonged as it is to sell to large enterprises, I don’t look forward to tomorrow when yet another question to be answered before any large org signs a contract with a small startup like us is: are you going to sell and shutdown?
This has happened so many times that it can really affect how small companies are preceived as too risky too deal with by larger customers.
sigh
Companies can do this, there's no regulation against it and there probably shouldn't be any. But there are going to be consequences and they're probably not good for the industry at large. Moves like this fundamentally change how much trust consumers have in SaaS apps. The risk factor of using new apps fundamentally changes, if ever so slightly. Unfortunately, this arguably works in favour of the companies that do this. By reducing trust in startups, there's fewer users who will take the leap on new actors.
Twitter and Google wins by having fewer competitors, but the customer loses doubly so. If Facebook acquires every fifth social network, followed by immediate shutdown, users will eventually stop using them, hence less competition as a whole to FB.
I assure you they are all paying for promoted tweets, ads, or using expensive social media tracking tools to monitor usage, respond to requests etc. in one way or another.
Twitter is a pretty big part of any modern online company’s strategy, whether they like it or not.
One thing that doesn't make sense is surely a service like this greatly benefits from the network effects of being able to see activity across multiple sites, so if Twitter wanted to continue using the technology, wouldn't they have been better off leaving it running?
Twitter may have sufficient scale that the additional data aren't meaningful.
Friend worked on a large, multi-institutional risk-modelling project ... until the largest institution decided it had enough data on its own, and didn't need competitors who were competitive in this regard, and pulled the plug.
Law of large numbers (statisttics) is powerful. Even for complex (multvariate) models.
It sounds as if there may well be other considerations at play, but I'm answering your question specifically.
I'm not convinced of your premise, spam is adversarial & spam detection is just pattern recognition. Twitter likely doesn't want to allow people to be able to test against their spam shields by spamming other people
As I understood it they were using machine learning models to identify attacks. Abuse and fraud strategies are constantly evolving, so the benefit of having many clients is that once you see a new attack strategy at one, you can instantly protect all your clients from that threat.
In your own example
"Twitter likely doesn't want to allow people to be able to test against their spam shields by spamming other people"
Yes, the attacker may find a new pattern that can bypass the shield, but as soon as that pattern is added to the machine learning recognition, they are then protected against it on their own system.
"spam detection is just pattern recognition"
The more patterns you can analyse, the better your recognition?
The call to wind down business with existing clients is a worrying trend. Why would I avail services of a startup if there is a risk that upon acquisition, the startup's new parent company won't service us anymore. It's a risk.
in twitter's case the closest alternative is standing in a crowded street, barking disjointed sentences at passers by in the hope somebody acknowledges you.
Do you actually need to bark at passers-by, or do you think you need just because every other dog is doing it?
Replacing Twitter is simple. Have a proper support channel (Intercom, etc) so your customers don’t need Twitter to get support from you, and use your existing marketing tools (email, push notifications, etc) to keep in touch with them. Done.
Well, there is mastodon/pleroma/fediverse, in which case you are standing in a less crowded street in your neighboor casually barking disjointed sentences and vaguely familiar passers.
I'm pretty sure that "to wind down" is not a proper euphemism for "to shut down".
"Winding down" literally means that before shutting down, there is an incremental process with the explicit purpose to smoothly move into the transition. There wasn't anything like that, not even a little.
What's especially rich is that Engineering VP Mike Montano nonsensically repeats this phrase after the fact: "we made the difficult decision to wind things down right away". That's plain bullshitting.
He should have said, "we made the difficult decision to shut things down right away". But that would have sounded super irresponsible! But it was super irresponsible, and describing it differently is just dishonest.
And that wasn't accidental, this entire situation is exactly about the difference between "winding down" and "shutting down".
Why do we allow people to communicate in this way? (publicly and/or from a position of leadership). Twisting words in plain sight. There is no good motivation behind this, it represents messed up priorities between appearance/saving-face and responsibility.
I don't know anything else about this Mike Montano. Maybe he's a very good manager when not talking like this. But leadership comes with responsibility and when you twist words like that, it's only because you're trying to wriggle out from underneath the responsibility.
3. Have a test environment where you can delete routes to external services and assess the failure modes of your application.
Does the whole page hang? Do UI elements disappear ot overlap others? Do JS error messages surface in the UI? Can the user continue with core functionality or at least receive an explanatory message?
It makes me wonder. Does anyone actually read the contracts they sign?
I usually hate it when people blame the victims for being so stupid whenever something bad happens. But some of the victims in this case are corporations with lawayers.
So either this really isn't a mission critical service and they just took a calculated risk, or these lawyers haven't done the job they were hired to do, or Twitter is in breach of contract.
Is this really that out if character for Twitter as an organisation though? They have a history of being rather hostile toward developers and their own APIs. Which is ironic considering the nature of the platform.
I would say we should use this as a sign to migrate away but we've had enough warnings already. I guess Twitter is too big to fail?
You could say that an API implements a protocol, but a protocol is much more general. For instance IRC is a well known and widely implemented protocol. Facebook messenger implements a protocol for client server commination, but there are few or none implementations if either side and the main implementation can change the protocol arbitrarily and without notice.
Right, but then wouldn't every API that has documentation also necessarily be a protocol (that might only be implemented by one provider?) If the idea is that you could switch to a different provider if you were using a protocol instead of an API, that only helps you if there is more than one provider of a protocol. In this case, someone else could take the public API docs and implement their own service that responds to the same API.
I would imagine the relationship similar to programming languages and standards; If there's no official standard, then usually for anyone trying to re-implement the language, they're "doing whatever the mainline version does". ie anyone trying to reimplement python does whatever cpython does, bugs and all, to claim "compatability".
But if a standard exists, that is, we've split the API from the primary implementation of the API, then we as a re-implementer no longer need to replicate bugs and all; we just need to do as the standard says, and anyone relying on cpython bugs is, well, at their own fault. Compatibility claims are no longer dependent on implementation-specific details.
And of course, it doesn't matter until a second implementation begins to appear; there's no reason for cpython to follow (or have) the standard unless it actually wishes to be compatible with jpython. At the same time, it makes a lot less sense for a second implementation to appear as well, as they have to put a lot of extra time and effort into matching cpython's implementation details (as far as they matter; a difficult determination itself), and this also makes it more difficult to produce an alternatively-optimized implementation (focusing on say, memory-usage instead of speed or whatever). So it's also a chicken and an egg problem.
So ideally everything would be an official standard, and we'd put little to no effort into simply matching whatever the popular implementation is at the moment, and we could criticize the mainline implementation for failing to adhere to the standard.
In this particular scenario, to replace smyte everywhere, transparently, you'd both have to adhere to the API and whatever details incidentally exposed by the API, because you have no protocol to actually reference (the protocol is whatever smyte was doing). But thats also particularly difficult here, since you can't even test it against an implementation of smyte anymore..
Unless it turns out that Twitter blatantly lied to Smyte's founders regarding how they would shut down the service, I personally will never use any service or software that these founders create in the future. This is absolutely unbelievable, even for Twitter.
[I've removed the founders' names, even though I strongly disagree with the characterization that I was somehow acting out of rage. My intent was rather consumer advocacy. I've had a service cut off in a similar way, and I view such behavior as being unethical. Fine if you disagree, but please don't characterize my emotional state that led to this post (which itself is in violation of HN's admonition to "Assume good faith")]
Bringing in people's names in order to attack them
is a breach of this site's civility rule. You needn't stoop to that in order to discuss issues of substance in this story, so please don't.
All: please keep the internet rage reflex off HN. Same for the online shaming culture.
So all the constant anti-Tesla articles blaming Musk personally on here are fine, but these guys are somehow above being named despite their names being public record? Why is this different?
You think an article criticizing a company and its executives is the same thing as inciting an online rage mob (along with a posse for some unnamed Twitter evildoer) on a message board? Because they're pretty different.
"Bringing in people's names in order to attack them is a breach of this site's civility rule"
I think there is an issue of personal integrity here, even legally.
When I worked for a large F50, our CEO's name had to be on every email we sent out from our German staff - though I'm not sure of the specific German regulation, 'his name is on it' because it's a matter of integrity to the organization.
The founders are generally 'Officers of the Company' - and this implies both a legal/fiduciary and moral obligation in America and most jurisdictions of relevance.
I don't believe that the 'limited personal liability' concept abnegates the responsibilities of the Officers of the Company in this specific regard.
I also don't believe this is a matter of arbitrary doxing if the Officers of the Company are making decisions which are considered foul play in terms of reasonable contractual obligations.
Unless there is an issue of national security or likewise, there's simply no excuse for ending service without reasonable notice.
This is not a political, or socially nuanced discussion such as those involving harassment etc. - this is a matter of commercial civility (and legality).
In American law: "In limited circumstances, such as the sale of the small business to a new owner, the business judgment rule does not apply, and it becomes the burden of CEOs and company directors to demonstrate that their actions were in the company's best interests." [1]
(FYI - the 'business judgement rule' protects Officers from liability.)
I guess I understand we don't want to be doxing folks here ... but I do think it's reasonable that individuals should be held publicly accountable for their commercial actions - for example, someone does an ICO and then - legally - walks away with the cash ... we should rightfully be wary of investing in their next ICO or company for that matter.
The inability for executives to be held accountable for actions of the company is a core problem in present day capitalism, in my opinion.
Loathe to be technical, but this is HN ... to the best of my 'definitely not a lawyer' knowledge, you are correct that Officers responsibility to shareholders (i.e. 'internal') is greater than it is to their contractual obligations (i.e. 'external') ... but, there definitely is still a legal obligation to contracted customers, of which there are thousands. Ergo - this could very well sit within boundaries for their personal obligations.
I know they're an YC company, so I don't want to ruffle anyone here, but I do think a class action lawsuit against Twitter is warranted, even if it has a shaky legal foundation.
I believe that Twitter's actions - and the response - will be duly noted among M&A camps and the last thing we want is this to become standard practice. This is just 'bad acting' and it will come back to bite everyone here in startup-land.
I risk my erstwhile happy HN account by getting into a fuss with pvg, but he indicated that we want to 'wait to find out what's happening' - I would argue that in light of Twitter's 'total and immediate blackout' the onus is upon them to provide forthright information, and that absent that, we can assume 'bad acting' given a total blackout without any information.
Maybe we can use this as a lesson and the powers-that-be can provide some leadership, and possibly push towards a 'best commercial practice' like '120 days minimum notice' or something like that as a standard.
This is exactly the community to do it in, as I frankly don't see where else anyone has any broad legitimate authority among early stage startups such as it exists here.
Hey dang, that response and edit was pretty civil, since the names are removed could the flagged state on OP's comment be removed as well (edit: please)?
That had nothing to do with it. I had no idea that Smyte was a YC company (or had forgotten, not sure which). But thanks for letting me know.
Since it sounds like you follow these things closely, I'm surprised you'd be so cynical. If there's literally a single case where someone did that where we haven't chided them, I'd like to know about it. In fact if there are any personal attacks of any kind on HN that we haven't moderated, I'd like to know. The likeliest explanation is that we just didn't see it. In the present case a user (not connected to YC as far as I know) emailed us to complain about the comment, I looked at it, and replied—same as it ever was.
I strongly disagree with this. The founders of a company are public figures, and their actions matter. They should be named, after all they are the ones who sold their company and then immediately pulled the plug.
Oh, I've long since learned not to depend on Twitter for anything critical or business related. The thing is about Smyte is that it was a b2b SaaS with paying customers, not a consumer social media company.
People justifiably have certain expectations when they pay for a serious b2b SaaS that they don't have for a free social media service.
I'm also a bit irked with having to prune all the time. I follow someone or something that I expect to be about something and really it's just a lot of noise about other things...
The sparse info is exactly why people are looking for answers. This was a pretty unethical business move, regardless of whether these people had themselves covered in contract language.
This is not looking for any kind of meaningful answers. It's basically a call for organized harassment, facts unknown. You are surely aware how that sort of thing has worked out on other forums.
I reserve judgment as to whether it was Twitter's fault or Smyte's fault, but at the very least, it was Smyte's fault for selling to Twitter without getting any guarantees for their existing customers. At the worst, they shut down the service like this themselves.
Again, if it turns out that Twitter lied to them about how the service would be wound down, I'm sure that will come out in due time. If I were one of the founders (and I know at least one of them has been often on HN), then I'd be on this thread making some explanations right now.
Finally, the name of the founders isn't some kind of secret. This is public information. The founders have been featured in many media pieces. They sold their company to a public company.
Based on the email from the VP, my wild guess is they figured out that Twitter operating Smyte would make Twitter itself not GPDR compliant (or some other big regulation), and they didn't figure it out until after the deal was done.
The reason companies shouldn't use each other's proprietary code for basic operations: it's insecure, there's little-to-none of your interests being protected.
Wow. That sort of thing can be the effect of little to now negotiating leverage by the acquired company. They only raised 6.3M according to Crunchbase.
I'm always surprised that Twitter has managed to stay in business so long. Anaemic growth, massive spam and bot accounts, an advertising platform that's hostile to most small advertisers, a user-base that makes YouTube's comments look downright gentle, constant hostility to third party developers...and yet it continues to chug along.
even as those eulogies were being published, things started changing. Twitter began beating earnings expectations. Star ex-employees trickled back in, finding a new, more positive internal culture than the toxic one they’d left. Advertisers came back too, as did users. The company finally began addressing its trolling problem. And its stock, once unappealing to analysts like Nathanson at $14, is now trading above $46.
It’s still somewhat taboo to say it, but it’s no longer possible to deny it: Twitter is making an unexpected, somewhat miraculous comeback. It is the first major consumer social company to lose users and start growing again in a meaningful way.
Fun fact: I just did a search on a news database. Roughly 15% of stories that mention Twitter also mention Trump, and roughly half of those seem to be about Trump's tweets.
I don't know if Trump is really causing Twitter usage to increase, but it has certainly made people who don't use Twitter more aware of it.
If you compare the awareness/user ratio on the various social media platforms, I bet Twitter's is pretty high, which means they have potential for significant growth if they can improve the UX and the "how do I use Twitter? Why would anyone use it?" problem.
It's enormously popular! The world's biggest social network focused on public interaction. They've got everyone from a hundred million normies following celebs to countless niche interest circles.
The real surprise is that they've failed to effectively monetize their huge, huge audience.
I agree, and one specific thing that is very annoying is that tech podcasts seem to rely on Twitter as their sole means of communication with the show. Some shows and networks don’t have email addresses, just twitter handles, and others actively tell you not to email their address as you probably will just get deleted. I’m not on twitter and have no desire to be but I would like to contact the show once in a blue moon.
> I'm always surprised that Twitter has managed to stay in business so long
I'm not. Millions of people are visiting Twitter daily to check out if Trump has declared trade or real war against yet another country. Or whatever the Orange in Chief has brain-farted over night.
In addition Twitter is a fine tool for starting shitstorms and one of the few safe havens left for adult material creators (= freelance sex workers) to post their content (but that may also change one day). Oh, and media often turn to Twitter for soundbites/other stuff to quote, which also creates demand.
>I'm not. Millions of people are visiting Twitter daily to check out if Trump has declared trade or real war against yet another country.
This is true and not just for Trump, most politicians are finding out they can bypass the media and go straight to enough people to control the message, especially if the message is bombastic or controversial.
Why rely on the faucet when you can drink from the fountain?
Off-topic, but twitter related: has the site fallen off search engines?
Yesterday I wanted to look at some joke account (by correctly spelled name), Google yields nothing from the domain, Twitter wants me to sign up to search there ...
(Not a habitual user, or even reader, as you may guess.)
It’s fairly common in so-called acquisitions of startups that the the acquirer never buys the company stock. This leaves annoying liabilities like existing contracts in the selling company which will be shut down. The buyer might purchase IP so that the company to be shut down can pay out their investors and old customers.
However that situation would usually never be described as an acquisition by the buyer. (The seller might put out an announcement that says “We’re proud to be joining FooGleZon, it’s been an amazing journey!”)
Going off on a tangent here, but it's interesting how the first thing on that list is "create product".
I would argue that most new companies (at least started by and solely run by technical founders) fail exactly because of that.
Product-oriented people tend to think that the success of a business is due to its product(s), when the reality is businesses succeed due to their ability to sell their products. Hence, the most important thing is actually being able to properly find and target an audience, which should be the first step of starting a business.
The second step of starting a business should be to test marketing channels to make sure the selected audience can be effectively/efficiently reached. Only after this has been properly done should a product be ideated, based on the knowledge acquired about the target audience and the marketing channel(s) that work.
Just my 2 cents to the HN community after having learned this lesson the hard way, failing multiple times and starting several businesses.
What's more, you could be product focused and still produce bad products. I just left a startup that seemed to have really bad decision making in areas like "what feature should we build first/second/third", and major areas needing design and careful thought were instead treated as an afterthought. But the tech was solid and lots of work was getting done, so... good enough?
Heads up, Y Combinator! Heads up, executives! Enough of this foolishness.
I'm a tech worker who doesn't live and work in the Sili Valley reality distortion field (SVRDF). From outside the SVRDF it's hard for most people to tell one company from another. They all look like SV. That means they all look a little untrustworthy and a little dangerous.
Now, sure, this Smyte had an insider's brand, sort of like "key gaffer" in the movie industry. But their shenanigans, and Twitter's, just made it harder for all the rest of us to sell to prospects outside the SVRDF.
Y Combinator, will you consider doing some time on basic business ethics with your incubator residents? (I'm talking about stuff like "don't screw your customers, suppliers, or shareholders: keep your promises and honor your contracts.")
People are so shocked by this precisely because it's an extremely abnormal way to do business, which demolishes your chip-on-the-shoulder premise. You're using a rare occurrence (thus the shock), to extrapolate that it applies exactly the same to the majority of Silicon Valley.
I'm fully prepared for whatever negativity comes with this comment because it's being asked from a place of genuine curiosity-but yes there some snark baked in:
Does the rapacious acquisition game of Silicon Valley and seemingly increased rate of consolidation of startups to established giants feel weirdly like he days of Ma Bell to anyone else?
It seems like more and more There's tales of these acqui-hires with promises that the aquirer will present some kind of "polished" alternative but under the larger brand and I've found quite frequently the promised offer-if ever shipped at all-is far less superior and far more self serving to the brand than what was previously offer by the acquired when they had more on the line by operating as a startup.
I'll one-up you with cynicysm: these days, getting acquihired - AKA having "an exit" - is the design goal of startups from day one. The whole point is baiting enough users to use your product that you can look good to some large company, get acquired, close up shop and enjoy your millions. For founders, it's "compress your work years and get rich sooner by working more now". For users, it's pure bait-and-switch though.
This is such a constant thing that right now, I'm very reluctant to use startup-made SaaS for anything that isn't throwaway. The expected lifetime of a service is just extremely low, and the cost of integration (lock-in due to platform idiosyncrasies, and your data being taken hostage) is too high.
Almost. Back in the Ma Bell days, connecting equipment to the Bell telephone lines without authorization was an actual crime. You could be be prosecuted for it.
You couldn't develop telephony product at all unless you were Bell.
Not much by way of a published study if that's what you're after, just a supposition if I'm being completely honestly.
Welcome to being wrong on this if some good soul does have the data on this, though. It's why I asked, maybe someone has research that would prove me wrong and I'll gladly take being corrected on the matter.
Yeah, I feel that way too. Like, every company whose products / services I liked that got acquired never released another interesting product or service. Dropcam is one that I can think of, nest is another which is related. Microsoft buying Wunderlist also seems like it’s only leading to Wunderlist getting shut down. CoreOS sold to RedHat and while I have my doubts, this sounds good: https://groups.google.com/forum/m/#!topic/coreos-user/GR4YlF... (but don’t these kinds of messages always paint a rosy picture of the future?). We’ll see what happens to CoreOS.
EDIT: just remembered WhatsApp. This happened recently: https://www.cnet.com/news/whatsapp-founders-may-have-left-1-... and it doesn’t seem like them selling did anything positive for their product or customers, only their bank accounts. Seems like they couldn’t even wait for FB to open up the golden handcuffs.
Anyone have examples where the acquired company goes on to be really amazing, on a trajectory that they wouldn’t have had access to without the big injection of funding from the acquiring company? I can’t name any, but that doesn’t mean they’re not out there.
Site's that block users for not allowing cookies and tracking are essentially paywalled. Communities like HN should do their part to penalise sites that pull these kinds of tricks. If not us, then who?
In the next installment of "How To Make A Successful Exit While Not Give A Flying Füçk About Your Existing Customer Base Or Industry Rep. That Got You There 101"
Is anyone else as tired of this schism of material gain over all other considerations? If the tech industry wants to end up with the same ill repute as Wall Street this is certainly the right way to go about it.
This is it? I've often considered the abuse of 1099 workers to be the thing that one day catches up with tech as a precursor to its own "Enron" moment. Maybe it'll be both.
Abuse of 1099 independent contractors is pretty much endemic to every large industry and profession in the US, though. The tech sector isn't going to get called out on it before sales, etc. It's just "the way it is", because the IC regs are almost entirely in the employer's favor financially. And since the capitalists have the money, the rest of us are unable to influence the government with the same net effect.
You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made.
Back when it was more traditional to buy a software licences and self-host rather than pay a rolling subscription to an XaaS, a service going dark just meant that you were shut out of updates but not the software itself. Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
> Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
I think people often underestimate the value of being able to manage their own destiny. They key point here is "manage" though: you have to actually do it - it's an active process.
If, on the other hand, you simply stick your head in the sand and ignore the situation it will eventually get to the point where it becomes extremely difficult and expensive to deal with due to a variety of factors such as no clear migration path, loss of institutional knowledge, and so on. I name no names, but worked for one UK company in particular with form here.
I'm not making an argument against SaaS (we use a few of them), but it's important to understand the trade-offs you're making in terms of ceding control.
Sure, but a service like Smyte requires significant expertise to make work well and run. Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.
Further, some of the biggest value they will provide is being able to see transactions between vendors. So eg calculating an IP address reputation score by seeing bursts of fraudulent purchases. That is impossible to do in house only.
> Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.
But that's not what either I or the GP were talking about: we were talking about hosting a product that you'd purchased from another vendor on your own infrastructure.
Choosing SaaS products that have self hosted licenses or open source alternatives is part of managing your own destiny and making the right trade off. If the former exists, this discussion is moot because there will almost certainly be a commercially supported migration path (or someone didn't do their research and chose the wrong vendor). If it's the latter you're back to having to invest capital into creating an in-house system while learning how to use, scale, and troubleshoot open source software. If neither... god help you.
I'm not going to argue that these sort of problems aren't ideally suited for SaaS however it is still worth remembering that systems like this have existed before SaaS took off. eg you would host the API yourself but cron a db update every n hours or days. So if the service went offline you still had a local database and API.
The obvious downside to that is you don't have real time updates (as well as the usual other benefits that SaaS brings), which is why SaaS suites this kind of business well. But it can be done and was sometimes done this way "back in the old days".
Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it, though. So situations like this are suited to having a centralized owner of the data who can perform rate limiting, restrict to its paying customers, etc.
Distributing the database to customers is still workable despite this drawback - antivirus systems are a common example.
> Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it
I don't agree with that. Having the database only allows bad actors to see if they've been added to the database - not the business logic of how they ended up in that database.
You could argue that knowing you've been added to the database is still an advantage, which is true. But those same bad actors could still test the system with SaaS solutions too. eg when one acquires stolen bank cards it is common to use those details to make small payments for things like food delivery to test if those card have been blocked or not.
Having a fast, cheap and detailed feedback loop is useful for testing for the bad guys, like in any programming process. For example, in the credit card case, you can check thousands of cards without needing to find a merchant who will take your thousands of small, frequently-declined transactions. Or you can check the details for cards you haven't fully compromised (maybe the guy selling you stolen bank cards only gives you a portion of the details until you've paid him).
Oh absolutely. Like many things security related, it's not always about stopping them but rather slowing them down enough so that it doesn't become cost effective.
I wouldn't say it's preferable for the customer. It's just one of many options.
> but it's obvious why a vendor wouldn't be keen
Yes, SaaS obviously has benefits for vendors too but what's most preferable to them is making money. If every customer refused to do business with SaaS then vendors wouldn't prefer providing SaaS solutions (yes, I know I said "solutions solutions" :P). The reason SaaS is so prolific is because it offers advantages to customers as well.
Yeah you're right. And weirdly I did know that so I haven't a clue why I wrote that comment. I can only assume it was a multitasking fail. Thanks for the correction though :)
It's not really a good argument. It's an ok (if stultifyingly predictable) argument when it's made about free services. When you enter long-term business relationships, pay for services, sign contracts, it's quite reasonable to expect to receive what you paid for. Not a lot of useful stuff would get done or made at all if we all turned into preppers.
You claim it's not a good argument yet stuff like this does actually happen. Sometimes it's genuinely because a business closes up shop, sometimes it is because it is bought out. Sometimes the business is still running but they just decide to deprecate a service or API without grandfathering existing customers / offering sufficient time to update customers code or even switch to a new provider.
Granted there are more cases of when SaaS et al works than when it does not. And I do agree with you that it's no ok to shaft paid customers in this way. But that doesn't mean that it doesn't still happen. This is why I've have to consider the self-hosted verses XaaS debate myself writing Business Continuity / Disaster Recovery / et al plans.
Of course it happens. Bad things and unexpected things happen. It is better to be prepared for them than not, etc. But saying completely predictable and trite things whenever some unusual bad thing happens is not 'a good argument'. It's barely even an argument.
The argument of self-hosting vs SaaS is still a valid one regardless of how predictable you might consider it. However the argument of arguing about arguments is definitely trite. But perhaps this is a good place to end things before it gets too meta ;)
"You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made."
I really dislike this idea that it's our responsibility to ignore inflammatory tones and trolling, instead of it being the responsibility of the poster to communicate clearer.
You're over analysing what I posted. I wasn't suggesting it's people's "responsibility" to look past trolling. I just felt there was a valid point buried in there that was worth reiterating in a non-inflammatory tone.
I assume that in the real world you grow your own food, generate your own electricity, dig up your own wells for water, and knit your own clothes? Companies have to rely on other companies for critical services, just like we do in the real world.
Multiple sourcing is the key. All of the above products can be bought from multiple places, this API could be "bought" from a single vendor.
Any usage kf SaaS would require a solid contract in place and a plan B (and C, and D, etc) to reach even a fraction the reliability of e.g electricity or food.
Managing risk comes into play here. Would you spend time integrating 2 services, negotiating 2 contacts, supporting 2 providers who may change features and approaches over time, splitting a fraction of traffic to make sure both still work, limiting yourself to common feature set... Or would that cost you more to implement them a few days of (potential) complete downtime.
Choosing the potential downtime is not a failure, if you considered your options.
You have to admit that the Smyte's customer list that's floated around, here, isn't all just some startups that can't afford to research their backups. And maybe they did, actually. Did anyone even hear if Zendesk was even affected by this happening?
Most of this has a latency that's orders of magnitude different. For electricity, those are microseconds (and for anything longer you do have UPS and generators and failover, right?), for the other things, it's days. "Half an hour" is just about the worst timeframe for a deadline.
Own a generator, have a stock pile of food and water, have a well I can run with the generator. Have plenty of clothing that can last many years. It isn't about being able to be self sufficient for forever, but being able to be self sufficient for long enough to weather a disaster (last time hurricane took down power for a few weeks, I only had minor inconveniences and was able to focus on helping others who weren't nearly as well prepared). Yes, it costs money and not all people can afford it, but for businesses, it should be something they do plan for.
I don't want to make assumptions about your metabolism, but I find that I actually need to eat food on a fairly regular basis, to the point that I would not characterize it as a "once off" purchase.
If the company I buy food from decides to ghost all their customers with 30 minutes notice, I can walk a block in the opposite direction and buy food from a completely different company. And I can walk a block further and buy food from yet another company. Any of these companies vanishing into thin air would only cost me a 10 minutes of my time at most.
When I need more food I'll make another once off purchase. If the place that had lunch today has boarded up it's windows tomorrow I'll just eat somewhere else. If I bought that food from a supermarket it doesn't disappear from my pantry when the supermarket closes down.
My food supply is a self organising network that can route around any single point of failure. This is about as far removed from SaaS as you can get.
My coworker wrote a service that scales extremely well (he pretty much maintains, monitors and operates it by himself, across dozens of servers).
He has written multiple deployment options. Kubernetes, Docker swarm, bare VMs, etc. He has tested all of them and written installation tools that can deploy to any of them.
For exactly the reason you imply, he doesn't trust any of these options to always be available. I think this mentality has a lot to do with why his service scales so well.
And while he was doing that, how many features could he have added that would increase revenue for the company?
There's more to building a long-term, sustainable business than optimizing revenue at any cost. And part of that "more" is exactly the work you to do insure against future risks.
What’s more likely to happen first - Amazon will stop offering an AWS service that you depend on at a price you are willing to pay or that the system you are building will become obsolete and when the time comes to replace it, you can choose different infrastructure? I would even say that your company going out of business is a bigger risk than depending on AWS.
The old saying that no one ever got fired for choosing IBM could be applied equally to AWS. The companies that chose IBM in the 70s and 80s can still buy new compatible systems. The companies that chose Stratus or VAX, not so much.
What’s more likely to happen first - Amazon will stop offering an AWS service that you depend on at a price you are willing to pay or that the system you are building will become obsolete and when the time comes to replace it, you can choose different infrastructure?
I think you're committing something of a "fallacy of the excluded middle" here. Those aren't the only two things that can happen. AWS could, for example, continue offering the service but not to you. We've already heard stories of AWS accounts being closed for various reasons, and it's hardly a stretch to imagine a CloudFront / Daily Stormer kind of scenario where Amazon decides to punt you for taking an unpopular political position, etc.
I would even say that your company going out of business is a bigger risk than depending on AWS.
I would say that I disagree, in at least some cases.
Let’s take today’s equivalent - what’s the likelihood of Amazon not offering AWS or even Microsoft walking away from Azure and pulling the rug out from under developers?
Even businesses that chose Microsoft technology in the 90s when they were the clear leader have had a relatively painless upgrade and support path. They abandoned VB6 15 years ago, but you can still wrap your VB6 code as a COM object and interact with C#.
I’m saying that it is the safer bet to go with the largest provider with the largest customer base. In the 80s that would be IBM, in the 90s, MS over SUN (and the royal claustrof%%% that Java has become under Oracle) or Borland.
What is the "largest customer base" relative to? In the early 2000s, most Fortune 100s used Solaris [1]- presumably this trend started in the 90s, so I don't know why you think that a manager during that time period would choose Microsoft over Sun except for specific use cases (e.g., smaller hosting providers).
Let's look at yesterday's equivalent to AWS- Sun Grid. Sun was viewed back during the first dotcom era in much the same way that AWS is today. All that it would take to completely bork it would be an economic catastrophe that mirrors the first dotcom bubble. Obviously, Sun Grid is no more.
Solaris didn’t disappear overnight like Smyte did. When Sun was acquired by Oracle they just didn’t abandon thier customers - that would have been too value destroying.
You can still buy Solaris based systems today but you would have plenty of time to migrate. But seeing the writing on the wall you would have plenty of time to migrate.
I think there’s a balance you can strike between things. But it’s not always a single variable. Learning those things was probably valuable to him in other ways though. If he eventually grows his service and has to hire someone to manage ops he’ll be much more capable of evaluating them.
Because his service scales so well, it is bringing in a lot of revenue for the company (sorry, don't know the exact amount), at a very low cost. He is a very productive developer.
Whatever his salary is as an engineer, he is definitely underpaid.
Since I have mostly avoided doing side projects outside of work to learn something new [1], I understand spending extra time at work to scratch an itch and doing a low priority project as long as it doesn’t interfere with revenue generating - or cost reducing - initiatives.
[1] I would much rather work on a work related project using a new to me technology and if it requires longer hours to complete it, I’m find with putting in the extra time. It seems like more of a win, I get to learn a new technology, get to see it actually be used by others, and it will be helpful during either review time or worse case a resume builder.
Well, I guess every company on AWS is screwed, huh?
Screwed? Maybe, maybe not. At risk? Abso-fucking-lutely. There are plenty of stories out there of companies who had their AWS accounts shut-down for some random reason, plenty of stories of AWS outages, etc. Even if Amazon does't pull a "Smyte", there's plenty of risk associated with being on (or "solely on") AWS.
By and large, anytime you are completely dependent on someone else's platform, you are increasing risk by putting your destiny in someone else's hands.
Eventually yes. Fortunately there is enough demand for actual machines, but if that ever goes away and all of our compute power is owned by a handful of companies then we'll be well on our way to a scary dystopian future.
At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
Not even eventually. Right now. How many companies out there absolutely depend on all the value add services that AWS provides that are hard to replicate?
Does your business depend on Amazon Transcribe (or any other AWS-only service)? Do you have engineers competent to turn around an equivalent product for you in a short release window? No? (Don't lie to yourself.
It's no.) Congratulations, you are now Amazon's bitch.
I think people will start feeling the pain when Amazon isn’t printing money anymore and starts looking at winding down less popular/profitable services.
> At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
And if that happens people (I'm looking at YOU, software and devops engineers, with your damned sketchy business cases, desperate to move projects onto AWS mostly so you can bolster your CVs) will only have themselves to blame.
Fortunately I suspect there are enough of us jaded and sceptical types around to avoid it.
And then you get yet another level of abstraction. It's about like developers who want to wrap all of their data access in a repository pattern just in case they want to change databases.
Two things about that -- hardly anyone ever changes infrastructure just to save a little money and their code has to be so generic that they can't take advantage of all of the features of the platform.
The chances are far greater that your company will go out of business way before Amazon stops offering AWS. Theory is great that Terraform allows you to seamlessly move infrastructure between cloud providers, but the amount of risk and regression testing it takes to do so would make it a non starter for most companies.
The idea I'm floating isn't not to use AWS, it's to not be locked into it. Don't use the things that only they offer.
This isn't just about cloud provider lock-in, but something you need to do to have a good DR strategy.
Back on that Christmas Day that Linode got DDOSed, my company was 100% on their infrastructure. I only had to work a couple of hours that day because I had worked and planned in advance to launch our infrastructure on another host. I saved my company from a multi-million dollar loss that day.
So if you're using AWS and instead of using their services, you're just hosting your own equivalents on EC2 instances, what's the point? The entire reason behind using AWS is the "undifferentiated heavy lifting" - allowing them to take most of the burden off of you except where your company can add value.
If all you're doing is hosting a bunch of VPS's, you're really not saving yourself too much over just using a bunch of colocated servers. You can also get much cheaper, less reliable, less geographic diverse hosting somewhere else.
But I would have my head handed to me if I even suggested to a company to base their entire infrastructure on Linode instead of AWS, Azure, or even GCP.
There isn't one way or reason to use cloud infrastructure. Maybe for smaller companies the "undifferentiated heavy lifting" narrative makes sense and I've used it as such in the past.
Managing tens of millions of dollars in physical and cloud infrastructure at multiple providers/dcs for a publicly traded company like I am now? No, that's not the reason to use cloud infrastructure at all. Cloud in this case is more about capacity planning and avoiding managing hardware inventory than anything.
That's not entirely true now but has been in the past. They've actually tied their fates to both AWS and GCE in different parts of their infrastructure. The obvious example of this is the one or two global outages they've had due to misconfiguration in GCE this year.
I know their video encoding platform is heavily tied to AWS because the teams that built it optimized for cost (which for them makes a certain amount of sense) and they massively parallelize the encoding work. I talked to some of their engineers at re:Invent last year and it was kind of interesting. They aren't using a lot of AWS-specific stuff though, just AWS EC2 reserved instances.
I don't think that just because they're Netflix that everything they're doing is correct. This can't possibly be true.
So you are saying if I take my AWS infrastructure, which I launch with my terraform config, I can simply use that same config to launch the same infra on GCP or DO? Didn't think so. It's only slightly more agnostic than AWS CloudFormation, in that I can at least use some of the same conventions when I completely rewrite my terraform files.
I think you are confusing "agnostic" with "compatible with lots of different providers", but will require a total rewrite if you want to switch providers.
I'm as big of a Hashicorp fanboy as you will ever find for on prem implementations -- I've used Consul, Nomad, and Vault. But that's simply not true. Here is an example of a Terraform template straight from their getting started page.
There's nothing preventing you from using other providers as well. You just have to understand how to achieve the same results when you build your modules.
So you have to rewrite your modules for each platform so you're still tying yourself to one vendor and that still doesn't alleviate risks from migration or the large amount of regression testing -- just in case Amazon goes out of business or you can might be able to find an equivalent slightly cheaper?
If you are a small company, the difference isn't worth the risk and the expense. If you are a large company you can probable negotiate with Amazon.
That doesn't even take into account all of the dependencies that your developers have taken on AWS services....
Employees wouldn't have to keep one eye on their CV and plan for the next job if employers treated employees better and gave them long term security & oppertunities.
I think you might be underestimating AWS migrations initiated at the executive level.
I dare say the AWS sales pitch is even more effective at that level: "Increase your agility, reduce your operational overhead, scale out with the push of a button. Be like Netflix and Apple."
Most executives aren't well-equipped to probe the sharp corners that aren't in the pitch.
I think we're already there, and it isn't much of a dystopia. There are only 3 big cloud providers, and many major companies rely on them for everything.
But their business is built on providing this service. At what point would Amazon, Google or Microsoft say "we don't want that $xB in revenue, just turn it off. NOW!
At that point government would step in and nationalise that part of the company :-)
More seriously I can see cloud going the way of telecoms in that there would be a provider of last resort if a telco goes bust another operator takes over to provide service.
Evaluating the business viability and solvency of any important suppliers is something that you should be doing yourself even if you have a years-long contract.
Also, such risks can be insured.
Also, such risks can be mitigated in various ways - I recall having a contract where the source code of any product version delivered to us would be held in escrow at a third party, and in case the supplier went out of business or a bunch of other conditions, we'd get their code and legal rights to use/modify/maintain it ourselves or through some contractor.
A contract is between you and some entity - if you want to protect yourself against that entity "dying", then that can be done by a contract with someone else (e.g. insurer, escrow, etc) who can make you whole even if that entity is dead.
The existance of a contract does not prevent actions like this. A contract will get you compensated later, but if the provider cuts you out, nothing can prevent it. The outage will be there wether it is legal or not.
It really depends on the circumstances but often a contract makes no difference. If the service provider disappears (bankruptcy, acquisition, natural disaster) and there is no one to sue then the contract is irrelevant. If it's core functionality and you go out of business then being able to sue but having no money to makes the contract irrelevant. If the provider can absorb the cost of a lawsuit because you're their only customer and it's cheaper to drop you then a contract could be irrelevant. If the provider want's to launch a competing service and breaching the contract is just a cost of doing business then the contract is irrelevant.
Contracts can be valuable but often they are over valued.
Contracts don’t disappear on acquisition, of course. The new owner takes on the responsibilities of the acquired company, as well as its assets.
Companies also are not physical items that can disappear due to natural disasters.
The only ways they can disappear is by closing down, and that does mean honoring contracts, and by going bankrupt and that means using all leftover assets to satisfy all the parties whose contracts you’re breaking. And no, you can’t just transfer all assets away and leave a shell to go bankrupt because that is fraud and the judge can nullify the transfers.
I am not 100% familiar with exactly what Smyte does but if it uses heuristics to determine if a comment is spam/harassment, then thats not really core infrastructure.
Your company isnt going to die without it (its not like Hootsuite losing access to the Twitter API or Heroku cut off from aWS) But it might be more effective with it.
I expect no less from a company that essentially elected Trump and continues to function as his mouthpiece to this day. Screwing over Smyte's customers must seem like nothing after they screwed over the entire country. What do employees there say to themselves every morning to justify staying? Trump continually breaks Twitter's rules, yet he hasn't been banned. He threatens world leaders, bullies, insults and lies non-stop, and Twitter is the bullhorn he uses to do it. Google's engineers stopped the company from bidding on war-related contracts, when are Twitter's engineers going to develop a backbone and get Trump banned?
I stopped using Twitter the day that jackass was inaugurated. Everyone who works at Twitter should be ashamed of themselves.
I do something similar (real-time detection of spam-comments) at https://blogspam.net/ and I'd be reluctant to even attempt to handle fraud.
If my site messes up then dodgy comments might get posted (unless they're blocked by other systems). If you misclassify an "is-fraud" check you either have a customer lose money, or a potential customer be denied service. Neither of those are great.
I suspect actually doing anti-fraud testing could be done simply with a few heuristics, knowing the kind of attacks I see on other sites I run, but the only way to be sure is to get a lot of data-volume and real-life results to compare against.
You could do that, and they'd quickly tell you that fraud detection requires vast datasets that your company almost certainly cannot generate or maintain in-house. Then what?
Find a way to live without it I guess. Certainly that would be better than having your entire production stop working because somebody else decided to take a multimillion-dollar check and run with no concern for you or contracts you had signed
If you can afford to cut multi-million dollar checks for a fraud/abuse prevention service, then you can also afford to retain a legal team who will recoup your losses and then some when those contracts are broken.
A temporary production outage due to a rare situation is far preferable to being at the perpetual mercy of hackers constantly pwning whatever half-baked homegrown security system you forced your engineers to implement with inadequate support & resources. Real businesses don't operate on "I guess" solutions and they don't just fork over huge chunks of cash willy-nilly without doing at least some cost-benefit analysis and making sure they're protected in case of contract breach.
My employers do just that - Fraud prevention is a fundamental part of our business operations.
It largely involves multiple teams of full time employees working on different aspects of fraud prevention (some of them are even Competent Developers!) a few dedicated offices, some teams embedded in regional development offices, a bunch of bespoke software and it's ongoing development and maintenance.
Thats in addition to the core dev teams being aware of and mitigating any possible vectors within the platform itself.
At any kind of scale it's rarely as easy as 'hiring a few competent developers'.
" which you cannot host a fully working copy yourself"
Sadly, no.
The majority of services have complex functions often requiring key data and updates, and it's just not possible, let alone feasible to 'self host' most things.
But it's a neat idea ... maybe there's a solution there ...
> You're going to host your own version of Google maps?
I've never worked on a service which involves the other things you mentioned, but I've worked on some map-based services, and we went with self-hosted OpenStreetMap instances for exactly this reason :)
I'm almost entirely sure that Google is not in the habit of turning Services down with no warning. I guess I should have been more specific: never rely on a start-up to be here tomorrow
I want names out there. Who is responsible for this!!!! Who was the CEO of smyte? Who is the manager at Twitter that gave the comand to shut down everything?
"Smyte" would be a good name for something that kicks users off a social network in a dystopian novel. "Smite" appears dozens of times in the Bible and means to strike someone, sometimes fatally.
It surprises me that this article has so many votes when nowhere has it actually provided any details of either the commercial terms of the acquisition OR whether any Smyte customers actually lost any money from prepaid service.
Sure, it sucks that people lost an API they were using, that much we know, but hundreds of votes for a “bad” commercial transaction on which we have no details?
The way I see it, a number of Smyte users just received a free billing period of API access?
> According to reports from those affected, Smyte disabled access to its API with very little warning to clients, and without giving them time to prepare. Customers got a phone call, and then – boom – the service was gone. Clients had multi-year contracts in some cases.
As a group response to the sibling comments I'll point out that yes I read the article, but my question related to something which the article didn't cover: did anyone lose prepaid services from their contracts? I can't see this being discussed in any of the coverage of the Smyte acquisition.
Does anyone inside Twitter know who the executive was who pulled the trigger on this decision? There's no way this person knew the gravity of accepting the risk for blowing this many high-profile customers out of the water all at once.
I'm pretty accustomed to executives making dreadful decisions without the approval/acceptance of other stakeholders within said person's firm. I'd rather know who made this error and not interact with that specific person's department rather than stop doing business (e.g. large ad-buys) with Twitter as a whole.
I don't want this to be a witch-hunt so much as I want the person to just come forward and own the decision, because unless they have an exceptionally good reason for it, it comes off as absurdly high-risk to both Twitter-the-business as well as to all the clients who've likely written serious penalties into their contracts for events like this, which again brings that business risk back full-circle to... Twitter.
I work at twitter and wasn’t able to find out immediately, but have been collecting responses like these + the article and communicating with someone who knows what to do. I only found out about this over twitter and I am personally deeply frustrated and working to understand whatever led to this.
Sorry to hear that, can imagine the frustration.
As someone down thread said - this smells like the long arm of GDPR unintending some consequences.
"I work at twitter and wasn’t able to find out immediately"
Sweet, please tell Jack I'm done with Twitter violating my copyright and it's game-on now.
I don't really think there's any polite way to put this: Your employers are simply assholes.
This is the sort of informationless pile-on comment that damages HN regardless of what you're talking about. Please don't post those here.
https://news.ycombinator.com/newsguidelines.html
To be honest, this reads less like an attempt to integrate anti-harassment measures and more like an attempt to completely destroy a business trying to offer them. It further solidifies Twitter management as complicit in aiding and abetting the harassers by using its capital to eliminate a threat from the market. After the speed with which Twitter went after the ICE employees list, the evidence is clear that Twitter only has a problem tackling harassment when it's against marginalized populations.
That's a pretty extreme leap when virtually no information is available yet. The HN guidelines ask you to "assume good faith" for a reason: people are all too ready to take such charges as givens and then pile more on top of them. It's a behavior that harms the container here, and it's not hard to wait until actual evidence appears.
That's fair, I will suspend judgement until further evidence appears, but the well of beneficial doubt is running low. I think a long past history of suspicious behavior is a good reason to update our assumptions.
I don't disagree. But the real reason for having an "assume good faith" rule is what it does to this community when people don't. Therefore it needs practicing even when it feels undeserved.
Here's the whole guideline you're talking about-
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
It's talking about how we all talk to each other. I think it's main purpose is not to lessen criticisms for corporations so much as to keep the discussion constructive.
You're probably not going to win an argument with a moderator by explaining to them the subtle logical error in the interpretation and purpose of their own guidelines.
Yet now we're failing to assume good faith in our moderators, by assuming that they will fail to recognize a conflict of interest arising from questioning the extent and source of their authority, which they have reason to see maximized.
The parent comment actually seemed spot on to me. The rule is to assume good faith of your interlocutors. The parent failed to assume good faith of a billion dollar corporation. I think it's reasonable to push back against being asked to tone down the latter, while honoring the rule for the former.
I think dang explained it pretty clearly in the comment. It is bad for the forum when, given some very limited set of facts, you offer the most heated, ragefest pile-on inducing explanation or theory. It doesn't require some talmudic reading of the guidelines which are, after all, guidelines and not every single thing is explicitly spelled out. 'Avoid taking discussions into flamey or ragey directions' is advice the mods dole out to some person or another nearly daily.
the well of beneficial doubt is running low
That is a metaphor made with an industrial grade cement mixer.
Hi. Former Twitter anti-abuse employee here. If I had to sum up Twitter's executive leadership, it would be with Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity."
Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way. They mainly care about it as a PR problem, so they'll do the minimum necessary to get out of trouble. But that's not because they hate anybody; it's because as a company they're not very good at getting anything at all done. They succeeded because they created a platform with a network effect; in that situation, competence is secondary.
My guess is that Smyte was failing as a business and looked around for an acquihire. I'd further guess that they had some connection at Twitter, and that getting a few engineers with relevant experience at a low price seemed like a good deal. So I'd suspect that nobody at Twitter cared either way what happened to Smyte customers.
>Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.
I don't like the implication here (i.e. white men don't care about harassment/abuse and they are also stupid.) I would wager that most anyone of any color or gender or religion in any high position of authority at any company cares first and foremost about the financial viability of the company they have been charged with running.
Many (most) "white men" do in fact care about this stuff and aren't bigots/sexists/racists. You're naive and part of the problem if you think this is a "white male" issue and not a much more complex problem. In fact, you're exhibiting the bias you attribute to these managers.
I think it's fair to say that people who are not white males (such as myself) are more likely to be targeted by a wide variety of harassment, and that without their views it may be much easier to miss the issue. As a white male I don't get targeted by misogyny or racism (at least not anywhere near the scale that others have). Having people who have experienced that around to help guide the decisions for how to deal with it- and how many resources to devote to it, since most business decisions come down to money.
Also, we can all have biases without it meaning people are being racist. My parents like rock music, so I'm biased towards listening to that over country. I went to certain schools and that colored my world view. I once heard bias described as a heuristic the brain uses to make quick decisions- something that was extremely useful for primitive humans but which needs to be actively balanced in our complex society. A huge part of that for decision makers is exposing themselves to other viewpoints as a way to collect information they themselves wouldn't have gotten. There's a reason why companies have used things like focus groups and surveys to understand their customers- having more people who understand those customers in leadership at companies is an even better way to do it.
Yea me neither. Generalizations like this aren't helpful, or least bit fair.
The abuse is disproportionately targeted at people who aren't white men, and is is disproportionately performed by white men.
This is nothing particularly new. It's the history of America, which was founded by and for white men, and whose story can be told as white men slowly and reluctantly giving up their power.
If "many (most) 'white men' do in fact care", history doesn't really demonstrate that. Not, at least, in terms of doing anything about it. This is not particularly shocking; people care more about problems they experience, and less about problems that happen to benefit them.
We need only look at how the majority of white men supported a blatant racist and sexist for president. And who continue to support him despite (or because) that has only become more clear.
Their leadership is mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.
To the folks inclined to keep piling on to this comment, I think a more charitable reading is that "The leadership consists of very privileged people who likely haven't dealt firsthand with the kind of harassment to which some demographics are routinely subjected. They appear to just not really get it at times."
I think that's a very realistic assessment of how some things remain intractable. People in power have blind spots because it isn't something they have to personally deal with. So they end up doing things that are counterproductive or seriously problematic without really meaning to.
That doesn't make it okay. That's just reality.
>The leadership consists of very privileged people
Same problem. Who's to say they are "privileged" as opposed to being hard working / talented people? You're pushing the same narrative; white men in positions of power are privileged and don't really deserve what they have. Nonsense. Sometimes that's true, sometimes it's not. Either way it's a useless, tribal way of thinking that does more harm than good.
It's completely fair to comment on this because it is the basis of the GP's argument.
I don't read the parent comment as claiming that the people in power are not hard-working or talented, nor that they don't care. Rather, I read it as asserting that many white men have had the privilege of not having to deal with discrimination and harassment, and that it doesn't come to mind as easily as it does for underrepresented individuals.
Just like a developer is going to spot defects in applications more than a non-technical user because they are exposed to defects more often and understand where things will fail, people who have lived discrimination will be more likely to see it where it happens, and understand where to look to find it.
I think it's asserting specifically that the management of Twitter has had the privilege of not having to deal with discrimination and harassment.
I imagine the leadership of a company like Twitter make really good money, have substantial power and don't have the problems of your average working stiff, regardless of skin color, gender etc.
It's sort of tautalogical to say "People in power don't live like the rest of the world." And this fact has been a problem in terms of creating policies that work well for the masses since time immemorial.
It's one thing to say that people in positions of power, on average, have not directly experienced the harassment or abuse that minorities or people belonging to marginalized groups have. That's fair and may account for some level of ignorance on their part. It is another to say that they _do not care about these things_ specifically because they are a member of a certain class/race/gender/whatever.
It's an important distinction as it sets the parameters by which we engage in conversation and debate. If you begin the conversation by accusing the other side of being inherently bigoted then there is no civil or useful exchange of ideas to be had. It puts the other side in a defensive posture from the get go and only leads to more tribal group-think.
And you refusing to accept my reframing as valid intentionally digs this hole deeper. This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something.
Both sides need to give a little here if we aren't going to remain stuck on such topics. And expecting all that giving to come from women, people of color, and other underprivileged groups is merely entrenching white male privilege in a really ugly, toxic manner.
>And you refusing to accept my reframing as valid intentionally digs this hole deeper.
Because I don't believe it is valid. Class/color/gender are not the only influencers on someone's decision making process, yet you two seem to think that's all we need to consider.
Where is the evidence that these people don't care about abuse and harassment because of their class and color? I see a pretty damning statement which includes zero evidence, just what someone believes to be true for <reasons>.
So no, I won't accept your premise or the GP's until I see evidence. If there is no evidence then fine, you and anyone else are free to believe whatever you like, but don't pretend it's a logically defensible position.
>This is a perpetual problem where one group feels unheard, unserved, left out, etc and then they complain about that and then are told their complaints aren't valid because they said them in a not nice enough fashion or something. Both sides need to give a little here if we aren't going to remain stuck on such topics.
Not every complaint _is_ valid, and we have no idea what the GP's motivations were. I'm not going to give any ground. You and the GP are taking a complex issue and boiling it down to something absurdly simple. All that I'm saying is that it's not simple and, if we're ever to have a reasoned debate, it needs to be predicated on facts, not "feelings". If you're going to attribute someone's actions to their gender and race then you had better be able to back it up.
It's telling that you believe _I'm_ the one doing harm when I have made no accusations at all and the GP was inarguably expressing a racist view.
It was not in fact a racist view. It's a view about power dynamics in a system that is racist and sexist.
If you really can't get over your feelings of personal insult and your need to defend white men, then please go back and substitute "part of the socially dominant racial group" for "white" and "part of the socially dominant gender group" for "men". In US history, those terms are equivalent. If you don't like that, talk to the founding fathers who set it up that way.
The reason it's material here is that a large portion of Twitter abuse is related to this. You can look at famous incidents, for example. Leslie Jones received a giant wave of racist, sexist abuse: https://www.thecut.com/2016/08/a-timeline-of-leslie-joness-h...
And the company admitted they had failed massively: https://people.com/movies/twitter-ceo-jack-dorsey-speaks-out...
You could also look at the differential experience of white men versus others on Twitter. E.g.: https://www.theroot.com/twitter-has-a-serious-harassment-and...
Note also your standard here. You, a white man, refuse to accept that race or gender could be relevant to a social problem until other people take the time to spoon-feed you. Why isn't it your job to know something about American history? Why isn't it your job to learn about the problem of online abuse before deciding that the privilege afforded to white men couldn't possibly be the relevant? Answer: because you were raised to believe that whiteness and maleness were the default and were good. So you choose that as your supposedly neutral stance, and require others to prove you wrong.
The true neutral stance would either be "I don't know about this problem and would like to learn" or "let's assume that online culture has some relationship to the existing offline social dynamic".
Wealth is very nice to have, but it does not insulate someone from suffering racism and sexism.
it's not either or. privileged AND hardworking is a possibility.
> Who's to say they are "privileged"
The employee who is familiar with them.
Yes, exactly.
The problem is not the genetic characteristics of whiteness or maleness. It's that they grew up in a society that favors whiteness and maleness, and is much more accepting of the abuse of people who aren't so favored. If Twitter had started in China, where the Han are favored over ethnic minority groups like Tibetans and the Uyghur, it would be the same dynamic but with different ethnic groups.
Of course, tech is disproportionately white and male, so it's entirely unsurprising that HN readers have risen up en masse to defend the honor of white men. It exactly proves the point. White men will say that fighting abuse is important. But when they have to show their real priorities by how they allocate time and effort, actually understanding the abuse problem and how it disproportionately effects minorities comes well after defending their own interests.
(And: Yes yes, not all white men. Generalizations are general statements that may or may not apply to an individual.)
It's late. I'm tired. I left a couple of replies earlier, then deleted them. This seems like a just no win situation to me.
I'm not a feminist. I'm a former homemaker that self proclaimed feminists routinely crap on. I come to Hacker News because I need a place where I can engage in meaty intellectual discussion so I don't lose my marbles. That is my only agenda for being here. However, dealing with sexual politics is not something I can entirely avoid what with posting as openly female.
Suffice it to say that I don't really want to be associated with this toxic attitude and I am kind of wishing I had not said anything at all. I am starting to feel like "The only winning move is not to play" and I failed to make that move.
While that's true, you could at least tip your hat to Jack Dorsey's $3bn net worth, the insanely high executive and engineering salaries in San Francisco, and the effect that being so insanely rich must have on the complacency and sheltered lives of everyone at Twitter. It's a little bit worrying that you think race and sex are greater assets than class (= "network" and "culture fit") and money.
There is also an interesting 'get off my property' aspect to this acquisition. It may indicate that there is a desire within Twitter the company for complete control over Twitter the social media platform. Because they can, Twitter will always reign over the mirage of an ecosystem built on their social media platform.
I think to a great extent that can be (just about) assumed about anyone in a leadership role at a company the size of Twitter. Regardless of anything else.
Implying that white males don't care is not helpful and is both racist and sexist. I'm a white male and I care about abuse and all kinds of other things. Maybe you should do some reading on unconscious bias.
White males who don't show compassion move up ranks very fast in this society.
Not judging the individuals, but the history of this culture is very well covered.
I don't think Hanlon's Razor is an acceptable excuse above a certain pay-grade.
100% agreed. And I definitely don't mean it as an excuse. Just an explanation for why conspiracy is unnecessary.
>mainly white and mainly men. I don't believe they really care about abuse in a meaningful way.
That's an awfully biased viewpoint you've got there, good grief.
Lot of people need to take a recent history course. The trend is generally powerful white men abusing the rest of the planet.
FYI
> Their leadership is mainly white and mainly men.
Wow. Racist much?
Another plausible scenario is the acquisition terms required Smyte to wind down the service over the past 6 months or build a replacement to fulfill contracts. Smyte didn't for whatever reason (either through negligence or negligence+hope to continue operations in a new capacity) and the deadline came to hand over the servers in the agreed condition (no externally accessible APIs available).
Twitter could have stepped in and halted things, but that would have required Smyte to have acknowledged breaching the contract and forfeiting $$.
This is all guess of what seems to me more likely than any Twitter exec pulling any triggers like this. Of course, it's still a screwup by Twitter to have not been tuned in and aware that fingers would point to them.
But that's a very different kind of mistake than "I have an idea, let's hit the power button at 6:30. Team: you have 30 minutes to let all our customers know."
You never, ever agree to a change in your business before the acquisition is complete. The chances that an acquisition won't go through are just too high, and it could leave you completely f'ed over.
Yeah, but if you're less experienced, or if you're more devoted to your potential profit than to your customers, you might agree to the term, and then realize what you just said, leaving you with the (immature but perhaps "seemed reasonable") situation we see here - doing nothing until the acquisition is complete and then immediately abiding by the terms.
To a large degree, the responsibility is also on Smyte, right? Selling to a big corp is one thing, and once you are no longer owner of a company you've washed your hands of future decisions. But at the time when they agreed to the sale they were still owners of the company and their hands had not been washed and they could have and arguably should have stipulated some conditions for how the service would be wound down following the sale.
Litigation aside, the founders of Smyte seem pretty culpable here. I would be wary of using another service that they created, given a demonstrated willingness to turn it off without any notice.
Only if the acquisition hasn't closed yet. If it has, who's there to go after?
But you're actually right -- if the acquisition hasn't closed yet, they're independent and independently responsible.
Twitter’s VP of engineering has posted some additional information: https://twitter.com/michaelmontano/status/101024630798476902...
"We couldn't operate their business and continue collecting data from their customers, while continuing to meet our own high standards as a global company."
My hunch as an outsider is that Smyte wasn't GDPR compliant. Their leadership knew it, they knew they couldn't easily become so (for instance, they may have been using Kafka in a way where compaction wouldn't help, and didn't want to build an encryption-based monstrosity [0]), realized that they wouldn't increase in value as a business due to that risk, took an acquihire for cheap in order to give their employees a decent landing and give a return to their investors, and couldn't tell anyone about these plans in advance because it might jeopardize the transaction.
EDIT: They were indeed using Kafka per [1], and due to the strict latency requirements on their business, that may have ruled out the type of encryption scheme in [0].
[0] https://danlebrero.com/2018/04/11/kafka-gdpr-event-sourcing/
[1] https://www.youtube.com/watch?v=6ByXQfIq5uU
I've seen lawyers advice that [0] doesnt do the trick anyway, ie deleting the key is not enough to comply.
GDPR does seem like the most likely culprit here, if they've been planning an aquihire for a while it wouldn't have been worth implementing GDPR and Twitter likely made it a condition that the service is shut down before the acquisition completes
I'm quite surprised none of the cloud vendors were interested/could offer more than Twitter for this team though, seems like a logical service to add.
That's a good thought. The only good reason to shut them down so abruptly is if they posed an existential threat to Twitter itself. The threat of GDPR fines or sanctions might have done it.
I still don't get why they couldn't just come out and say that. Was incompetence and unprofessionalism really a better look then a understandable engineering challenge? Or was this another case of "being honest and doing the right thing would have created legal liability so we opted not to do it"?
Generally your lawyers will advise you that the less you say, the better.
Maybe, but then why would Twitter acquire Smyte in that case?
Presumably because Twitter was interested in the project, but didn't need the actual implementation immediately. Get the team on the cheap for now, and then just have them rebuild it in-house in a compliant way.
This is one of those instances in which I think employees (developers) and companies working with Twitter need to have a "never forget" attitude. I actually know someone personally at Twitter corp dev (the group that usually manages M&A activity) and I hope that individual was not involved here, as it would be pretty out of character.
Why would Twitter care? They make money from ads, right? APIs and legacy deals are going to be time consuming and boring for them. Stuff like filtering hate speech etc is something which can be done globally and in a way which doesn't require highly paid employees, APIs etc. Hasn't Twitter repeatedly show that third parties aren't really that interesting to them?
All companies have suppliers and partners, including Twitter. Getting a reputation for this kind of toxic behaviour harms their ability to build relationships with them, as nobody will feel they can trust them.
It could also hamper their ability to make further acquisitions. In future, a startup being linked with a possible Twitter acquisition is going to cause its customers to panic and start moving elsewhere. That'll make startups more reluctant to engage, and may turn some off entirely, if they're not comfortable with the prospect of totally screwing over the customers who've supported them during their early stages.
A startup's customers panicking? Not sure that's on twitter's radar.
I think potential for reputation damage, on an activity they don't do very often, and where they will justify it, will be weighed up against the financial benefit now, and disregarded.
I think it's possible to overstate the relevance and importance of people typing furiously on internet forums about things like this. Twitter is what it is - most people aren't using APIs, third party clients etc and will never notice this.
I never said Twitter cares about startups' customers panicking, but it cares about being able to make strategically useful acquisitions, like Smyte. And acting like this makes it harder.
This isn't about people typing furiously on internet forums. This is about business risk, as weighed up by people whose job it is to care. Acquisitions take time and negotiation. Just because a startup is talking to Twitter, doesn't mean the acquisition will definitely happen. But if customers get wind of it, then they may start looking to move elsewhere, anyway, instead of waking up to find services they rely on have been shutdown without warning. If you're the customer of a startup Twitter is negotiating to acquire, then you're negligent not to reduce your reliance on them. And if you're the CEO of that startup, then you have to consider the risk of customer flight (and your own comfort with the thought of screwing them over) vs. the likelihood of the acquisition happening.
if the Smyte shutdown limits the opportunity of current and future Twitter associated startups, whose business model is to help companies safely navigate the social media landscape, then that sounds like a win-win for Twitter
But the very behavior you're talking about... didn't make Smyte not sell. So even they didn't care.
Why would anyone else?
They can shrug it off, still those companies that had contracts with Smyte are not one person businesses on the other side of the world. Those Twitter people are going to have lunch or dinner with other people and get to talk about this issue for a while. Not very pleasant, even for employees totally unrelated with this case like the one who answered the very first comment in this page (sorry.)
> A startup's customers panicking? Not sure that's on twitter's radar.
Let's not forget that those customers are also likely candidates to advertise on Twitter
Twitter sells data through an API. Unsure how profitable it is. See: https://developer.twitter.com/en/enterprise.html
We had 20 minutes notice, and then everyone was kicked out of the Slack support channel and API responses simply died. What the actual fuck. They have mobile client SDKs out in the wild that are now just eating up battery life as they retry an impossible query forever.
That sucks. We (Sift Science) have been building something similar for 7 years and aren’t going anywhere. If we can help, please ping me - jason at siftscience dot com
> We (Sift Science) have been building something similar for 7 years and aren’t going anywhere.
But that doesn't come with any actual guarantees does it?
> and aren’t going anywhere
Yet... that you know of.
Nothing specific against Sift Science whatsoever in my comment, but many, many times this sentiment has been conveyed by companies and it rings hollow IMO. There are a lot of different ways a company can change or disappear at some unforeseen point in the future, and claims that "you can trust us to be here forever" do not carry much weight for the large group of experienced users, devs, management, etc who have been burned multiple times.
So are you implying that you should never enter into a business relationship with another company?
Clearly, you need to always be prepared for your partner to go away, but you still have to work with other companies.
I think the parent commenter is just saying that it's not something you can actually believe, just like a company saying they'll never sell your data.
Sure, that's the intention but when someone comes around waving a checkbook, they can in turn buy you and shut you down, sell your customers' info, etc. Sure, its not YOU allowing that to happen but by ceding control you essentially allowed that to happen.
"You can trust us to be here (until enough money comes our way)" is just how it is for most businesses like this. An unfortunate, and in my opinion largely unresolvable, side effect of an ecosystem where specific useful tools are hard enough to create and manage that full companies have to be formed around them.
What kind of guarantee would be satisfying?
A third party code escrow account with instructions on how to build the service and a contract that specifies conditions when a customer gets access to that escrow.
I've been on both the customer and service end of such agreements.
Is there a popular provider for doing this? It sounds like a lot of work to find a solution in which both parties trust (both technically versed and reliability wise). A common notary probably wouldn't cut it, right?
Both companies I’ve worked for - as a client and a customer - used Iron Mountain.
http://www.ironmountain.com/information-management/software-...
There's a bunch of companies doing it, some dedicated, some as part of larger related offerings. E.g. I know NCC Group (large IT security company) does offer it, TÜV (which does all kinds of certification and compliance work), ...
The contract should specify. The contract probably was sided on smytes favor. Screwing even paying customers.
You can have a contract where you specify indemnification of damage in case the business ceases operations, goes bankrupt, get acquired, etc...
A guarantee is only as good as the guarantor. A guarantor can be a company providing a service, as well as all sorts of assorted guarantees about said service.
In time, the situation can change. The product can get sold. Cash can be spent. The guarantor can no longer make good on its guarantees.
You're back to square one where the guarantees are as worthless as the original service.
You need 3rd party backing (insurance) in such situations, but that costs money. This money is a cost which makes competing against unbacked entities tougher.
In most cases, you can not have 100% foolproof guarantees of anything. The closest I can think of is governments standing behind their banking institutions. Even there though, governments have defaulted on their guarantees.
The world is not a stable, perfect, and cut-and-dry place as many would like to believe. It is dynamic, and ultimately backed by trust.
It's much harder to mitigate this kind of risk in the world of services as against purchased software.
The problem has always been present especially when larger slower moving companies buy from smaller, riskier, companies.
In the days of software, code escrow was possible to mitigate some of this kind of risk. That's still got it's costs but can be an effective hedge against a supplier going bust.
The parent comment is justified.
While you can have a contract that indemnifies that isn’t going to help your going concerns when the other party just pulls the plug.
I think company bylaws that prevent a 'triangle merger' like this might be useful.
what's stopping the company owners -who are generally the ones executing such a merger- from changing the bylaws ?
Do you have a high level overview of what I’d need to do to get started? Are there any specific areas you specialise in as opposed to general users and what information will I need to share with you?
Shockingly bad. I wonder if they shitcanned the whole technical team and they turned the lights off on the way out. Not that it’s really much better but at least it wouldn’t make sense.
And there will be absolutely no consequence to these actions. Completely messed up!
I hope you think over that design the next time. Put it in the cloud they said ...
I actually ripped out Smyte's Android SDK because of how bad it was and replaced it with my own implementation against their API which has a hard limit on retries. But I imagine that other customers were using their SDK and still have live apps out there using it... not good :/
Oh please. Don't pretend that battery destroying, relentlessly network hogging apps are a good thing.
No app should retry an impossible query forever. This is not the same as some sort of laughable abstinence-only birth control policy, where you get to roll your eyes, and say "yeah right. good luck with that."
What's more, an app that hits the network of an uncontrollable third party, should always check with a controlled system periodically, to verify that the third party shall continue to respond predictably, as a sanity check.
And beyond even that, if the sanity check within your own domain, for your own app that you created, and are responsible for, is unresponsive, it really should cripple the app's connections to things you don't control.
Finally, as one last safety measure, if your status/activation end point dies, and your native apps go dormant, you should expect to be able to re-activate them with a number of channels, including emails, push notifications and new releases that up-version the app.
If you build your kingdom within a walled garden to begin with, and rent all your tools of the trade from an external platform, expect to behave as a subsistence farmer.
> Indiegogo, GoFundMe, npm, Musical.ly, TaskRabbit, Meetup, OLX, ThredUp, YouNow, 99 Designs, Carousell, and Zendesk
Half an hour of warning, to screw over all of these?
They broke npm's user sign-ups, and publishing of packages, with half an hour of warning.
I can't imagine the havoc over at Zendesk either.
That is not 'winding down'. That's ghosting.
It's worse for companies that have shipped mobile apps with the API embedded in the client. Web apps can at least try an alternative or disable the affected features until they work around this mess..
This confirms my belief that I don't want to ship anything that uses directly a third party API. Instead always go through your own server which then calls the API so at least you have a chance to replace that.
That works for a lot of cases, but for many tracking APIs you want the request to come from the end user, so the tracking service can get their IP and network metadata. That would certainly apply to a fraud detection API.
Tbh that has always frustrated me though. I should be able to pipe those requests and just add an X-Forwarded-For header.
This is not my area of experience, but is it feasible to capture all raw tracking data from the client devices and then just pass it on to the API?
That's basically a proxy or MITM, no?
Yeah, but it sounds like it would satisfy the concern of being able to replace the 3rd party API internally without the mobile apps having to change.
But it will have the cost of 2x API bandwidth (vs 0) and another point of failure that you are responsible for. A DNS CNAME probably wouldn't work if you had to go over SSL. Maybe a 30x redirector (still have the SPoF, but simpler and much less bandwidth)? Except in the past, I've found that most clients react poorly to redirects for anything other than GETs.
Is there any indirection that would avoid having to walk the traffic over my own network?
The way I would implement it is as a “proxy” on my own network, eg trackingservice.myapp.com, that is just a web server listening and forwarding requests (technically a MITM). Override the request method of the client SDK to hit my endpoint instead of the tracking service.
Obviously you then pay the bandwidth, but it’s likely negligible compared to your app traffic. As a bonus you’ll probably circumvent adblockers.
If you want to avoid traffic forwarding, but keep flexibility over the endpoint, you can override the init of the sdk to first query your server for which endpoint to use. That way if the third party service goes down, you just need to change the config on the server.
Damn right I'm the man in the middle if I'm the SSL-offloader. It actually strengthens security because there's one less party to trust for the client.
When I was developing mobile apps, this is what we did. Too many times something changes and trying to update the code in the mobile app is way more of a pain than updating the server.
> Clients had multi-year contracts in some cases.
I wonder how much Twitter is going to pay on top of the acquisition cost of what I can only imagine constitutes breach of contract with all of these companies. (If it doesn't, why even bother having a contract with a duration?)
Kind of depends on the terms of the deal, namely whether it's an asset deal as opposed to a stock deal. In the latter Twitter would assume liability for contracts. In the former they just acquire IP and hire away the employees.
Acquirers usually prefer asset deals for this reason as it allows them to leave unpleasant liabilities behind. There's generally some sort of shell left that goes through wind down but it has few/no assets attached. In this case you can try to go after the original investors including founders or other shareholders (the IRS may do this if there tax issues) but since there's nobody home you are less likely to get much out of a favorable judgement.
This seems like the kind of thing a legal system should be designed to help defend the market against. It's a net negative to the economy as a whole to let torpedoes like this fly without consequence; and evidently, a totally Laissez-faire market will engineer ways to insulate the perpetrators against the damages rather than the insulate economy as a whole against the splash.
On the flip side asset deals also let acquisitions proceed that would otherwise be blocked due to uncertainty about liability, which is another way of saying that the acquirer can't properly assess value of the investment. I have been through this process and have some experience with the type of issues that can arise--in our case they had nothing to do with screwing customers. It's possible Smyte would have just disappeared anyway.
Moreover recovering damages in this kind of case is time-consuming, doubtful, and too late to fix anything. (Like closing the door after the horse is gone.)
What's left is reputation--the names of the founders of Smyte are listed at https://www.crunchbase.com/organization/smyte. I would have some pretty hard questions if they ever showed up in a deal or looking for work.
p.s., According to the Techcrunch page "Smyte stops bad actors on marketplaces and social networks." You can't make this stuff up.
I'm pretty sure their contracts covered their ass. Hence why lawyers write up contracts in the first place.
You can write just about any words you like into a contract. But that doesn't mean they are enforceable and doesn't mean a legislature or court can't add or modify clauses. Then there's the whole doctrine of equity thing over the top of it. In law there are always a whole bunch of fine-grained questions that can change the whole outcome.
I expect one or more lawsuits to emerge, maybe as a class action. It might not come to trial and simply wash up as a settlement.
I am not now, nor have I ever been, a lawyer.
Both parties presumably had lawyers writing ass-covering contracts, so who loses? This isn't the usual "Wells Fargo screws you over then forces arbitration where you lose again" imbalance, there's no particular reason to think Smyte could have swung a favorable contract with e.g. ZenDesk.
It really depends on who was negotiating, whose paper was used to write the contract, and how much each side wanted the deal. Legacy enterprises like Disney often force vendors to use the legacy company's paper, which case substantial protections would be written in or the deal would not happen.
Self-service web services on the other hand generally come with take-it-or-leave-it conditions of use. When's the last time you redlined Github terms of use when starting a project? ;)
More companies than that, and large ones. These are just the ones that went public... some of them don't want you to know their trust/safety/moderation features just went dark. Smyte just ruined a lot of people's day(s).
Twitter. Twitter just ruined a lot of people’s days.
No. Smyte had a responsibility to its customers, and should not have entered into any deal that did not include some reasonable wind-down plan.
Had.
Twitter owns Smyte. When you buy a company, you acquire all of its responsibilities. There is no longer a Smyte to point the finger at. It's all Twitter.
This is pedantic. "Smyte" means "the people who formerly made up Smyte, especially the founders".
Legally, the onus is on Twitter for pulling the plug on an API that was being used actively without much notice.
I’m actually surprised that it was done in this manner, as many startups have a “business continuity” section in whatever contracts are drawn up with customers, detailing the specific steps and timeframe of retiring a particular service. I would have thought this is standard practice for a SaaS company.
My point was that we shouldn't point the finger at Smyte because that lets the blame drop on the floor when the "Smyte" brand evaporates.
Twitter owns this now and the way to ensure it's their reputation that is affected over the long term is by correctly calling this a "Twitter" failure.
Have we read the contract? Do we know what happened?
Twitter is the one responsible now.
What about blaming the companies who decided to depend upon a service that they have no control over? I find arguing against using third party APIs to save a few days of development a constantly losing battle, with it being presented as a benefit with no costs or risks. Do those pushing/agreeing with such a view not share some blame?
> What about blaming the companies who decided to depend upon a service that they have no control over?
Unless you're advocating against any kind of *aaS, this is kind of a silly argument.
A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
Obviously when you make your product/company depend upon these things, you're taking a risk that they could go away, but you protect yourself against that by having robust contracts.
Outages are different, but for a significant outage you expect to see some kind of RFO and explanation of how they're going to mitigate it.
Deliberately withdrawing service for all customers with 30 minutes notice? That's entirely the fault of whomever is in charge at Smyte.
Even 30 days, while it'd probably ruin some existing plans, would at least allow people to have a chance to migrate without things just breaking.
>A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.
For the cases I was involved in, there were options of buying the service but hosting it internally with some update pipeline. So even if the updates were suddenly turned off, you have a far longer time to swap to a different system. Generally deploying these aren't to much time, but what is really the issue to me isn't that the options that were chosen were the ones which were chosen, but that the process of choosing them did not evaluate the risks of creating such a strong external dependency at all. Its one thing if it is a risk taken with full knowledge of the potential costs and benefits, it is another when people do it because they think there is no possible downside.
Are you saying that:
* No website or app should ever call out to any third party API
* Every single piece of functionality a website or app has should be developed in-house
* The functionality third party APIs provide would often take "a few days" to reproduce in-house
No website or app should ever call out to any third party API
Definitely. Your website and app should only call your APIs. Your API can then call the third party API from the server. It's a lot easier to change something on the server than having to update the app.
I wanted to argue with you for posting this, but you're right, this is a smart way of handling API calls.
It's not really addressing the whole issue of depending on third party APIs if your API is calling on them, but you are technically correct, which I've heard is the best kind of correct.
> What about blaming the companies who decided to depend upon a service that they have no control over?
No. Delegation is the cornerstone of civilization. Trying to build everything yourself is a terrible idea.
> that they have no control over
Per the article, they did at least have long-term contracts locked in with Smyte. That's not the same as control over the feature, obviously, but plenty of companies do business on the strength of contracts instead of vertically integrating their entire product.
Obviously we don't have many details yet, but I think there's a real difference between simply using a third party API that works for the moment, and buying a service from a third party. If Twitter/Smyte simply broke contract with all their existing customers, that's very much their responsibility.
This isn’t really a leftpad issue. This was for fraud prevention.
Twitter has ruined a lot of people's days for a long time now. Especially in the Trump era. Nothing new here!
Not sure why you are being downvoted since Twitter has a well-established history of pulling the rug out from under people who depend on it.
What would be these businesses backup plans if Smyte went bankrupt instead being acquired?
A company going bankrupt usually doesn't give 30 minutes warning that they are shutting down services though. They give more warning than that.
Presumably you would get more than half an hour to migrate.
Your supplier going bankrupt is a well known risk that you cover by checking its credit rating and by paying on or after delivery.
That's why often startups can't do business with big companies that hold to rules like that.
Would Smyte's rating have cut it for people?
Your supplier getting acquired is also a well known risk that you cover by reading/negotiating the contract.
The problem is, if all suppliers use the same contract clauses you may not have a choice.
How do you check the credit rating of a startup?
and how would their credit rating be affected if they pay all their bills but just reduce the number of bills gradually over time?
One way that's sometimes done is that you require them to submit an independent respectable audit of their financials, or in case of confidentiality issues, not submit the actual financials but a statement from a respectable auditor that they have verified the suppliers financials and their cashflow/"runway"/etc satisfies the criteria that you required.
I'm not denying that what they've done is really bad, but who puts a 3rd party API in a critical path without having any fallback (alternative service, reasonable timeouts, etc.) for when it goes down (as almost any service will eventually do)?
Everyone that does online transaction of money, and many more. Like all those who use AWS services on AWS.
People who have multi-year contracts with those API providers, probably with an SLA detailed in the contract. Especially since this API call would have to be on the critical path; you don't want people signing up when the anti-spam service isn't running.
1. You have an SLA guaranteeing 100% uptime?
2. If it's that critical, you have redundancy (an alternative provider, or a naive implementation that maybe causes more false positives but that will cover you during an outage)
I might be wrong, but I'm getting the sense that you don't have much actual experience.
I've been a senior developer in an organization and known that a solution isn't perfectly robust. Clearly I want to fix it because I (like most other developers) like building robust solutions, but I have six weeks of tasks to finish in two weeks, and that isn't one of them. What do I do?
You might say "do it anyways" and, if so, that's a junior developer attitude. I'd strongly urge you not to do that as shipping big projects is really more about choreography than engineering.
Or, you might say "fight for permission" and that's not a bad idea, as long as you accept that you'll likely lose.
You'll likely lose for business reasons. On some level, we all know that a SAAS on a critical path is a bad idea, but we do math. Our SLAs are not 100%, but we can accept a few hours of downtime a year. And, the probability of a service you've seemingly vetted being acquired and shutting down this quickly is low. Is the probability of a shut down like this high enough to justify adding and testing redundancy?
The math usually works out in favour of bug fixes and new features > redundancy.
In the first place, it was an awful business decision to take the route of trusting a SaaS provider with such an important task.
It is extremely irresponsible to forward sensitive user data and site interactions to a third party, even if it is in the name of spam/abuse/scam filtering. That its implementation brings down production websites in the case of service failure only adds insult to injury, originally caused by plain-awful product design.
If there's one thing that you should be doing in-house when you run a large online community of privacy-conscious users, it's the kind of service that Smyte provided. If sound business reasoning had prevailed, npm would have suffered no downtime.
It's easy to say that now that everything has gone to hell, but in business, we usually have to make decisions with imperfect knowledge.
I don't agree that it's irresponsible to forward sensitive data to third parties. Rather, I believe that intelligent companies have intelligent policies around sharing data with third parties. Some third parties provide an amazing service that would be extremely expensive for companies to mimic in-house. And, you cannot convince me that the probability of a well vetted service shutting down as fast as Smyte did is high enough to justify bringing this in-house when you have other things to work on.
No, but if I were npm, I'd have an SLA guaranteeing three-and-a-half or four nines of uptime (which is far from unreasonable) and just accept that there might be times that signup is not accessible because that'd only cost me a couple hours of signups a year.
And no, I wouldn't have an alternative provider. It would not make sense to, if I'm paying a company to provide a service. Why should I pay another to do nothing?
I know for a fact that you've done worse in your own life. Spare us the moral posturing, please.
What you just did is called a “Tu quoque” argument, and it is a type of Ad Hominem attack; i.e. a fallacy.
https://en.wikipedia.org/wiki/Tu_quoque
Good effort, buddy, but no.
This whole thread is moral posturing about how unprofessional Twitter/Smyte have been, but if we want to talk about professionalism we should look at ourselves.
If you upload to AWS S3 you put in a fallback to another service? And any other AWS services you use? That's a lot of fall backs. Is the second fallback good enough? Should we add a third? In any practical sense development would grind to a halt. Twitter/Smyte are totally in the wrong for not giving a reasonable notice. I think whoever made this decision should actually be removed from their post and the servers should powered back on again with an apology to all of their clients. If they still want to shut it down later then they should provide a reasonable time frame.
Maybe I'm in the minority, but I take an interest in the latest software/tech/ideas for dealing with the hard problem of moderation, and I had never heard of Smyte, until now (which made me think TC was being literal with its "Twitter 'smytes' customers" headline).
Apparently, they were a dominant player in this space, such that its shutdown impacted such a wide array of significant-sized companies, from ZenDesk to TaskRabbit to Meetup. Even more surprising is that they were part of YC (W15), and yet from what I can tell, have only had one significant mention on HN from 3 years ago (65 upvotes, 11 comments)
https://news.ycombinator.com/item?id=9758464
I would've guessed that Smyte would've been the kind of Silicon Valley business that was made for media hype (what with all the attention/criticism given to fake news and Google/FB/Twitter moderation). Then again, it does seem that content moderation -- important as it may be -- isn't a field as sexy as AI/deep-learning and other forms of automation.
They were also noteworthy for their founder Pete Hunt, who is largely credited with instigating the open sourcing of React. (As I recall, it was internal Facebook tech that he extracted into a separate library when he moved from FB to Instagram after that merger.)
Which is funny because they were actually dabbling in some AI/ML.
Given that these businesses had contracts with Smyte, how is this not a flagrant violation of the contracts? Surely Twitter took on Smyte's contractual obligations when they acquired the startup?
There are many ways to accomplish this, for example, doing it as an asset purchase instead of acquiring all the stock.
In the former, you can take the assets (including employee contracts) without the liabilities.
There's actually a reasonable quora answer (for once) on this, saving me from writing it up for you :)
https://www.quora.com/What-happens-to-legal-contracts-financ...
In a case like this, does that mean that Smyte (the company, even if they no longer own the trademark) still exists and is liable for all the penalties? Would they usually ask for the cash to pay for them as part of the asset purchase?
In a case like this what's left other than an empty sack? Unless you're able to pierce the corporate veil there isn't anyone/anything with assets left to be made whole by (e.g. sue).
One could argue the transfer was fraudulent. And go after both parties. Twitter and the founders.
Don't know, not a lawyer.
IANAL but I was a plaintiff in a similar case. We were made whole in a settlement but before that, the judge tore the defendants a new one over this tactic leading to a bitter dispute between the acquierer and shareholders (or the board of officers, don't remember) over who was liable for breach of contract. Eventually it came out that the auditors had done the math and included cost of litigation/liabilities on these contracts into their valuations, basically admitting that they knew about the obligations beforehand, so the acquierer ended up settling.
At the end of the day, judges (and appellate panels) have to interpret the law and they don't function like automatons. They take into account the spirit of laws/contracts as well as the letter and most have enough common sense to get missed off at these naive tactics.
The argument that penalties should be paid out of the purchase price would probably hold water. When Smyte entered into those contracts they took on some duty to honor them and to suddenly dishonor them and divert all the assets out of the company so that penalties wouldn't ve avoided would be constructive fraud.
in that case smyte would still exist and now have lots of money from the acquisition you could sue for
Well, usually they are using the money to wind down.
(All of this is a good reason i hate LLC's. They aren't necessary anymore to actually do risky innovation in 90%+ of cases. In a non-LLC, they shareholders would be liable, and then you'd still have someone to go after)
Yes, it does mean the former.
Usually, they would use the cash to wind down.
This is the thing that occurred to me. Surely those contracts don't have a clause that says "Smyte can terminate all services with no notice"? If so, surely someone spotted it by now? Or is this normal in service contracts in the US?
I guess the liability is covered by the usual "no liability outside provision of the service" clause - so you can't claim that you were damaged by their lack of service. But I can see a lawyer making a convincing case that the liability should rest with Smyte. IANAL, obviously, but it seems like the sort of thing they love to fight about ;)
gets popcorn this one will be interesting...
It is in fact a possibility that Smyte's terms were extremely one-sided.
Perhaps the default terms were Smyte-friendly, but I can't imagine that some of these big companies (with savvy lawyers) would sign an agreement with a change-of-control provision that would permit this.
There might be a carveout for consequential damages (which would be huge here), but this seems like it will trigger a number of sizable lawsuits against Smyte's new owner.
Does anyone think they companies that just got the shaft have legal recourse?
If nothing than just getting a pound of flesh from twitter.
Incredible. Shutting of API access to paying customers after a 30 minute heads up at 6AM.
Astoundingly unprofessional.
listen all y'all this is sabotage - such a good Bestie Boys song!
"trust and safety as a service"
So... what was the Service Level Agreement? and the Terms Of Service?
People get all hyped by -aaS stuff, but services do go down, and if you aren't the one controlling when and how they will go up, then you better have a contract with the Service Provider on some terms you like.
Even then, prepare for when they go down, because they will sooner or later, for shorter or longer periods of time.
I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
I find it silly to rely on some service, any external service
This is always trotted out and it's, by this point, a completely content-free thing to say. Short of removing yourself from the economy and society in general to live off potatoes grown in your own dung, you're going to rely on some external services and the fact that, for the most part, the parties you do business with will behave responsibly. When they behave very irresponsibly, unpleasant things happen and people quite reasonably complain.
Exactly. You want reliable CAPTCHAs? CDN service? Malware detection? Customer support software? Website uptime monitoring? Accounting software? Analytics? Email delivery? Good luck building and running all that yourself.
All of those can be bought and then self hosted except for CDN.
I don't know why you're getting downvoted because you're absolutely right. Perhaps most of the people in this thread haven't managed servers for more than a decade - because it used to be (and still is in some organisations) the norm to self-host half the stuff that was listed.
Isn't that pretty must how most malware detection works? Sure they use the cloud to pull periodic updates but the software itself is self hosted I've lost count of the number of solutions available to self host. Both free and commercial. While there is a strong argument for hosting that in the cloud, there are umpteen solutions for website monitoring. I didn't even realise running accounting software as a service was now a thing! Fair enough, self hosting email can be a massive headache. Less so if you run Microsoft Exchange (one of the few things Windows actually makes easier than it's Linux / UNIX counterparts). You can even go for a hybrid approach and self host the mail server but have a mail proxy between your server and the outside world - so you benefit from spam detection as a service while still hosting your own mail.I'm not going to argue that one is better than the other when it comes to SaaS vs self hosting. But people in here seem to have very short memories when arguing that self-hosting isn't at all viable. In fact I have personally supported all of the above in self hosted versions in the last couple of years.
I don't think anyone is disputing that it's possible, but rather how cost-effective and time-efficient it is [1].
I also have the experience of managing many of these services, and I therefore know just how much more productive you can be when you don't need to do it yourself.
Of course there are trade-offs; business is nothing if not a game of tradeoffs. We're just talking about what is optimal.
[1] Not to mention how much more stable and secure; Self-hosting can involve huge security and reliability risks.
> We're just talking about what is optimal.
I don't believe what is optimal can be generalised like that. "Optimal" is going to be specific to the business in question. Which is why I'm a firm believer of leaving all the options on the table.
> Self-hosting can involve huge security and reliability risks.
In fairness so can the cloud. I personally don't see difference as 'risk' but more 'responsibility'. Some of the places I've worked have been PCI DSS and Gambling Commission audited so we had that responsibility already. Self hosting a few other resources didn't really add much extra in terms of our security responsibilities. But the case would be very different of lots of other different types of businesses.
I think we're agreed that one shouldn't be absolutist about these things and that different approaches will be optimal depending on the circumstances.
It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.
So the discussion is easily resolved: don't be absolutist :)
> It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.
Even that strikes me as a mischaracterization, or, at best, taking just one statement out of context.
It was an assertion about the folly of relinquishing control, not being prepared for that lack of control, and thereby subsequently suffering a prod outage.
To be fair a couple of those don't qualify as "if it goes down it would cause a prod outage".
hosted where? On a VPS, in a data center, or in your closet? All of these depend on third party services and good luck keeping them up and running as well as someone who makes just that ONE little thing their entire reason for existing.
> On a VPS, in a data center, or in your closet?
Yes, any of those sounds like a good enough plan B. You don't need it to be perfect, just good enough to avoid getting stranded in a pinch. Then you can start shopping around for someone who does that one little thing much better than you do. That way it no longer is a show stopper and rather just an optimization problem.
I don't agree with the parent either (I'd rather use a SaaS if it makes sense) its also not as easy as you make it sound.
For starters, most smallish companies I know of use housing for their servers. So, they actually do own the hard metal servers and have full access to them. Both over SSH and on the console for debugging purposes. The housing provider generally replaces HDDs in case of disk failures. naturally, you're still dependent on electricity and internet, but even that can be mitigated by spreading your servers across several locations.
All of that is obviously possible with AWS, GCP or other VPS hosts, but doing it yourself is not that much harder if you've got an experienced ops person that wants to bother with all of that.
but i repeat... Its often not worth the time investment to get that system properly set up. Just renting VMs or going straight into dockerized microservices is an increasingly better alternative
That definitely solves the building part.
but real-time promiscuity where you spread your users info everywhere and have no backup plan is much more FUN. (/s)
No. Whether it's potatoes or servers, having a single point of failure with no plan B in place, is silly.
Your local restaurant going out of business is no reason for you to starve to death, you just go to another one, or to a grocery store, or indeed grow your own food if you're in some post-apocalyptic world or got no money... but somehow a service provider going out of business is reason enough for your production to stop? No, just no.
You've constructed a straw man, by removing an important part of that sentence and indeed the rest of what you are replying to.
Indeed, it's spurred a discussion of an over-simplification of the issues.
I don't think anyone is suggesting never to outsource, or even that the answer to "build vs. buy" is always "build".
The point is not having a critical dependency on a single vendor.
I don't think anyone is even saying "never", either.
The question is one of risk.
The risk of your tax accountant "going down" (or being crooked or something) is sufficiently low, and the cost of making that redundant is sufficiently high, that I don't think anybody does that. This could even apply to general accounting software, though one might argue that's not critical in the same way and/or more easily replaceable.
There's a sibling sub-thread that essentially implies, if not outright states that there's no choice but to "buy" in "build vs. buy". Although it may be true that building is no longer a practical choice due to time/cost, under certain circustances, it by no means eliminates the risk if there's only one vendor.
It's also a decision that can end up having tremendous costs at scale, if not re-evaluated, especially with even a soft version of vendor lock-in, like AWS.
That's not what the comment I replied to said, though. It just said it's 'silly'. If they had said something specific about, let's say, npm and made some reasonable and knowledgable comment or even speculation about how in this particular case they'd mis-assessed risk, that would be a pretty interesting comment. As it is, it's just a clone of the same tedious comment we get in droves in every thread about some multi-party outage. They're generic and free of insight.
> It just said it's 'silly'.
you did not include "so much that if it goes down it would cause a prod outage" in your quote, which is why the strawman claim was put forth.
It doesn't really change the quality of the comment. There is nothing inherently and outright wrong with such a dependence, whether it causes an outage in prod or not. In some cases it matters (maybe you're making a nuclear reactor) in many, it's a completely sensible tradeoff. Reflexively saying 'zomg you depend on something else and this is bad' every single time something is affected by a problem in another thing is 100% uninteresting. Including or not including 'in prod' or 'in bed' doesn't make it any more interesting or clever. It's just lazy grousing.
> There is nothing inherently and outright wrong with such a dependence
This is yet another strawman. The original commenter found it "silly", which, I agree, was a poor choice of wording.
Perhaps "overly risky" would have been better. I don't know. I just take the most charitable reading, per the guidelines.
It also comes after an exhortation to have a contingency plan, so it's at least implied that the "silliness" isn't inherent merely to the dependence but to the dependence without such a plan.
> in many, it's a completely sensible tradeoff
Is the tradeoff actually considered, though? Or is there just an automatic "we're not a nuclear reactor" decision process?
> is 100% uninteresting
Were that true, I doubt there would be replies. This is not one of those traditionally emotional/political issues.
> Including or not including 'in prod'
I (again, charitably) read "in prod" as a metaphor for "business critical". I'm sure we can all come up with examples, even in Internet companies, where one does not necessarily mean the other (or vice-versa). In the instant case, it seems to apply, so it made sense as shorthand.
> It's just lazy grousing.
I would agree if there were no built-in suggestion on how to avoid the problem at all, but there were.
Just because the point has been made before doesn't make it any less valid, until it has been refuted.
This is yet another strawman.
Only if 'strawman' means 'thing I disagree with'. The comment says such a dependence is bad and how you have to prepare for it with SLAs or whatever. This isn't universally true at all.
I doubt there would be replies.
A lot of really boring, trite things generate piles of replies. That's why they should be avoided.
Just because the point has been made before doesn't make it any less valid
The stated goal of the forum is not 'an exhaustive, if repetitive collection of generically valid things'. Remember that time you wanted to print something and the thing just wouldn't print? That's bad and annoying. It's valid that it's bad and annoying. We probably don't need to talk about it in the general sense every time it happens, though.
> This is always trotted out and it's, by this point, a completely content-free thing to say.
Only if you pretend there are no companies that do in fact responsibly nail this stuff down. There's plenty. Your argument seems to be that nobody does it because it's hard.
Welcome to the world of third-party Twitter clients. Not content with screwing over businesses relying on Twitter's API, it seems, they have now set out to screw over any company relying on any other third-party API as well.
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
There are pretty solid SLA's for that. We provide some services as kind-of-saas but the SLAs are watertight.
Usually in those kind of sla's you can't sell the company (or move assets) without guarantee of delivery of the contract. In some cases they (the contract holders) even own you if you screw up and go under.
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.
I'm curious; what would you have these companies do in such a scenario? If 100% uptime is impossible then what you seem to be suggesting is that people should never use third party services or if they do use them they must not be in a critical path.
Are you suggesting everyone should do everything in house?
Use free (FLOSS) software.
If you have to use non-free software, buy software, not services.
If you have to buy services, always have a well-tested fallback path for when that service is not available.
Care to point out the FLOSS equivalent of what Smyte did?
You missed two thirds of my comment.
This was in order of preference. If you can, use free software that you can fix and support yourself if need be. If not, buy the software and run it yourself, so you can at least control its operation. The last option is to by SaaS, and if you do, you need to have contingency plans in case it goes away. You should take this risk seriously, and factor it in to your decision when buying SaaS, and invest the engineering resources to make sure you aren't overly dependent on it.
For any critical service, they should have an alternate provider ready at all times so that they could instantly switch to it. Whether it's another external provider or in house, is not really relevant, although having a bare bones in house alternative for the critical parts is a good idea.
What I'm suggesting is that people should use external services for their lower costs, better performance, or extra features, not to 100% depend on any of them.
No. Smyte wasn't so business critical that they should consider a failure in the service a reason to stop the site. Netflix had an example where if thier authentication service was down, they didn't stop people from watching videos.
There is a popular C# package called Polly that has all sorts of fault tolerant strategies.
https://github.com/App-vNext/Polly/wiki/Transient-fault-hand...
In this case, use a Fallback strategy that responds to an API failure with some type of generic success response.
I feel like there must be more to this story. Maybe they immediately saw security problems, or legal issues. Not saying it's going to exonerate Twitter (maybe the truth is even worse than it sounds), but this is such an obvious PR mess that I can't see them doing it for no reason.
The @HelloSmite Twitter account stopped abruptly on June 4th after daily posts in May.
I postulate more than few roles were halted at that time.
That they didn't notice during due diligence?
Perhaps they still wanted the company for some reason? Someone knows. Time will tell most likely.
>Clients had multi-year contracts in some cases.
I'm calling it now: B2B startups will soon have to sign poison pill contracts that specifically detail what happens if they're bought out. This might ruin the "buy them for their people and discard the business" method of hiring - or at least it will if the customers have any sense.
I'm baffled if this wasn't already the case. Surely the entire point of signing a long-term contract (as opposed to just paying by the 'unit' used) is to protect against exactly this kind of outcome? I know it doesn't always happen, but it seems like most rapid "hire and shutter" moves hit either consumer-facing services or B2B stuff like analytics that isn't production critical.
I had assumed Twitter was going to be paying out a bunch of breach-of-contract penalties, because this is basically what contracts are for. Is there a standard way around this?
I don't know how it works in Silicon valley, but finance companies I have worked for have pretty much always required that source code be turned over if the vendor folds when doing business with small vendors.
Yes I worked for the company doing the .coop registry and ICANN had very strict rules about code escrow we had to follow.
It's amateur hour if your contracts do not already contain such clauses. For anything business critical that's the norm. Change of control clauses are super common and tend to trigger penalties, escrow arrangements, contract voidance and other niceties.
YC W15, I wonder if this will have any impact on the ecosystem of startups like Smyte getting established YC companies as early adopters.
Also I wonder how many customers are staying quiet, because "our fraud protection provider just stopped working" isn't something they want to reveal.
This is incredibly fragile ecosystem if one obscure API getting shutdown causes so much damage. This could be prevented by a simple periodic check that would determine if API is available and some fail-safe alternative to remain online. Instead this is like a house of cards that breaks with any of the cards fold. I'm not excusing the abrupt shutdown, its obviously a wrong way to end service, but being prepared for it is much better.
This is a hard call when the system is question is your anti-abuse provider. If you "fail open" when they're down, you risk allowing a flood of bad users during outages. Depending on your business, that may be much worse than having an outage yourself.
(Disclaimer: I work for a smyte competitor).
You can turn on something like manual verification of new users, an alternative security service, or just temporary halt new account registration. All of which don't result in system wide failure. Even a few abusive users is a small and temporary cost to absorb vs complete outage.
Ah, yes, just the casual manual verification of checks notes a fire hose of data.
> just temporary halt new account registration
I'm confused; how do you know these companies did not do that? In my experience even small companies will typically abstract away a third party service so that if it does fail it fails in an expected way.
What you seem to be suggesting is that all of the companies failed in expected ways. I didn't see that in any of the articles. Did you have a source? I'm curious if it hit some harder than others.
>just temporary halt new account registration.
That's an outage...
>manual verification of new users
So is that, unless you have tons of customer support reps sitting around doing nothing when shit hits the fan.
I can see cases where you want an anti-abuse system to "fail closed".
What you're suggesting is something for large companies. Most of these companies affected probably don't have time and resources to develop and pay for alternatives to every single third party service they require. And why would they? They have contracts you'd expect to be honored.
Seems unrealistic especially since the majority of times when these services go down it's likely just a short time so they probably have abstracted around them to avoid unexpected failure and just have routine, down time failures.
When I work on systems, I ask, "how does it scale and how does it fail?" Everything I work on considers handling of failure cases, especially 3rd party APIs. I'm surprised by how many people assume that the network will just work and their provider will be available.
I'm surprised that npm was affected so much since they provide service to so many people.
I think everyone is missing that this is the canary in the coalmine for how the tech giants are going to behave going forward. Hell, Twitter isn't even that big!
Every tech company big enough probably sees the same writing on the wall the rest of us do that it's time to press their advantages. That means shutting everyone else out.
We probably all have mission-critical parts of our infrastructure that aren't core competencies. We probably can't afford for them to go away tomorrow, but now it's time to expect and plan for it.
Get ready to ride the consolidation wave.
but why? If they have contracts with these external companies, and revenue, why shut it down. Even if it isn't for a huge amount of $.
The goodwill itself has to be worth a TINY bit.
Goodwill works for good companies proud of what they do.
Companies like Twitter whose core business is to shit on their users and customers don’t care about goodwill.
While this should be upsetting, and I'm sure it was to the people who were affected, I'm actually happy to see it. I'm of the mindset that this will have an overall positive effect on the state of the Web. Why? Because users and devs both need to remember that companies are not friends, and contracts are only as enforceable as the courts will allow.
Imagine if Google decided that tomorrow at 6AM PT they were pulling the plug on YouTube embedding, or GMail / GDrive integrations. The court's going to say, "They put in the contract that they can change the terms of service at any time, and they're not liable for outages or lack of service. And you signed it. Sucks for you."
Then again, I'm just anti-corp in general and this gives me better reason to avoid Twitter.
There's a difference between Google turning off a feature and Google cutting off a whole service. This is the kind of thing negotiated with contracts. Lawyers on both sides. If you build your business around another company, you protect yourself. Twitter with surely get sued for this, if for no reason other than failure to provide a service that they promised to customers in writing. Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.
> Thankfully, your flavor of cynicism generally does not apply in situations like this where many tens (or hundreds) of thousands of dollars are at stake.
This will be how it turns out for a few big players with deep enough pockets. Surely though this is hurting somebody's business that can't afford the lawsuit -- potentially to the point of putting them out of business. That's good for Twitter (and especially Twitter, as precarious as they are).
We've all joked for years about how FAANG pay their engineers so much mostly to keep other companies from being able to hire great engineers. We've gotten the big three slapped for colluding against their employees on wages. Why do you think they would engage in some anti-competitive business practices and not others?
Couldn't you file a amicus curiae, advice the court and state the company has harmed multiple people which are unable to defend due to cost, wait for the case to settle with the big guys and then sue with reference to the original case?
Not a lawyer, just interested.
Google does pull the plug on things, but generally give the customers a little time. Google gave 6 months notice for its ITA travel data.
https://techcrunch.com/2017/11/01/google-will-pull-its-qpx-e...
I'm not sure how good apple is, but when they buy software available on Windows that version usually gets killed. (See Logic music production, killed 3 months after purchasing, but presumably the old installs still worked.).
Even Halo was demoed on mac before MS bought it (The mac version never came out)
> Because users and devs both need to remember that companies are not friends, and contracts are only as enforceable as the courts will allow.
I bet their term of services mentioned that they can cease operations on no notice.
Doubtful. For most long-term corporate contracts there is no "we might cease operations at any point and leave you hanging before end of the contract".
No company lawyer I know would willingly put their signature on such a contract unless the other side is using a free tier. If someone actually signed 3 year contract with such a clause, congrats, you managed to throw a lot of money into a black hole.
It'd be good if some customers share the terms of the contract. That'll resolve a part of the discussion.
For a free product, sure. But when you're signing a multi-year contract where money is changing hands, no way in hell would any customer accept that kind of clause in the contract.
What other unpleasant things would you wish on people as a way to educate the public about the realities of contract law? Your internet service just deciding to cut you off with no notice? Water? Electricity? Ambulance or firefighters not showing up when you need them? That might help cull the hopelessly naive from the herd and have an overall positive effect on the state of the Web.
I’m aware of your sarcasm but will reply at face value:
1) If your business depends on Internet access you shouldn’t pay for a residential plan. You need an SLA.
2) Natural disasters happen. Not having emergency drinking water is very naïve.
3) If your business depends on electricity you should also get business rate with SLA and maybe have a backup generator or two. Heck, I have a UPS just to not lose the 15 minutes of work that’s not automatically backed up.
4) Yes, get a weapon (preferably a gun) and at least know enough first-aid to be able to stop a bleeding. Keep an aspirin on hand in case of a heart attack. Know your emergency exits and evacuation plan. Make sure you have the mindset to leave all your possessions behind in a burning inferno.
All of the above used to be common sense before technology made us think we are immortal and entertainment distracted us from real world.
I'm not sure that's a reply to something I said. The poster thinks a crappy thing happening to someone is good for the web because, I dunno, it's some sort of teachable moment. That's pretty silly, to put it politely. I'm not wishing you an earthquake so that you'll learn basic disaster preparedness. And if you did, fates forbid, end up in an earthquake and were short on water, I don't think I or any sane person is going to be happy about it.
I wouldn't be happy about any loss of life, you're right. However, has the Smyte situation led to anyone coming to physical harm or put in mortal danger? If it has, I'm sorry, I didn't see anything about that.
What I've seen was a select group of companies and their users who've been inconvenienced by the abrupt shutdown of a service; and whose unpleasant experience should be used to teach every user - this can happen to you. I would not be at all happy if, say, the servers for medical tracking software used in hospitals around the world went down the same way; or if suddenly air traffic control software that every airport uses goes offline.
This particular situation doesn't cause potential physical harm to a human being. It's not actually an emergency situation. That, to me, is a major difference.
What if it’s a little earthquake? It will make you think twice about building your home on a bad foundation.
I think the issue here, other than earthquakes being potentially very deadly, which makes this comparison a little absurd, is that they are in a sense unavoidable.
On the other hand you can avoid using SaaS. For example, there’s enough market demand for Github to provide on-premises secure hosting. Clearly then, if SaaS as a whole is seen as fundamentally unreliable it will create more market demand for similar on-premises, or open source, or otherwise “buy not rent” model services, which I see as the motivation for OP’s comment.
I mean, we can stretch these analogies like pieces of tasteless gum all day. I don't think being happy at the misfortune of others is a particularly constrictive attitude nor do I think Twitter buying and then apparently shutting down some service with 20 minutes notice is good for the web, teaches anyone anything or is in any way a desirable outcome. I don't understand how any reasonable person would.
I don't think I or any sane person is going to be happy about it.
You don't spend a lot of time on Facebook, do you?
If you're looking for a Smyte replacement, check out Koko. It's AI for content moderation and classification, developed at the MIT MediaLa.
https://www.koko.ai
Solid track record. Especially the part about spamming suicidal people on Reddit.
https://www.reddit.com/r/SuicideWatch/comments/5jin1q/warnin...
https://www.reddit.com/r/SuicideWatch/comments/5x60f0/update...
Another SaaS that may suddenly stop functioning tomorrow.
That's really amazing - I am having great difficulty imagining a situation in which doing this would make sense. Just for starters, even if you completely discount goodwill, ethics or even PR, I'm guessing Twitter just bought a bunch of lawsuits. This just seems nuts.
Twitter seem to have an institutional blindess towards the importance of APIs. Their execs need to be sat down and have it drummed into them that breeding this much bad “karma” (for lack of a better term) is going to come back and bite them in the ass.
> Twitter seem to have an institutional blindness towards the importance of APIs.
Which is especially weird for Twitter. Their official iOS client was a third-party API client they bought. Third-parties created @replies, #hashtags, etc. If anyone should recognize the important of third-parties using APIs, it should be Twitter... yet they've been probably the most abusive of the large APIs.
This is the same company with NIH Syndrome severe enough to in-house their own woefully-under-performing message queue. Twice. And they blamed Rails after the first go of it.
I'm not sure they know what interop means.
Yup. They started the "rails don't scale" narrative.
Something something, a good craftsman doesn't blame his tools.
https://joinmastodon.org/
Twitter will never change how they do business, replacement is the best strategy.
This is not an appropriate place for your Mastodon spam.
Seems like it is, considering its a twitter replacement and the op is kinda anti-twitter
TFA has nothing to do with Twitter the product, which is what Mastodon purports to replace.
They had contracts that they took on when they acquired the company. Twitter are now legally liable for these contracts, I won't be at all surprised if someone sues them. And frankly, they would be in the right.
What an absolute debacle. This doesn't bode well for the entire Twitter platform.
Often not.
I went through an acquisition of an unsuccessful company. My employer created a shell which is what was acquired. The old corporation wasn't acquired and existed as an independent entity to wind down contracts / deal with the acquisition hold-back, etc.
If eg smyte was out of money, this is plausible. And all of a sudden no-one was there to run the old corp.
crunchbase says they raised a $4m A in Mar 2017, but linkedin says they had 23 employees. They could burn that in 1.25 years with 23 employees and have hit a cash crunch.
I have no knowledge, just speculation.
Don't know to which jurisdiction smyte belongs, but in many countries creating figureheads/companies, dumping them, firing all employees and then buying the leftovers and rehiring people is considered the same a acquiring the company. E.g. finding ways around laws does not make them invalid and (sometimes, not always) you have to follow them anyway.
But if it is all within the states, it could be just that.
> If eg smyte was out of money, this is plausible. And all of a sudden no-one was there to run the old corp.
Sound more like a tech/knowledge acquisition to me.
4M$ for 23 employees only works if you have proper financial management though.
I'm still owed money from a start-up that went bankrupt, got bought by another company which is operating it as if nothing happend
You could probably sue them, though. Have you spoken to a lawyer?
Disclaimer: I am not a lawyer.
lol, it's just enough money to hurt, not enough to sue :/
That is not a good structure for the acquisition, especially if you are on the hook for the shell. Quite often you have all of the responsibility to deal with shit hitting the fan but no resources or personnel to actually do it.
This is a very common setup. It's called a reverse triangle merger.
https://www.investopedia.com/terms/r/rtm.asp
Yes, I see:
> Because the reverse triangular merger retains the seller entity and its business contracts, the reverse triangular merger is used more often than the triangular merger.
I was going to say "I wish that I could do that!" But then realized that bankruptcy comes pretty close.
But why would a judge not go, "No, Twitter, you clearly acquired the company. Just because you broke it into pieces, and then picked up all those pieces does not mean that you didn't acquire the company."
That is (obviously) the trick: the "acquirer" doesn't acquire the shell. And therefore is on the hook for nothing. Customers have contracts with a corporation with no assets and no employees.
I wonder if this flies with a judge.
Otherwise it would be a good way of making debts disappear.
Yeah, it could be borderline with fraud.
In some parts of the world it's actually a very common scenario. You not only get rid of debts in this way, but also employees.
That's a pretty shitty thing to do. I would call such behavior immoral even consider it's all about not honoring contracts with customers.
It doesn't work that way. You can't just transfer assets into a shell and leave the liabilities. Any lawsuits would be able to go after the assets, even if they were in a separate shell companies.
You can't transfer assets into a shell, but you can buy certain assets you want (e.g. IP) for a reasonable price even if it is much less than the liabilities, and take over the employees (which aren't "owned" by the company and thus an asset), and leave the old company to bankruptcy.
Also, it's quite plausible that Smyte had negative value before the acquisition - that it's going to be an "empty sack" not because valuable assets were transferred elsewhere (which can be reversed in courts) but because its debts already exceeded any value it had, that it was an empty sack in the first place and no debts can be paid in full in any case.
Two things:
* If the assets were bought, they'd be the property of the original company, and the money would go to the company, which would therefore have money to be sued for.
* If the directors took that money out of the company, full in the knowledge that they had legal liabilities as the result of terminating service, at least in the case the UK (and I can't speak for US law), they'd be guilty of a number of crimes, as they are responsible for acting in the best interests of the business.
What I mean by "its debts already exceeded any value it had" is that it's quite plausible that any money paid for the assets immediately goes to the company's current debtors (including some investors, if convertible debt was used instead of equity) to partially cover their existing debts (which would be a fully legal and correct action by the directors), and there's nothing left to be sued for future liabilities.
There are startup acquisitions which result in lots of money being paid to investors, and there are acqui-hires which amount to a pre-bankruptcy firesale to get part of the money back, but resulting in unpaid debt and bankruptcy anyway. I don't know about Smyte's financials, but it may well be that their situation is like the latter case.
I don't see any reason to suppose that there some big bag of money (even post-acquisition) that's somehow illegally taken out of the company. It may well also be that the money paid to the company itself for acquisition is insignificant, and the rest is in form of some "hiring bonus" to make the transfer of employees work, so not something that the debtors can claim.
Obviously this is massively speculative at this point in the conversation, largely a thought experiment, but --
Any side payment to the directors and staff outside of the acquisition - such as a "hiring bonus" as you suggest, which lead to the sale of assets at a lower price than would otherwise be the case - I'm pretty certain (again, in UK law, and I'm certainly not a lawyer), that this would be seen by the courts as a bribe, ie. money paid to perform their duty to the company improperly.
See item 1.2 from the Bribery Act 2010: https://www.legislation.gov.uk/ukpga/2010/23/crossheading/ge...
Obviously not exactly relevant in this case, but I'm guessing there's a US analogue.
it's totally kosher in the US.
In my failing startup that was acquired, all eng got a carve out. It was basically a bonus to get us to remain employees after it became obvious the company was failing and was going to be sold.
I'd imagine the legality goes like this: without that bonus, I certainly would have left. And there would have been nothing to be sold. So it was long-term better for investors.
It's only legal because there is a legitimate reason to pay it. The board members in charge have a fiduciary duty to the shareholders.
So you will see payouts and bonuses to executives to get them to stay with the sinking ship in hopes of eeking out more money before the collapse, but if you were to just pay a bonus to an executive on their way out, you would have a very hard time justifying that and could be sued for breach of your fiduciary duty.
The original company could have argued that the value received for its assets was less than its liabilities. It’s perfectly legal, but way overused in tech (where assets are overwhelmingly in the form of IP without an easily established market value).
This type of uncertainty does lead to a situation where potential customers are wary of using a startup service — I can’t build a business off your service if I don’t have confidence your company will want me as a customer in 3 years.
Ultimately I think we’re witnessing the final consolidation of the software industry. I expect the cloud/software industry of the next 30 years to start to look much like telecom over the last 30 years.
Probably worse for consumers short term, but ultimately the plumbing isn’t all that interesting and it will hopefully enable the next round of innovation.
So, in a way, you're saying smyte shut down smyte. Then Twitter swept in and picked up the pieces. Kinda?
The problem, it seems, Twitter got the timing __all wrong.__
This seems like by far the most likely case. That said, Twitter should know better than to let this happen, anyone could have predicted that the optics would place a lot of blame on them. I can't imagine Twitter couldn't support an additional 7 days after giving warning.
They had customers (hence the problem) and so they had revenue. So they should have had plenty runway.
It's not unlikely the costs of work performed for these customers wasn't enough to keep the company running. Honestly, it looks more like "let's cut down on costs, immediately", rather than a mistake on a part of an exec.
It's very easy to have revenue but no runway. If your monthly burn is $1M and monthly revenue is $800K and you have no cash in the bank, you have no runway.
They had a $4m raise a year ago and 22 employees. They must have had a very high compensation scheme.
not really.
$4m over 1.25 years (and don't forget comp costs the company 1.3) == $2.5m comp to employees per year. That is not much for 22 employees.
math: (4/1.25)/1.3
Sued by all of them, more likely
Ideally, yes. In practice, it depends on the contract. My bet on this is its provisions were pretty weak (and definitely not multi-million as it's suggested elsewhere) so Twitter's lawyers went through it and said "screw that", knowing that any lawsuit isn't going to be successful.
Just because you buy a company doesnt mean you buy the contracts. Shit was on Smyte to tell its userbase.
If you buy a company you do usually buy the contracts. Buying a company or the majority of the shares means you now own it, good and bad.
If you buy the assets of the company then usually you do not buy the contracts unless they are specified as sold.
In both cases if there are change-of-control clauses those could trigger with various penalties or rights becoming unlocked.
There are a few reasons why it's probably not entirely Smyte's fault. Given that Smyte comms dropped off over the past month, as another commenter mentioned, there might not have been anyone around to warn the users.
And if you're thinking "well they could just personally reach out to their customers if they're fired because it's the right thing to do" -- if the last-minute nature of everything is any indication, current/former employees were probably ordered not to say anything.
Depends entirely on how you buy the company and structure ownership.
Twitter is large enough to plan these details. I think it's fair to assume they are at fault until shown otherwise.
That should never be the case, and any jurisdiction that allows such is wrong.
Twitter bought a company, then found out that is isn't production-ready because it doesn't handle customer data safely, so immediately shut down the service.
I wonder if that triggers any clawbacks in the purchase price, as Smyte was running a substandard service.
This is the best theory I've heard on this so far.
So Smyte's customers had a valid contract (article mentions a 3 year contract with one customer who was cut off).
I find it hard to believe being acquired magically relieves you of your contractual obligations... could this be cause for litigation?
It depends. Was Smyte acquired, or was it merely an acquisition of their assets?
Even if the assets were acquired, I wonder why you can't sue the remaining shell and get a cut of the money that was exchanged for the assets.
I guess the real issue is that smyte's contract probably said something like (from GCP's terms of service):
> 13.1 Limitation on Indirect Liability. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NEITHER PARTY, NOR GOOGLE’S SUPPLIERS, WILL BE LIABLE UNDER THIS AGREEMENT FOR LOST REVENUES OR INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL, EXEMPLARY, OR PUNITIVE DAMAGES, EVEN IF THE PARTY KNEW OR SHOULD HAVE KNOWN THAT SUCH DAMAGES WERE POSSIBLE AND EVEN IF DIRECT DAMAGES DO NOT SATISFY A REMEDY.
> 13.2 Limitation on Amount of Liability. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, NEITHER PARTY, NOR GOOGLE’S SUPPLIERS, MAY BE HELD LIABLE UNDER THIS AGREEMENT FOR MORE THAN THE AMOUNT PAID BY CUSTOMER TO GOOGLE UNDER THIS AGREEMENT DURING THE TWELVE MONTHS PRIOR TO THE EVENT GIVING RISE TO LIABILITY.
Perhaps their TOS said that the contract could be canceled automatically if there was a change in control. That's not uncommon.
Twitter continues its insanely tone-deaf approach to dealing with third parties that interact with / rely on their services. I don’t expect anything more from them, but am surprised Smyte sold out their users just like that, as I’m sure their fate was part of the negotiations.
Given there are actual damages coming out of this to paying customers perhaps we’ll finally see them taken to task.
What the hell are npm doing that this caused a prod outage.
Either way, really poor from Smyte. No reason to immediately turn off access
If I was including an anti spam/security service as part of my pipeline I probably would want my system to stop if it went down, rather than just skipping the protection. Especially as cutting me off from said service is a possible attack option for a bad actor.
Seriously. Shutting down a security product without warning is just reckless...
I could imagine (just guessing): submit new repo -> scan with Smyte -> on success, publish repo. Maybe not handling an error properly and skipping Smyte.
This problem creeps up with external mfa/2 factor auth APIs that go down, bringing down the main login. Some choose to skip mfa if down, some don’t
It's the not handling an API failure correctly that surprises me. That seems like it'd be quite common
I mean, sometimes handling an API failure correctly is failing over. If an API call actually matters, then when the API stops responding, throw an error. Even if you're probably catching that and giving out an error message, I'd still call that a production outage.
We have a differing opinions on what outage means then. Catching the error and showing it to the user is obviously the right thing to do, but not what I'd call an outage.
I can't say I've ever met a developer who thinks that a 100% error rate to valid requests isn't an outage. Not sure why you have such a strong view of your semantics.
Sure it is, if it affects all users at once, and makes your system unusable. If I dial a phone number and get a busy signal, that's an "error" of sorts. If every Verizon user gets that busy tone on every call, that's an outage.
This is a spam filter, so failing closed seems reasonable. (Failing open is also reasonable too.)
There are lots of potential security issues with npm (compromised accounts, spam packages, etc). I think failing closed is the right thing to do for them. It's a temporary annoyance and loss of productivity for a day - a good time to put your feet up and relax. Failing open could lead to a costly security breach that affects many people, and could lead to npm's demise.
npm gets a lot of flak on HN, but this seems fine (or rather, it should have been fine) given that they and others apparently had multi-year contracts with Smyte. What kind of situation would you be in if Github went away, for comparison?
This seems like a clear breach of contract. I can’t imagine how anybody thought it was a good idea. Maybe it was a mistake, although that wouldn’t be a whole lot better.
After hearing so many stories like this for so many years, any time a company acquires another, it usually seems advisable to jump ship from the acquired to other vendors before things start going south.
PR will come up with all sorts of reasons said acquisition will benefit both company's customers but I've yet to see it.
Oooh.. I'm glad they screwed over well-funded companies with multi-year contracts. Now at least they're going to get the shit sued out of them unlike when they go after regular users.
Sued for what? Have you seen the contracts these companies signed w/ Smyte?
If these companies signed an agreement with a vendor that allowed them to cut off service with 15 minutes notice, they deserved everything they got.
it seems curious that someone would sign (and pay for) a service provider contract that's largely pointless, but maybe there's a non-obvious loophole.
How's the legal situation in America regarding breach of contract?
At least in Austria we have a law that exactly tries to prevent such things (§38 et seq. UGB): when you buy a company you enter into all of their existing (non highly personal) contracts and obligations, unless you specifically don't want that, in which case you must tell the other side and give them some time.
I am not lawyer. I don't think there's a valid loop hole that would pass a judge in a court of law. Either the contract specifies early termination damages or it was neglected.
lmao this is hilarious. Never seen something like this happen with such a large company. Let's hope the codebases out there had a simple integration with Smyte and the change to that "smyte" filter was fast. Crikey!
Hm, one of the tweets mentions a "3 year contract". A contract means two parties agree to something, with penalties if they don't do what they agree to, right? Couldn't they sue Smyte/Twitter or something?
Usually there are terms on acquisitions deeply nested in contracts.
Usually those terms involve "keeping the lights on". I imagine Twitter will just pay people off, which eases a lot of pain.
That depends on how the contracts are written and whether Smyte continue as a company on its own or just as a software product of Twitter.
Ah I see the plan here: Twitter knows their platform is a cesspool of abuse and harassment, but instead of fixing it they're pulling the rest of the internet down to their level so they stand out less!
Genius. /s
That will make ALL companies think twice about trusting a startup when choosing a SaaS provider. VC got its way but at the same time decreased any future startup success potential by a huge measure.
As multi-stepped and prolonged as it is to sell to large enterprises, I don’t look forward to tomorrow when yet another question to be answered before any large org signs a contract with a small startup like us is: are you going to sell and shutdown?
This has happened so many times that it can really affect how small companies are preceived as too risky too deal with by larger customers. sigh
Wow, twitter really are just "the worst" these days.
Say what you like about Facebook but it does at least seem as though somebody is calling the shots in there.
Companies can do this, there's no regulation against it and there probably shouldn't be any. But there are going to be consequences and they're probably not good for the industry at large. Moves like this fundamentally change how much trust consumers have in SaaS apps. The risk factor of using new apps fundamentally changes, if ever so slightly. Unfortunately, this arguably works in favour of the companies that do this. By reducing trust in startups, there's fewer users who will take the leap on new actors.
Twitter and Google wins by having fewer competitors, but the customer loses doubly so. If Facebook acquires every fifth social network, followed by immediate shutdown, users will eventually stop using them, hence less competition as a whole to FB.
Wow, who would do this, especially since all of the brands mentioned are probably also Twitter customers.
They're twitter users, unlikely twitter customers.
I assure you they are all paying for promoted tweets, ads, or using expensive social media tracking tools to monitor usage, respond to requests etc. in one way or another.
Twitter is a pretty big part of any modern online company’s strategy, whether they like it or not.
One thing that doesn't make sense is surely a service like this greatly benefits from the network effects of being able to see activity across multiple sites, so if Twitter wanted to continue using the technology, wouldn't they have been better off leaving it running?
Twitter may have sufficient scale that the additional data aren't meaningful.
Friend worked on a large, multi-institutional risk-modelling project ... until the largest institution decided it had enough data on its own, and didn't need competitors who were competitive in this regard, and pulled the plug.
Law of large numbers (statisttics) is powerful. Even for complex (multvariate) models.
It sounds as if there may well be other considerations at play, but I'm answering your question specifically.
I'm not convinced of your premise, spam is adversarial & spam detection is just pattern recognition. Twitter likely doesn't want to allow people to be able to test against their spam shields by spamming other people
As I understood it they were using machine learning models to identify attacks. Abuse and fraud strategies are constantly evolving, so the benefit of having many clients is that once you see a new attack strategy at one, you can instantly protect all your clients from that threat.
In your own example
"Twitter likely doesn't want to allow people to be able to test against their spam shields by spamming other people"
Yes, the attacker may find a new pattern that can bypass the shield, but as soon as that pattern is added to the machine learning recognition, they are then protected against it on their own system.
"spam detection is just pattern recognition"
The more patterns you can analyse, the better your recognition?
I hope this ends up litigated. The idiots at Twitter need to understand what accountability is about.
The call to wind down business with existing clients is a worrying trend. Why would I avail services of a startup if there is a risk that upon acquisition, the startup's new parent company won't service us anymore. It's a risk.
I'd really like names to be named here, so everyone knows who never to do business with in the future.
Twitter
Basically, don't do business with Twitter.
Don't give them your data, don't sell your ads on twitter, don't develop for Twitter.
There are alternatives.
in twitter's case the closest alternative is standing in a crowded street, barking disjointed sentences at passers by in the hope somebody acknowledges you.
Do you actually need to bark at passers-by, or do you think you need just because every other dog is doing it?
Replacing Twitter is simple. Have a proper support channel (Intercom, etc) so your customers don’t need Twitter to get support from you, and use your existing marketing tools (email, push notifications, etc) to keep in touch with them. Done.
Well, there is mastodon/pleroma/fediverse, in which case you are standing in a less crowded street in your neighboor casually barking disjointed sentences and vaguely familiar passers.
I'm pretty sure that "to wind down" is not a proper euphemism for "to shut down".
"Winding down" literally means that before shutting down, there is an incremental process with the explicit purpose to smoothly move into the transition. There wasn't anything like that, not even a little.
What's especially rich is that Engineering VP Mike Montano nonsensically repeats this phrase after the fact: "we made the difficult decision to wind things down right away". That's plain bullshitting.
He should have said, "we made the difficult decision to shut things down right away". But that would have sounded super irresponsible! But it was super irresponsible, and describing it differently is just dishonest.
And that wasn't accidental, this entire situation is exactly about the difference between "winding down" and "shutting down".
Why do we allow people to communicate in this way? (publicly and/or from a position of leadership). Twisting words in plain sight. There is no good motivation behind this, it represents messed up priorities between appearance/saving-face and responsibility.
I don't know anything else about this Mike Montano. Maybe he's a very good manager when not talking like this. But leadership comes with responsibility and when you twist words like that, it's only because you're trying to wriggle out from underneath the responsibility.
Lessons:
1. Trust small, third-party, closed-source SaaS vendors at your own peril.
2. Do not spend/waste your time integrating with random third party services of questionable/unknown sustainability.
3. Have a test environment where you can delete routes to external services and assess the failure modes of your application.
Does the whole page hang? Do UI elements disappear ot overlap others? Do JS error messages surface in the UI? Can the user continue with core functionality or at least receive an explanatory message?
It makes me wonder. Does anyone actually read the contracts they sign?
I usually hate it when people blame the victims for being so stupid whenever something bad happens. But some of the victims in this case are corporations with lawayers.
So either this really isn't a mission critical service and they just took a calculated risk, or these lawyers haven't done the job they were hired to do, or Twitter is in breach of contract.
Is this really that out if character for Twitter as an organisation though? They have a history of being rather hostile toward developers and their own APIs. Which is ironic considering the nature of the platform.
I would say we should use this as a sign to migrate away but we've had enough warnings already. I guess Twitter is too big to fail?
and this is yet another example why we need protocols and not APIs.
I treat the two as synonyms. What do they mean?
You could say that an API implements a protocol, but a protocol is much more general. For instance IRC is a well known and widely implemented protocol. Facebook messenger implements a protocol for client server commination, but there are few or none implementations if either side and the main implementation can change the protocol arbitrarily and without notice.
Right, but then wouldn't every API that has documentation also necessarily be a protocol (that might only be implemented by one provider?) If the idea is that you could switch to a different provider if you were using a protocol instead of an API, that only helps you if there is more than one provider of a protocol. In this case, someone else could take the public API docs and implement their own service that responds to the same API.
I would imagine the relationship similar to programming languages and standards; If there's no official standard, then usually for anyone trying to re-implement the language, they're "doing whatever the mainline version does". ie anyone trying to reimplement python does whatever cpython does, bugs and all, to claim "compatability".
But if a standard exists, that is, we've split the API from the primary implementation of the API, then we as a re-implementer no longer need to replicate bugs and all; we just need to do as the standard says, and anyone relying on cpython bugs is, well, at their own fault. Compatibility claims are no longer dependent on implementation-specific details.
And of course, it doesn't matter until a second implementation begins to appear; there's no reason for cpython to follow (or have) the standard unless it actually wishes to be compatible with jpython. At the same time, it makes a lot less sense for a second implementation to appear as well, as they have to put a lot of extra time and effort into matching cpython's implementation details (as far as they matter; a difficult determination itself), and this also makes it more difficult to produce an alternatively-optimized implementation (focusing on say, memory-usage instead of speed or whatever). So it's also a chicken and an egg problem.
So ideally everything would be an official standard, and we'd put little to no effort into simply matching whatever the popular implementation is at the moment, and we could criticize the mainline implementation for failing to adhere to the standard.
In this particular scenario, to replace smyte everywhere, transparently, you'd both have to adhere to the API and whatever details incidentally exposed by the API, because you have no protocol to actually reference (the protocol is whatever smyte was doing). But thats also particularly difficult here, since you can't even test it against an implementation of smyte anymore..
Perhaps "protocol" is meant as a standard specification (like ActivityPub) and "API" as a specific implementation for a company/service?
The term API has evolved to mean "a RESTful web service", almost always run by a company or organization. This surprised me on multiple occasions.
Kind of like how "cookie" is not web specific but unless mentioned in an explicitly nonweb context, it means browser cookies.
Given the speed there might have been a legal reason.
Perhaps they were in violation of either the GDPR or their customer data agreements.
This is also a responsibly of the founders of Smyte. They cannot just leave the customers hanging.
Unless it turns out that Twitter blatantly lied to Smyte's founders regarding how they would shut down the service, I personally will never use any service or software that these founders create in the future. This is absolutely unbelievable, even for Twitter.
[I've removed the founders' names, even though I strongly disagree with the characterization that I was somehow acting out of rage. My intent was rather consumer advocacy. I've had a service cut off in a similar way, and I view such behavior as being unethical. Fine if you disagree, but please don't characterize my emotional state that led to this post (which itself is in violation of HN's admonition to "Assume good faith")]
Bringing in people's names in order to attack them is a breach of this site's civility rule. You needn't stoop to that in order to discuss issues of substance in this story, so please don't.
All: please keep the internet rage reflex off HN. Same for the online shaming culture.
https://news.ycombinator.com/newsguidelines.html
(Edit: I didn't mean that you were feeling rage, but that by posting this way you were stirring it up. In any case, thanks for removing the names.)
So all the constant anti-Tesla articles blaming Musk personally on here are fine, but these guys are somehow above being named despite their names being public record? Why is this different?
You think an article criticizing a company and its executives is the same thing as inciting an online rage mob (along with a posse for some unnamed Twitter evildoer) on a message board? Because they're pretty different.
Wasn't the original commentor criticizing, exactly, an (acquired) company and its executives?
No.
You are right, but Tesla itself pushes Musk as the face of the company.
Personal attacks on anyone aren't fine on HN, but when there's an entire industry of articles about them it gets a lot harder to enforce the rule.
I haven't seen a single anti-Musk personal attack deleted or addressed, yet I saw this. A lot harder != impossible.
"Bringing in people's names in order to attack them is a breach of this site's civility rule"
I think there is an issue of personal integrity here, even legally.
When I worked for a large F50, our CEO's name had to be on every email we sent out from our German staff - though I'm not sure of the specific German regulation, 'his name is on it' because it's a matter of integrity to the organization.
The founders are generally 'Officers of the Company' - and this implies both a legal/fiduciary and moral obligation in America and most jurisdictions of relevance.
I don't believe that the 'limited personal liability' concept abnegates the responsibilities of the Officers of the Company in this specific regard.
I also don't believe this is a matter of arbitrary doxing if the Officers of the Company are making decisions which are considered foul play in terms of reasonable contractual obligations.
Unless there is an issue of national security or likewise, there's simply no excuse for ending service without reasonable notice.
This is not a political, or socially nuanced discussion such as those involving harassment etc. - this is a matter of commercial civility (and legality).
In American law: "In limited circumstances, such as the sale of the small business to a new owner, the business judgment rule does not apply, and it becomes the burden of CEOs and company directors to demonstrate that their actions were in the company's best interests." [1]
(FYI - the 'business judgement rule' protects Officers from liability.)
I guess I understand we don't want to be doxing folks here ... but I do think it's reasonable that individuals should be held publicly accountable for their commercial actions - for example, someone does an ICO and then - legally - walks away with the cash ... we should rightfully be wary of investing in their next ICO or company for that matter.
The inability for executives to be held accountable for actions of the company is a core problem in present day capitalism, in my opinion.
[1] http://smallbusiness.chron.com/legal-relationship-between-sh...
Internal to a single company is a totally different context than a large internet forum.
Loathe to be technical, but this is HN ... to the best of my 'definitely not a lawyer' knowledge, you are correct that Officers responsibility to shareholders (i.e. 'internal') is greater than it is to their contractual obligations (i.e. 'external') ... but, there definitely is still a legal obligation to contracted customers, of which there are thousands. Ergo - this could very well sit within boundaries for their personal obligations.
I know they're an YC company, so I don't want to ruffle anyone here, but I do think a class action lawsuit against Twitter is warranted, even if it has a shaky legal foundation.
I believe that Twitter's actions - and the response - will be duly noted among M&A camps and the last thing we want is this to become standard practice. This is just 'bad acting' and it will come back to bite everyone here in startup-land.
I risk my erstwhile happy HN account by getting into a fuss with pvg, but he indicated that we want to 'wait to find out what's happening' - I would argue that in light of Twitter's 'total and immediate blackout' the onus is upon them to provide forthright information, and that absent that, we can assume 'bad acting' given a total blackout without any information.
Maybe we can use this as a lesson and the powers-that-be can provide some leadership, and possibly push towards a 'best commercial practice' like '120 days minimum notice' or something like that as a standard.
This is exactly the community to do it in, as I frankly don't see where else anyone has any broad legitimate authority among early stage startups such as it exists here.
Hey dang, that response and edit was pretty civil, since the names are removed could the flagged state on OP's comment be removed as well (edit: please)?
Thank you. As far as I can recall, it's my only flagged comment in my ~8 years on HN.
Back to zero.
Sure - that's a good suggestion. Thanks for looking out for a fellow user.
Strongly disagree with this decision. These are public names and not a personal attack. People made these decisions and were responsible for them.
Bringing in people's names in order to attack them is a breach of this site's civility rule.
Yes, we are aware that Smyte is YC W15. Thanks for the reminder that you protect your own from base accountability at high cost to others.
That had nothing to do with it. I had no idea that Smyte was a YC company (or had forgotten, not sure which). But thanks for letting me know.
Since it sounds like you follow these things closely, I'm surprised you'd be so cynical. If there's literally a single case where someone did that where we haven't chided them, I'd like to know about it. In fact if there are any personal attacks of any kind on HN that we haven't moderated, I'd like to know. The likeliest explanation is that we just didn't see it. In the present case a user (not connected to YC as far as I know) emailed us to complain about the comment, I looked at it, and replied—same as it ever was.
I strongly disagree with this. The founders of a company are public figures, and their actions matter. They should be named, after all they are the ones who sold their company and then immediately pulled the plug.
I can vouch for Pete Hunt based on what he's done for React. There's gotta be more to the story.
But you'll use Twitter? Kinda a weird dynamic there don't you think?
Oh, I've long since learned not to depend on Twitter for anything critical or business related. The thing is about Smyte is that it was a b2b SaaS with paying customers, not a consumer social media company.
People justifiably have certain expectations when they pay for a serious b2b SaaS that they don't have for a free social media service.
I like to use Twitter from a distance. Like a street brawl. It's fascinating to watch, but I sure as hell wouldn't want to get involved. Too messy.
I feel the same way.
I'm also a bit irked with having to prune all the time. I follow someone or something that I expect to be about something and really it's just a lot of noise about other things...
Let's not be starting witch hunts on HN, especially given the sparse info available so far. You should edit this stuff away.
The sparse info is exactly why people are looking for answers. This was a pretty unethical business move, regardless of whether these people had themselves covered in contract language.
This is not looking for any kind of meaningful answers. It's basically a call for organized harassment, facts unknown. You are surely aware how that sort of thing has worked out on other forums.
[I removed from here an argument that this was public information, without mentioning the names again.]
You don't have to post their names or attack them publicly in order not to do business with them.
Please stop now.
I reserve judgment as to whether it was Twitter's fault or Smyte's fault, but at the very least, it was Smyte's fault for selling to Twitter without getting any guarantees for their existing customers. At the worst, they shut down the service like this themselves.
Again, if it turns out that Twitter lied to them about how the service would be wound down, I'm sure that will come out in due time. If I were one of the founders (and I know at least one of them has been often on HN), then I'd be on this thread making some explanations right now.
Finally, the name of the founders isn't some kind of secret. This is public information. The founders have been featured in many media pieces. They sold their company to a public company.
Seconded.
Based on the email from the VP, my wild guess is they figured out that Twitter operating Smyte would make Twitter itself not GPDR compliant (or some other big regulation), and they didn't figure it out until after the deal was done.
Shocking in many ways. But not in others.
I predict some back-pedalling later today and access switched back on and properly 'wound down'.
If not, customers with contracts should have every right to angrily sue.
The reason companies shouldn't use each other's proprietary code for basic operations: it's insecure, there's little-to-none of your interests being protected.
Wow. That sort of thing can be the effect of little to now negotiating leverage by the acquired company. They only raised 6.3M according to Crunchbase.
Note to self: add "to smyte" to vocabulary
Was there something so dangerous for Twitter in Smyte's business model that required such a brutal shutdown?
The fact they had an actual product they were proud of? Clearly that’s considered unacceptable by Twitter’s standards.
Playing devil's advocate. Here's a possible scenario:
1. Smyte had a security flaw that could potentially (or already did) affect twitter customers.
2. Leaving Smyte service on could do alot of damage, so turning it off would be the sensible thing to do.
3. Telling the world about this security flaw would create a really bad backlash.
I'm always surprised that Twitter has managed to stay in business so long. Anaemic growth, massive spam and bot accounts, an advertising platform that's hostile to most small advertisers, a user-base that makes YouTube's comments look downright gentle, constant hostility to third party developers...and yet it continues to chug along.
Buzzfeed posted an article called "How Twitter Made The Tech World's Most Unlikely Comeback" just today:
https://www.buzzfeed.com/alexkantrowitz/how-twitter-made-the...
even as those eulogies were being published, things started changing. Twitter began beating earnings expectations. Star ex-employees trickled back in, finding a new, more positive internal culture than the toxic one they’d left. Advertisers came back too, as did users. The company finally began addressing its trolling problem. And its stock, once unappealing to analysts like Nathanson at $14, is now trading above $46.
It’s still somewhat taboo to say it, but it’s no longer possible to deny it: Twitter is making an unexpected, somewhat miraculous comeback. It is the first major consumer social company to lose users and start growing again in a meaningful way.
I really think they need to thank President Trump for that. He saved Twitter.
They did, in December, when they exempted "government entities" from their abuse policies. http://www.businessinsider.com/donald-trump-has-immunity-in-...
It's not illegal/abuse/threats of violence when the President does it.
Fun fact: I just did a search on a news database. Roughly 15% of stories that mention Twitter also mention Trump, and roughly half of those seem to be about Trump's tweets.
I don't know if Trump is really causing Twitter usage to increase, but it has certainly made people who don't use Twitter more aware of it.
If you compare the awareness/user ratio on the various social media platforms, I bet Twitter's is pretty high, which means they have potential for significant growth if they can improve the UX and the "how do I use Twitter? Why would anyone use it?" problem.
There is an entire cottage industry of Twitter pundits now who just focus on talking about Trump.
I remember following Scott Adams, whose work I enjoyed. Before he started talking about Trump, he had some 40k followers.
Since Trump, his follower count has ballooned to 250k and he almost exclusively talks about Trump now
It's enormously popular! The world's biggest social network focused on public interaction. They've got everyone from a hundred million normies following celebs to countless niche interest circles.
The real surprise is that they've failed to effectively monetize their huge, huge audience.
I agree, and one specific thing that is very annoying is that tech podcasts seem to rely on Twitter as their sole means of communication with the show. Some shows and networks don’t have email addresses, just twitter handles, and others actively tell you not to email their address as you probably will just get deleted. I’m not on twitter and have no desire to be but I would like to contact the show once in a blue moon.
Its userbase (myself included) continue to use the site in spite of Twitter's best efforts to make us not want to.
Related: "How is Twitter supposed to know what we want?" https://theoutline.com/post/4936/what-do-twitter-users-want
> I'm always surprised that Twitter has managed to stay in business so long
I'm not. Millions of people are visiting Twitter daily to check out if Trump has declared trade or real war against yet another country. Or whatever the Orange in Chief has brain-farted over night.
In addition Twitter is a fine tool for starting shitstorms and one of the few safe havens left for adult material creators (= freelance sex workers) to post their content (but that may also change one day). Oh, and media often turn to Twitter for soundbites/other stuff to quote, which also creates demand.
>I'm not. Millions of people are visiting Twitter daily to check out if Trump has declared trade or real war against yet another country.
This is true and not just for Trump, most politicians are finding out they can bypass the media and go straight to enough people to control the message, especially if the message is bombastic or controversial.
Why rely on the faucet when you can drink from the fountain?
This is why I don‘t really like SaaS for important parts of infrastructure. It‘s not under your control.
Off-topic, but twitter related: has the site fallen off search engines?
Yesterday I wanted to look at some joke account (by correctly spelled name), Google yields nothing from the domain, Twitter wants me to sign up to search there ...
(Not a habitual user, or even reader, as you may guess.)
It’s fairly common in so-called acquisitions of startups that the the acquirer never buys the company stock. This leaves annoying liabilities like existing contracts in the selling company which will be shut down. The buyer might purchase IP so that the company to be shut down can pay out their investors and old customers.
However that situation would usually never be described as an acquisition by the buyer. (The seller might put out an announcement that says “We’re proud to be joining FooGleZon, it’s been an amazing journey!”)
In Smyte’s case, Twitter PR does use the word “acquire”: https://blog.twitter.com/official/en_us/topics/company/2018/...
That sounds like something else than an “amazing journey” non-acquisition. But IANAL, just speculating out loud.
I don’t know how to run a business. But, I know how not to run a business. That’s one way.
Funnily enough, they had a HN "Who's Hiring" post less than 3 months ago.
https://news.ycombinator.com/item?id=16739153
Wow, even Smyte's website was gutted. Bold move, to say the least.
> We had a 3 years contract with them and they just disappeared overnight. No communication at all, they just turned their servers off
Hmm, is the new owner not bound by that contract?
Tell me again how the silicon valley system works and works well at what it does.
Create a product, gather funding.
Attract customers.
Get acquired, founders make bank.
New owners trash the product, stranding customers.
Tell me again how this is good system not for only making money but actually providing value to customers.
Going off on a tangent here, but it's interesting how the first thing on that list is "create product".
I would argue that most new companies (at least started by and solely run by technical founders) fail exactly because of that.
Product-oriented people tend to think that the success of a business is due to its product(s), when the reality is businesses succeed due to their ability to sell their products. Hence, the most important thing is actually being able to properly find and target an audience, which should be the first step of starting a business.
The second step of starting a business should be to test marketing channels to make sure the selected audience can be effectively/efficiently reached. Only after this has been properly done should a product be ideated, based on the knowledge acquired about the target audience and the marketing channel(s) that work.
Just my 2 cents to the HN community after having learned this lesson the hard way, failing multiple times and starting several businesses.
Yes, yes, and yes.
What's more, you could be product focused and still produce bad products. I just left a startup that seemed to have really bad decision making in areas like "what feature should we build first/second/third", and major areas needing design and careful thought were instead treated as an afterthought. But the tech was solid and lots of work was getting done, so... good enough?
Heads up, Y Combinator! Heads up, executives! Enough of this foolishness.
I'm a tech worker who doesn't live and work in the Sili Valley reality distortion field (SVRDF). From outside the SVRDF it's hard for most people to tell one company from another. They all look like SV. That means they all look a little untrustworthy and a little dangerous.
Now, sure, this Smyte had an insider's brand, sort of like "key gaffer" in the movie industry. But their shenanigans, and Twitter's, just made it harder for all the rest of us to sell to prospects outside the SVRDF.
Y Combinator, will you consider doing some time on basic business ethics with your incubator residents? (I'm talking about stuff like "don't screw your customers, suppliers, or shareholders: keep your promises and honor your contracts.")
cc sama, pg
People are so shocked by this precisely because it's an extremely abnormal way to do business, which demolishes your chip-on-the-shoulder premise. You're using a rare occurrence (thus the shock), to extrapolate that it applies exactly the same to the majority of Silicon Valley.
Acquisitions are pretty much always bad for the acquired company and its employees.
How so? Aquisitions happen all the time, many of them improve the product / processes, embrace the VP!
this is typical Twitter. no surprise.
Remember when the Trending section wasn't advertisements?
Twitter is starting to seem awfully authoritarian...
I'm fully prepared for whatever negativity comes with this comment because it's being asked from a place of genuine curiosity-but yes there some snark baked in:
Does the rapacious acquisition game of Silicon Valley and seemingly increased rate of consolidation of startups to established giants feel weirdly like he days of Ma Bell to anyone else?
It seems like more and more There's tales of these acqui-hires with promises that the aquirer will present some kind of "polished" alternative but under the larger brand and I've found quite frequently the promised offer-if ever shipped at all-is far less superior and far more self serving to the brand than what was previously offer by the acquired when they had more on the line by operating as a startup.
Or am I just a cynic?
I'll one-up you with cynicysm: these days, getting acquihired - AKA having "an exit" - is the design goal of startups from day one. The whole point is baiting enough users to use your product that you can look good to some large company, get acquired, close up shop and enjoy your millions. For founders, it's "compress your work years and get rich sooner by working more now". For users, it's pure bait-and-switch though.
This is such a constant thing that right now, I'm very reluctant to use startup-made SaaS for anything that isn't throwaway. The expected lifetime of a service is just extremely low, and the cost of integration (lock-in due to platform idiosyncrasies, and your data being taken hostage) is too high.
Acquisions have been the principal way big companies have always done "innovation". Nothing new here.
Also the principal way they remove competition.
You may be right. It's not a topic I follow with much earnest compared to other items to be fair. The inquiry was just a hip shot response.
Almost. Back in the Ma Bell days, connecting equipment to the Bell telephone lines without authorization was an actual crime. You could be be prosecuted for it.
You couldn't develop telephony product at all unless you were Bell.
> seemingly increased rate of consolidation of startups to established giants
Do you have any data to support increased acquisition rate?
Not much by way of a published study if that's what you're after, just a supposition if I'm being completely honestly.
Welcome to being wrong on this if some good soul does have the data on this, though. It's why I asked, maybe someone has research that would prove me wrong and I'll gladly take being corrected on the matter.
Yeah, I feel that way too. Like, every company whose products / services I liked that got acquired never released another interesting product or service. Dropcam is one that I can think of, nest is another which is related. Microsoft buying Wunderlist also seems like it’s only leading to Wunderlist getting shut down. CoreOS sold to RedHat and while I have my doubts, this sounds good: https://groups.google.com/forum/m/#!topic/coreos-user/GR4YlF... (but don’t these kinds of messages always paint a rosy picture of the future?). We’ll see what happens to CoreOS.
EDIT: just remembered WhatsApp. This happened recently: https://www.cnet.com/news/whatsapp-founders-may-have-left-1-... and it doesn’t seem like them selling did anything positive for their product or customers, only their bank accounts. Seems like they couldn’t even wait for FB to open up the golden handcuffs.
Anyone have examples where the acquired company goes on to be really amazing, on a trajectory that they wouldn’t have had access to without the big injection of funding from the acquiring company? I can’t name any, but that doesn’t mean they’re not out there.
> Anyone have examples where the acquired company goes on to be really amazing,
Instagram is a fairly high-profile example.
DOS did quite well after Microsoft acquired it.
I wonder if twitter uses npm...
They utilize SSR React, so, yes, yes they do.
If the umpteenth demonstration was needed that you can't rely on any 3rd-party internet service, here it is.
npm? that sounds way worse than the others
We don't know what happened to the other ones though.
What on earth do they mean by "safety"?
if you aren't the customer then you're the product... no wait, they are the customers.
Site's that block users for not allowing cookies and tracking are essentially paywalled. Communities like HN should do their part to penalise sites that pull these kinds of tricks. If not us, then who?
So, anyone got a non-paywalled version of this?
In the next installment of "How To Make A Successful Exit While Not Give A Flying Füçk About Your Existing Customer Base Or Industry Rep. That Got You There 101"
Is anyone else as tired of this schism of material gain over all other considerations? If the tech industry wants to end up with the same ill repute as Wall Street this is certainly the right way to go about it.
This is it? I've often considered the abuse of 1099 workers to be the thing that one day catches up with tech as a precursor to its own "Enron" moment. Maybe it'll be both.
How visible is that issue though? How likely to be digested by the masses?
"Tech company takes money and screws over customers" is a headline that writes itself.
Abuse of 1099 independent contractors is pretty much endemic to every large industry and profession in the US, though. The tech sector isn't going to get called out on it before sales, etc. It's just "the way it is", because the IC regs are almost entirely in the employer's favor financially. And since the capitalists have the money, the rest of us are unable to influence the government with the same net effect.
The screwed over themselves as soon as the decided to rely on third party SaaS solutions for core infrastructure.
How many more examples do we need for this to sink in?
You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made.
Back when it was more traditional to buy a software licences and self-host rather than pay a rolling subscription to an XaaS, a service going dark just meant that you were shut out of updates but not the software itself. Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
> Sure you'd still eventually need to migrate away but at least you could then do it on your own terms.
I think people often underestimate the value of being able to manage their own destiny. They key point here is "manage" though: you have to actually do it - it's an active process.
If, on the other hand, you simply stick your head in the sand and ignore the situation it will eventually get to the point where it becomes extremely difficult and expensive to deal with due to a variety of factors such as no clear migration path, loss of institutional knowledge, and so on. I name no names, but worked for one UK company in particular with form here.
I'm not making an argument against SaaS (we use a few of them), but it's important to understand the trade-offs you're making in terms of ceding control.
Sure, but a service like Smyte requires significant expertise to make work well and run. Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.
Further, some of the biggest value they will provide is being able to see transactions between vendors. So eg calculating an IP address reputation score by seeing bursts of fraudulent purchases. That is impossible to do in house only.
> Pretending you can snap your fingers and implement a Sift Science or Smyte is ludicrous.
But that's not what either I or the GP were talking about: we were talking about hosting a product that you'd purchased from another vendor on your own infrastructure.
Choosing SaaS products that have self hosted licenses or open source alternatives is part of managing your own destiny and making the right trade off. If the former exists, this discussion is moot because there will almost certainly be a commercially supported migration path (or someone didn't do their research and chose the wrong vendor). If it's the latter you're back to having to invest capital into creating an in-house system while learning how to use, scale, and troubleshoot open source software. If neither... god help you.
But this sort of systems is particularly suited to being hosted by a third party as you benefit from a network effect via the other customers.
I'm not going to argue that these sort of problems aren't ideally suited for SaaS however it is still worth remembering that systems like this have existed before SaaS took off. eg you would host the API yourself but cron a db update every n hours or days. So if the service went offline you still had a local database and API.
The obvious downside to that is you don't have real time updates (as well as the usual other benefits that SaaS brings), which is why SaaS suites this kind of business well. But it can be done and was sometimes done this way "back in the old days".
Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it, though. So situations like this are suited to having a centralized owner of the data who can perform rate limiting, restrict to its paying customers, etc.
Distributing the database to customers is still workable despite this drawback - antivirus systems are a common example.
> Having the db freely available makes it a lot easier for a bad actor to figure out how to avoid getting in it
I don't agree with that. Having the database only allows bad actors to see if they've been added to the database - not the business logic of how they ended up in that database.
You could argue that knowing you've been added to the database is still an advantage, which is true. But those same bad actors could still test the system with SaaS solutions too. eg when one acquires stolen bank cards it is common to use those details to make small payments for things like food delivery to test if those card have been blocked or not.
Having a fast, cheap and detailed feedback loop is useful for testing for the bad guys, like in any programming process. For example, in the credit card case, you can check thousands of cards without needing to find a merchant who will take your thousands of small, frequently-declined transactions. Or you can check the details for cards you haven't fully compromised (maybe the guy selling you stolen bank cards only gives you a portion of the details until you've paid him).
Oh absolutely. Like many things security related, it's not always about stopping them but rather slowing them down enough so that it doesn't become cost effective.
That's preferable for the customer but it's obvious why a vendor wouldn't be keen.
> That's preferable for the customer
I wouldn't say it's preferable for the customer. It's just one of many options.
> but it's obvious why a vendor wouldn't be keen
Yes, SaaS obviously has benefits for vendors too but what's most preferable to them is making money. If every customer refused to do business with SaaS then vendors wouldn't prefer providing SaaS solutions (yes, I know I said "solutions solutions" :P). The reason SaaS is so prolific is because it offers advantages to customers as well.
>>yes, I know I said "solutions solutions" :P
When has SaaS stood for anything besides Software as a Service? Because software as a solution could describe almost any business model?
Yeah you're right. And weirdly I did know that so I haven't a clue why I wrote that comment. I can only assume it was a multitasking fail. Thanks for the correction though :)
It's not really a good argument. It's an ok (if stultifyingly predictable) argument when it's made about free services. When you enter long-term business relationships, pay for services, sign contracts, it's quite reasonable to expect to receive what you paid for. Not a lot of useful stuff would get done or made at all if we all turned into preppers.
You claim it's not a good argument yet stuff like this does actually happen. Sometimes it's genuinely because a business closes up shop, sometimes it is because it is bought out. Sometimes the business is still running but they just decide to deprecate a service or API without grandfathering existing customers / offering sufficient time to update customers code or even switch to a new provider.
Granted there are more cases of when SaaS et al works than when it does not. And I do agree with you that it's no ok to shaft paid customers in this way. But that doesn't mean that it doesn't still happen. This is why I've have to consider the self-hosted verses XaaS debate myself writing Business Continuity / Disaster Recovery / et al plans.
Of course it happens. Bad things and unexpected things happen. It is better to be prepared for them than not, etc. But saying completely predictable and trite things whenever some unusual bad thing happens is not 'a good argument'. It's barely even an argument.
The argument of self-hosting vs SaaS is still a valid one regardless of how predictable you might consider it. However the argument of arguing about arguments is definitely trite. But perhaps this is a good place to end things before it gets too meta ;)
Go back to reddit
"You're getting downvoted but if people look past your inflammatory tone there is a good argument to be made."
I really dislike this idea that it's our responsibility to ignore inflammatory tones and trolling, instead of it being the responsibility of the poster to communicate clearer.
You're over analysing what I posted. I wasn't suggesting it's people's "responsibility" to look past trolling. I just felt there was a valid point buried in there that was worth reiterating in a non-inflammatory tone.
That's not a very inflammatory language in the previous post
I assume that in the real world you grow your own food, generate your own electricity, dig up your own wells for water, and knit your own clothes? Companies have to rely on other companies for critical services, just like we do in the real world.
Multiple sourcing is the key. All of the above products can be bought from multiple places, this API could be "bought" from a single vendor.
Any usage kf SaaS would require a solid contract in place and a plan B (and C, and D, etc) to reach even a fraction the reliability of e.g electricity or food.
Managing risk comes into play here. Would you spend time integrating 2 services, negotiating 2 contacts, supporting 2 providers who may change features and approaches over time, splitting a fraction of traffic to make sure both still work, limiting yourself to common feature set... Or would that cost you more to implement them a few days of (potential) complete downtime.
Choosing the potential downtime is not a failure, if you considered your options.
You have to admit that the Smyte's customer list that's floated around, here, isn't all just some startups that can't afford to research their backups. And maybe they did, actually. Did anyone even hear if Zendesk was even affected by this happening?
Most of this has a latency that's orders of magnitude different. For electricity, those are microseconds (and for anything longer you do have UPS and generators and failover, right?), for the other things, it's days. "Half an hour" is just about the worst timeframe for a deadline.
Own a generator, have a stock pile of food and water, have a well I can run with the generator. Have plenty of clothing that can last many years. It isn't about being able to be self sufficient for forever, but being able to be self sufficient for long enough to weather a disaster (last time hurricane took down power for a few weeks, I only had minor inconveniences and was able to focus on helping others who weren't nearly as well prepared). Yes, it costs money and not all people can afford it, but for businesses, it should be something they do plan for.
Food and clothing aren't services they are once off purchases that you own, these are more like buying the software and self hosting it than SaaS.
Electricity and water are heavily regulated and often government owned, not run by Silicon Valley cowboys.
I don't want to make assumptions about your metabolism, but I find that I actually need to eat food on a fairly regular basis, to the point that I would not characterize it as a "once off" purchase.
If the company I buy food from decides to ghost all their customers with 30 minutes notice, I can walk a block in the opposite direction and buy food from a completely different company. And I can walk a block further and buy food from yet another company. Any of these companies vanishing into thin air would only cost me a 10 minutes of my time at most.
When I need more food I'll make another once off purchase. If the place that had lunch today has boarded up it's windows tomorrow I'll just eat somewhere else. If I bought that food from a supermarket it doesn't disappear from my pantry when the supermarket closes down.
My food supply is a self organising network that can route around any single point of failure. This is about as far removed from SaaS as you can get.
> Electricity and water are heavily regulated and often government owned, not run by Silicon Valley cowboys.
Yet, you still get Puerto Rico (electricity grid recovery failures of a few levels), and Flint (no clean water for you, because no).
Well, I guess every company on AWS is screwed, huh?
My coworker wrote a service that scales extremely well (he pretty much maintains, monitors and operates it by himself, across dozens of servers).
He has written multiple deployment options. Kubernetes, Docker swarm, bare VMs, etc. He has tested all of them and written installation tools that can deploy to any of them.
For exactly the reason you imply, he doesn't trust any of these options to always be available. I think this mentality has a lot to do with why his service scales so well.
And while he was doing that, how many features could he have added that would increase revenue for the company?
And while he was doing that, how many features could he have added that would increase revenue for the company?
There's more to building a long-term, sustainable business than optimizing revenue at any cost. And part of that "more" is exactly the work you to do insure against future risks.
What’s more likely to happen first - Amazon will stop offering an AWS service that you depend on at a price you are willing to pay or that the system you are building will become obsolete and when the time comes to replace it, you can choose different infrastructure? I would even say that your company going out of business is a bigger risk than depending on AWS.
The old saying that no one ever got fired for choosing IBM could be applied equally to AWS. The companies that chose IBM in the 70s and 80s can still buy new compatible systems. The companies that chose Stratus or VAX, not so much.
What’s more likely to happen first - Amazon will stop offering an AWS service that you depend on at a price you are willing to pay or that the system you are building will become obsolete and when the time comes to replace it, you can choose different infrastructure?
I think you're committing something of a "fallacy of the excluded middle" here. Those aren't the only two things that can happen. AWS could, for example, continue offering the service but not to you. We've already heard stories of AWS accounts being closed for various reasons, and it's hardly a stretch to imagine a CloudFront / Daily Stormer kind of scenario where Amazon decides to punt you for taking an unpopular political position, etc.
I would even say that your company going out of business is a bigger risk than depending on AWS.
I would say that I disagree, in at least some cases.
You're kind of operating on hindsight bias here.
Let’s take today’s equivalent - what’s the likelihood of Amazon not offering AWS or even Microsoft walking away from Azure and pulling the rug out from under developers?
Even businesses that chose Microsoft technology in the 90s when they were the clear leader have had a relatively painless upgrade and support path. They abandoned VB6 15 years ago, but you can still wrap your VB6 code as a COM object and interact with C#.
I’m saying that it is the safer bet to go with the largest provider with the largest customer base. In the 80s that would be IBM, in the 90s, MS over SUN (and the royal claustrof%%% that Java has become under Oracle) or Borland.
What is the "largest customer base" relative to? In the early 2000s, most Fortune 100s used Solaris [1]- presumably this trend started in the 90s, so I don't know why you think that a manager during that time period would choose Microsoft over Sun except for specific use cases (e.g., smaller hosting providers).
Let's look at yesterday's equivalent to AWS- Sun Grid. Sun was viewed back during the first dotcom era in much the same way that AWS is today. All that it would take to completely bork it would be an economic catastrophe that mirrors the first dotcom bubble. Obviously, Sun Grid is no more.
[1] https://news.netcraft.com/archives/2004/11/16/solaris_remain...
Solaris didn’t disappear overnight like Smyte did. When Sun was acquired by Oracle they just didn’t abandon thier customers - that would have been too value destroying.
You can still buy Solaris based systems today but you would have plenty of time to migrate. But seeing the writing on the wall you would have plenty of time to migrate.
I think there’s a balance you can strike between things. But it’s not always a single variable. Learning those things was probably valuable to him in other ways though. If he eventually grows his service and has to hire someone to manage ops he’ll be much more capable of evaluating them.
Because his service scales so well, it is bringing in a lot of revenue for the company (sorry, don't know the exact amount), at a very low cost. He is a very productive developer.
Whatever his salary is as an engineer, he is definitely underpaid.
Since I have mostly avoided doing side projects outside of work to learn something new [1], I understand spending extra time at work to scratch an itch and doing a low priority project as long as it doesn’t interfere with revenue generating - or cost reducing - initiatives.
[1] I would much rather work on a work related project using a new to me technology and if it requires longer hours to complete it, I’m find with putting in the extra time. It seems like more of a win, I get to learn a new technology, get to see it actually be used by others, and it will be helpful during either review time or worse case a resume builder.
Well, I guess every company on AWS is screwed, huh?
Screwed? Maybe, maybe not. At risk? Abso-fucking-lutely. There are plenty of stories out there of companies who had their AWS accounts shut-down for some random reason, plenty of stories of AWS outages, etc. Even if Amazon does't pull a "Smyte", there's plenty of risk associated with being on (or "solely on") AWS.
By and large, anytime you are completely dependent on someone else's platform, you are increasing risk by putting your destiny in someone else's hands.
Eventually yes. Fortunately there is enough demand for actual machines, but if that ever goes away and all of our compute power is owned by a handful of companies then we'll be well on our way to a scary dystopian future.
At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
Not even eventually. Right now. How many companies out there absolutely depend on all the value add services that AWS provides that are hard to replicate?
Does your business depend on Amazon Transcribe (or any other AWS-only service)? Do you have engineers competent to turn around an equivalent product for you in a short release window? No? (Don't lie to yourself. It's no.) Congratulations, you are now Amazon's bitch.
Internally we have physical servers, externally we us linode deployed via ansible, we don't rely on any linode specific features (api'snor anything).
I (as the sole programmer) made that decision early on.
AWS is a none starter because of the gradual vendor lockin (also nature of our business means growth/scalability is very easily predicted).
I think people will start feeling the pain when Amazon isn’t printing money anymore and starts looking at winding down less popular/profitable services.
> At the moment everyone is trying to do a land grab, but once that phase is over and the market consolidates on 2 or 3 cloud providers and sysadmin skills have disappeared we'll see what a disaster it is.
And if that happens people (I'm looking at YOU, software and devops engineers, with your damned sketchy business cases, desperate to move projects onto AWS mostly so you can bolster your CVs) will only have themselves to blame.
Fortunately I suspect there are enough of us jaded and sceptical types around to avoid it.
Truly competent Ops folks are setting up Terraform/Terragrunt modules for multiple providers and designing with cloud agnosticism in mind.
If they aren't, either they're incompetent, the CTO is, or you don't really have a company yet.
And then you get yet another level of abstraction. It's about like developers who want to wrap all of their data access in a repository pattern just in case they want to change databases.
Two things about that -- hardly anyone ever changes infrastructure just to save a little money and their code has to be so generic that they can't take advantage of all of the features of the platform.
The chances are far greater that your company will go out of business way before Amazon stops offering AWS. Theory is great that Terraform allows you to seamlessly move infrastructure between cloud providers, but the amount of risk and regression testing it takes to do so would make it a non starter for most companies.
The idea I'm floating isn't not to use AWS, it's to not be locked into it. Don't use the things that only they offer.
This isn't just about cloud provider lock-in, but something you need to do to have a good DR strategy.
Back on that Christmas Day that Linode got DDOSed, my company was 100% on their infrastructure. I only had to work a couple of hours that day because I had worked and planned in advance to launch our infrastructure on another host. I saved my company from a multi-million dollar loss that day.
So if you're using AWS and instead of using their services, you're just hosting your own equivalents on EC2 instances, what's the point? The entire reason behind using AWS is the "undifferentiated heavy lifting" - allowing them to take most of the burden off of you except where your company can add value.
If all you're doing is hosting a bunch of VPS's, you're really not saving yourself too much over just using a bunch of colocated servers. You can also get much cheaper, less reliable, less geographic diverse hosting somewhere else.
But I would have my head handed to me if I even suggested to a company to base their entire infrastructure on Linode instead of AWS, Azure, or even GCP.
There isn't one way or reason to use cloud infrastructure. Maybe for smaller companies the "undifferentiated heavy lifting" narrative makes sense and I've used it as such in the past.
Managing tens of millions of dollars in physical and cloud infrastructure at multiple providers/dcs for a publicly traded company like I am now? No, that's not the reason to use cloud infrastructure at all. Cloud in this case is more about capacity planning and avoiding managing hardware inventory than anything.
You realize that this is exactly what Netflix is doing? They are using a lot of AWS services and have completely tied their fate to AWS.
That's not entirely true now but has been in the past. They've actually tied their fates to both AWS and GCE in different parts of their infrastructure. The obvious example of this is the one or two global outages they've had due to misconfiguration in GCE this year.
I know their video encoding platform is heavily tied to AWS because the teams that built it optimized for cost (which for them makes a certain amount of sense) and they massively parallelize the encoding work. I talked to some of their engineers at re:Invent last year and it was kind of interesting. They aren't using a lot of AWS-specific stuff though, just AWS EC2 reserved instances.
I don't think that just because they're Netflix that everything they're doing is correct. This can't possibly be true.
That's also what's attractive about Kubernetes - it's nearly a provider-independent cloud API.
What part of terraform is cloud agnostic?
Uh, the whole thing? Do you need a link to the documentation or something?
So you are saying if I take my AWS infrastructure, which I launch with my terraform config, I can simply use that same config to launch the same infra on GCP or DO? Didn't think so. It's only slightly more agnostic than AWS CloudFormation, in that I can at least use some of the same conventions when I completely rewrite my terraform files.
I think you are confusing "agnostic" with "compatible with lots of different providers", but will require a total rewrite if you want to switch providers.
I'm as big of a Hashicorp fanboy as you will ever find for on prem implementations -- I've used Consul, Nomad, and Vault. But that's simply not true. Here is an example of a Terraform template straight from their getting started page.
https://github.com/terraform-providers/terraform-provider-aw...
There's nothing preventing you from using other providers as well. You just have to understand how to achieve the same results when you build your modules.
So you have to rewrite your modules for each platform so you're still tying yourself to one vendor and that still doesn't alleviate risks from migration or the large amount of regression testing -- just in case Amazon goes out of business or you can might be able to find an equivalent slightly cheaper?
If you are a small company, the difference isn't worth the risk and the expense. If you are a large company you can probable negotiate with Amazon.
That doesn't even take into account all of the dependencies that your developers have taken on AWS services....
Employees wouldn't have to keep one eye on their CV and plan for the next job if employers treated employees better and gave them long term security & oppertunities.
I think you might be underestimating AWS migrations initiated at the executive level.
I dare say the AWS sales pitch is even more effective at that level: "Increase your agility, reduce your operational overhead, scale out with the push of a button. Be like Netflix and Apple."
Most executives aren't well-equipped to probe the sharp corners that aren't in the pitch.
I think we're already there, and it isn't much of a dystopia. There are only 3 big cloud providers, and many major companies rely on them for everything.
But their business is built on providing this service. At what point would Amazon, Google or Microsoft say "we don't want that $xB in revenue, just turn it off. NOW!
Is that what you're suggesting is the future?
At that point government would step in and nationalise that part of the company :-)
More seriously I can see cloud going the way of telecoms in that there would be a provider of last resort if a telco goes bust another operator takes over to provide service.
The more likely risk is not them turning it off, but increasing prices.
Of course, it's a competitive market, so the other two providers would eat their lunch.
Does this argument apply if there was a signed contract with years remaining?
Evaluating the business viability and solvency of any important suppliers is something that you should be doing yourself even if you have a years-long contract.
Also, such risks can be insured.
Also, such risks can be mitigated in various ways - I recall having a contract where the source code of any product version delivered to us would be held in escrow at a third party, and in case the supplier went out of business or a bunch of other conditions, we'd get their code and legal rights to use/modify/maintain it ourselves or through some contractor.
A contract is between you and some entity - if you want to protect yourself against that entity "dying", then that can be done by a contract with someone else (e.g. insurer, escrow, etc) who can make you whole even if that entity is dead.
The existance of a contract does not prevent actions like this. A contract will get you compensated later, but if the provider cuts you out, nothing can prevent it. The outage will be there wether it is legal or not.
It really depends on the circumstances but often a contract makes no difference. If the service provider disappears (bankruptcy, acquisition, natural disaster) and there is no one to sue then the contract is irrelevant. If it's core functionality and you go out of business then being able to sue but having no money to makes the contract irrelevant. If the provider can absorb the cost of a lawsuit because you're their only customer and it's cheaper to drop you then a contract could be irrelevant. If the provider want's to launch a competing service and breaching the contract is just a cost of doing business then the contract is irrelevant.
Contracts can be valuable but often they are over valued.
Contracts don’t disappear on acquisition, of course. The new owner takes on the responsibilities of the acquired company, as well as its assets.
Companies also are not physical items that can disappear due to natural disasters.
The only ways they can disappear is by closing down, and that does mean honoring contracts, and by going bankrupt and that means using all leftover assets to satisfy all the parties whose contracts you’re breaking. And no, you can’t just transfer all assets away and leave a shell to go bankrupt because that is fraud and the judge can nullify the transfers.
I am not 100% familiar with exactly what Smyte does but if it uses heuristics to determine if a comment is spam/harassment, then thats not really core infrastructure.
Your company isnt going to die without it (its not like Hootsuite losing access to the Twitter API or Heroku cut off from aWS) But it might be more effective with it.
Just like all those people that screwed them over by depending on 3rd party CPUs instead of creating their own?
Daily reminder that every SaaS is a liability out of your control.
Proper answer: Sue Twitter for tortious interference of contract. Especially those that had something like a 3-year CONTRACT.
There needs to be a class action suit.
This is BS.
Fortunately for Smyte, its former customers probably all signed arbitration agreements.
c2
c1
I expect no less from a company that essentially elected Trump and continues to function as his mouthpiece to this day. Screwing over Smyte's customers must seem like nothing after they screwed over the entire country. What do employees there say to themselves every morning to justify staying? Trump continually breaks Twitter's rules, yet he hasn't been banned. He threatens world leaders, bullies, insults and lies non-stop, and Twitter is the bullhorn he uses to do it. Google's engineers stopped the company from bidding on war-related contracts, when are Twitter's engineers going to develop a backbone and get Trump banned?
I stopped using Twitter the day that jackass was inaugurated. Everyone who works at Twitter should be ashamed of themselves.
womp womp
Sure is rich seeing Laurie Voss (of NPM) complaining about unprofessionalism from a vendor.
And this, boys and girls, is why you never use any service of which you cannot host a fully working copy yourself.
What is your suggestion for a self hosted client fraud platform?
I do something similar (real-time detection of spam-comments) at https://blogspam.net/ and I'd be reluctant to even attempt to handle fraud.
If my site messes up then dodgy comments might get posted (unless they're blocked by other systems). If you misclassify an "is-fraud" check you either have a customer lose money, or a potential customer be denied service. Neither of those are great.
I suspect actually doing anti-fraud testing could be done simply with a few heuristics, knowing the kind of attacks I see on other sites I run, but the only way to be sure is to get a lot of data-volume and real-life results to compare against.
Hire a few competent engineers and make one
You could do that, and they'd quickly tell you that fraud detection requires vast datasets that your company almost certainly cannot generate or maintain in-house. Then what?
Find a way to live without it I guess. Certainly that would be better than having your entire production stop working because somebody else decided to take a multimillion-dollar check and run with no concern for you or contracts you had signed
If you can afford to cut multi-million dollar checks for a fraud/abuse prevention service, then you can also afford to retain a legal team who will recoup your losses and then some when those contracts are broken.
A temporary production outage due to a rare situation is far preferable to being at the perpetual mercy of hackers constantly pwning whatever half-baked homegrown security system you forced your engineers to implement with inadequate support & resources. Real businesses don't operate on "I guess" solutions and they don't just fork over huge chunks of cash willy-nilly without doing at least some cost-benefit analysis and making sure they're protected in case of contract breach.
It's not better.
My employers do just that - Fraud prevention is a fundamental part of our business operations.
It largely involves multiple teams of full time employees working on different aspects of fraud prevention (some of them are even Competent Developers!) a few dedicated offices, some teams embedded in regional development offices, a bunch of bespoke software and it's ongoing development and maintenance.
Thats in addition to the core dev teams being aware of and mitigating any possible vectors within the platform itself.
At any kind of scale it's rarely as easy as 'hiring a few competent developers'.
" which you cannot host a fully working copy yourself"
Sadly, no.
The majority of services have complex functions often requiring key data and updates, and it's just not possible, let alone feasible to 'self host' most things.
But it's a neat idea ... maybe there's a solution there ...
Make your own and host that, or risk this. again and again and again.
If this was not a painful enough lesson, the next time will be. :)
"Make your own and host that"
You're going to host your own version of Google maps?
Host your own version of 'background check' or 'credit check'?
Host your own DB of global financial transactions?
Host your own instance of weather data? Geospatial data?
It's not possible in most situations, and not feasible in most others, unfortunately.
> You're going to host your own version of Google maps?
I've never worked on a service which involves the other things you mentioned, but I've worked on some map-based services, and we went with self-hosted OpenStreetMap instances for exactly this reason :)
I'm almost entirely sure that Google is not in the habit of turning Services down with no warning. I guess I should have been more specific: never rely on a start-up to be here tomorrow
Here's a cookie, I'll take your life in exchange.
Based on this I assume the Smyte founders were retiring on a tropical island before the ink even dried.
I want names out there. Who is responsible for this!!!! Who was the CEO of smyte? Who is the manager at Twitter that gave the comand to shut down everything?
"Smyte" would be a good name for something that kicks users off a social network in a dystopian novel. "Smite" appears dozens of times in the Bible and means to strike someone, sometimes fatally.
It surprises me that this article has so many votes when nowhere has it actually provided any details of either the commercial terms of the acquisition OR whether any Smyte customers actually lost any money from prepaid service.
Sure, it sucks that people lost an API they were using, that much we know, but hundreds of votes for a “bad” commercial transaction on which we have no details?
The way I see it, a number of Smyte users just received a free billing period of API access?
I may have just stumbled in to the middle of this, but why the brouhaha?
Did customers pre-pay for Smyte service and not receive it?
FTA:
> According to reports from those affected, Smyte disabled access to its API with very little warning to clients, and without giving them time to prepare. Customers got a phone call, and then – boom – the service was gone. Clients had multi-year contracts in some cases.
Well, you know what they say: If you aren't paying for it, you aren't the customer, you're the product.
Some customers had 3 years left in their contracts. Some customers were huge, like npm.
Everybody got 30mins before lights out. It broke stuff.
As a group response to the sibling comments I'll point out that yes I read the article, but my question related to something which the article didn't cover: did anyone lose prepaid services from their contracts? I can't see this being discussed in any of the coverage of the Smyte acquisition.
Did you read the article?