Those skills at 18, the integrity to not sell something like this on the black market (assuming here that an 18 year old in Uruguay isn't exactly swimming in money), and a bounty from Google under his belt - he won't have trouble finding work. If I was considering hiring him, the creative bit of guerilla marketing for The Expanse he threw in there wouldn't hurt his chances either.
> a student from Uruguay who worked with Sugar Labs. Sugar Labs is the organization behind Sugar, the operating system for the [One Laptop per Child] XO-1 which the Uruguayan government has distributed to public primary schools. The XO-1 was Ezequiel’s first computer.
> Ezequiel’s curiosity in computer science was piqued when a technician came to his school to solve a simple bug that was affecting most XO’s. The technician used the command line which, up to that point, Ezequiel thought was useless. Realizing that the command line offered him a lot of power, Ezequiel began his exploration.
Let's try a thought experiment. To make things easier, imagine you're 21 years old, not 18, and have made up those 3 years working in the industry. You found this vulnerability, and have decided not to submit it for a bounty, but rather to the black market.
Who do you sell it to? I assume your answer will involve putting it up on some darknet version of Craigslist. That's fine, but then tell me: who's paying for it? What price do they assign to it? For instance: if you think you can sell it for $50k, who's paying that, and for what purpose?
Finally, what are the steps you take to safely complete the transaction?
(This is intended only to clarify arguments about the market for vulnerabilities like these, and not to suggest that the finding and the writeup aren't excellent, which they sure appear to be.)
Given the way this exploit was developed, I honestly think he got something on the order of the ceiling that someone who is not already known/connected on the black market could hope for. The act of advertising it in any way that could convey its potential value would likely burn it. Any pitch on the black market would tip your hand to people who had access to service logs for the past several weeks.
The only way you could sell it is if you already had connections directly to people who were known to need an exploit for this service and had a channel to approach them privately, IMO.
For someone in the author's situation, I think he got a very good deal and almost certainly made some great contacts that he will be happy to have in the future.
(That's a long way of saying that when I run that particular thought experiment, in the context of a vulnerability where discovery provides a certain amount of disclosure should anyone go back and check logs, I have a hard time seeing a more lucrative black market. I think the math would be different in less centralized cases.)
First, you can just go look at Zerodium's website and see what they'll buy. Notice that one-off vulnerabilities aren't there at all: there are no vulnerability types on their rate sheet that a single vendor can instantaneously fix worldwide with a single patch.
Notice also that with just a couple exceptions, RCEs in extremely widespread serverside web components are valued at $10k (if you believe their price list; I'm skeptical of it). Those are vulnerabilities that all have half-lives after patches are issued --- that's a ceiling for what anything like this could be worth.
I was in fact thinking of exploit brokers as well, so my wording was unclear. Let's call it the grey/black market.
In the scenario you described, without any other contacts and/or experience with transactions like this, I would approach an exploit broker. As for the payout - I assumed that any RCE vulnerability that qualifies for Google's highest bounty is likely to fetch a higher price elsewhere.
My experience with the field is limited and considering yours, if you are suggesting that this particular exploit would not fetch a significantly higher price, I shall stand corrected.
It actually turns out to be not that simple to approach organized crime for an one-off transaction. If that would be simple for you, then it would be exceedingly easy for LEO to get to these players as well.
Clearly. As pointed out/clarified in another comment I was mostly thinking of the grey area companies who buy 0-days and sell them to governments, law enforcement and god knows who.
I have not and I haven't been looking for one either.
I'm not quite sure what you're getting at here. If you're trying to point out that I haven't done my homework on this and that I don't have a sufficiently specific/workable plan how to approach it - that is accurate. I don't have exploits to sell.
In my previous comment I already stated the assumption that I made, if you feel it's incorrect, which clearly you do, feel free to correct/fill in whatever you think is missing. I'm not getting into a debate about something with which I don't have in-depth experience with someone who does.
Sorry, I'm interested in anyone's response to this, since the HN community reaction to any price paid in a bug bounty by a big company always seems to be "people can make more money on the black market". Rather than recapitulating all the previous debates about why that's not true, I'm interested in seeing someone --- doesn't have to be you --- work their way to an educated guess at a black market price for a bug like this one.
I think a lot of HN'ers believe that there's a market for high-severity bugs of any ilk, when, in reality, there's really only a liquid market for a pretty specific subset of those bugs.
I understand. I did in fact believe that this applies to any highest reward RCE vulnerability, thanks for pointing out that this may not be the case.
As for the black market price - I don't consider my security background sufficient for my guess to be anywhere near educated enough, so I'm bowing out.
Hah. As soon as I saw your initial setup I knew it was Wargames. The only winning move is not to play. I think you are right. When you add in all of the conditions required to sell a bug like this it becomes obvious (to me) that Google is offering more than you can get anywhere else without a lot of effort and or risk on the sellers part.
And not to spoil the game, but the subset of vulns that fetch good money has only narrowed in the last years as exploit mitigation has improved. The true unicorn 0days of yesteryear are almost always multiple hard earned bugs these days. Bugs in one vendor's project, even Google, it is cool they have such a high end reward, let alone 36k. Unless you crossed a line and exfiltrated data (high risk), I can't imagine getting this much money anywhere else.
Amusingly, 36k does look very similar to ~3wks of boutique infosec consulting, though, so for Google the price while generous probably makes sense.
Why to vendors pay bug bounties, if not to defend against financially motivated attackers? To defend against "digital vandals" who would damage systems but not profit from them? To defend against widespread grassroots attacks if an attack is published publicly?
I think there's a trust problem with that. How do you arrange the transaction? If I offer to sell you a zero-day for X BTC or something, how do you know that my zero-day is real and exploitable, and that I will actually give it to you and nobody else? How do I know you will actually send me the BTC? How do I demo it in a way that at least proves that it works without giving away enough info to recreate it?
I guess Zerodium has enough reputation that you can be reasonably sure they'll fork over the cash after you show them your bug. Building that kind of reputation on the darknet with cryptocurrency isn't going to be easy. If there was anyone like that out there, we'd already know about them.
Genuinely curious, what qualifies as one-off in these circumstances? Your phrasing-- "vulnerability...that a single vendor can instantaneously fix worldwide with a single patch" seems to me (non-professional but security interested techizen) to describe what a lot of vulnerabilities are. How would you (or a company like Zerodium) differentiate between a "one off" RCE exploit vs. the kind they would pay out for?
BTW -- not to fan out too hard, but I put together the pieces recently and realized in addition to Latacora you worked on Cryptopals and Microcorruption. As someone interested in exactly those areas with little knowledge of how to break in, thank you! I've found both great learning resources (although I'm embarrassingly early in the crypto challenges still..)
> How would you (or a company like Zerodium) differentiate between a "one off" RCE exploit vs. the kind they would pay out for?
In the GAE case, Google can patch all deployments of their service on the same day, which is an "instantaneous fix".
If it were an RCE with a webserver, OS or some other popular software, that can't be patched "instantaneously" "worldwide". Each entity who maintains their own (i.e. nginx, Drupal, Redis, etc.) instance needs to be alerted to the patch and then apply it. That's an incredibly long tail.
Also, if you look at the Zerodium payout chart --- again, I'm skeptical of it, but not of this observation --- you'll notice the serverside RCE stuff is at the bottom of the rate sheet. What buyers really want are clients, not servers. This despite the fact that clients are more efficiently patched than servers (the software targets that command the highest payouts are all auto-updated).
This also seems backwards to me? If you can get RCE or privilege escalation on a server, isn't that much worse? Or is it the difference in purpose between targets/adversaries? E.g., a server side vulnerability maybe you get to dump a companies records but something like no-click jailbreak + a high value individual => all their personal information?
It should now be starting to click that the extant markets for vulnerabilities don't value "severity" (or any other abstract scale message board nerds want to apply to vulnerabilities), but merely utility.
The hypothesis I like to come back to --- I'm pretty sure it's true --- is that vulnerabilities have value on the black market only if they fit into an existing business model, such that they can be dropped in and immediately be used to make money. People have to already be using some other vulnerability to do the exact same thing, and reliably making money with it.
People on HN like to tell stories about how a master criminal could make money with everything from Facebook CSRFs to serverside RCEs. But none of those kinds of exploits support current ongoing business concerns; they're all one-of-a-kind. Nobody buys a vulnerability speculatively to see if they might make a go of it --- they especially don't do that for a vulnerability that could be extinguished universally in moments by Google or Facebook's security team.
Aren't we seeing lots instances of the server-side model playing out? e.g., Company XYZ announces they've been breached and leaked XXX million user's data?
I could see an argument here towards the market price of the information -- breaching a company in the above scenario might get you zilch. Breaking an iOS device of an individual of interest gets you a lot of value.
Circling back though -- you're saying that this model isn't one in use. Do you agree with my real-life counterpoint? If so, why wouldn't there be a market? Or, is the market there and the low payouts from the likes of Zerodium reflect the actual low value of the product (and by extension, business model)?
There isn't a "low" payout on Zerodium for these bugs; there is no payout for them. Zerodium explicitly will not buy bugs in individual websites. Every vulnerability Zerodium will buy has a half-life.
Thanks, this makes sense. I was thinking perhaps a bit too narrowly from the perspective of someone producing the patch, rather than about that long tail of users on the vulnerable version.
Why wouldn't you just do both? You can get Zerodium to pay you and then go to the 'target' and submit the vuln through their channels. It seems plausible.
Any third parties reading this should take anything tptacek says with a grain of salt -- he's a well-known security contractor and his business model specifically consists of convincing his clients that they can't protect themselves by offering bug bounties.
There are dozens of firms like this, and have been for something like a decade now. The ostensibly totally above-board ones, like Zerodium, aren't "arms dealers"; they're controlled disclosure venues, which capture a premium from big companies and government buyers for access to threat intelligence information before they sit in vendor bug tracking systems for 3-9 months waiting for patches.
There are, of course, real brokers who will buy zero-day vulnerabilities for use by national IC's and LEO's. Their names don't get around like Zerodium's --- Zerodium sponsors conferences --- but they're not hard to find.
If you've got the kind of bug that these firms buy --- essentially, clientside RCE in hugely popular platforms --- you can probably do better selling to them than you can by collecting bounties from the vendors directly. It takes some moral flexibility, though, since really what you're doing is profiting from other people's exposure, and, especially with mobile clientside RCE, what you're really really doing is getting dissidents in Western-friendly dictatorships imprisoned. But you can do that.
But none of these firms (that I know) buy one-off vulnerabilities like a GCE RCE. All the vulnerabilities with high market values have half-lives, which is to say that even after they're patched, it will take weeks, months, or sometimes even years to see them eradicated, which gives them the residual value that props up their market price. In contrast to that, a GCE RCE that was actually exploited would be detected pretty quickly, and shut down with finality in a matter of hours.
It's not morally offensive to profit from the mistakes of a commercial entity. I'd guess that a large portions of companies are ultimately in this class.
And, really, dictators don't need help imprisoning dissidents -- it's sort of in the jobspec. It's not likely that a vuln would cause this to happen unless it was in systems created and secured specifically for sedition, in which case a researcher should be considerate of the potential damage.
"It's not morally offensive to profit from the mistakes of a commercial entity. I'd guess that a large portions of companies are ultimately in this class."
I would find it morally offensive but I'm a naive country bumpkin I guess. I have pointed out something that was mispriced rather than profit from their error. It cost me something like ~$50-100 extra but I slept better.
If I had to sell a random exploit, I'd try the NSA. Email random NSA types or political appointees with the offer until someone got back to me. I'd give them the exploit first and ask for payment greater than what I estimated it was worth from disclosing it.
If the NSA failed to pay, I would disclose it and try to collect that way. I doubt the NSA would reveal I went to them first as that would reveal that they didn't pay for exploits.
Government organizations should have the money and an incentive to pay.
That will be the equivalent of a 21 year old with no backgrounds robbing a bank. He'll just get caught, somehow. Either through financial transactions, taxes, leaves a trace somewhere, get involved with that criminal organization, etc... He'll eventually mess up.
But back to your question. If I remember correctly, the big companies started giving higher bounties because they noticed that some of the bugs are being sold on the dark market. Some of these bugs are certainly not unmarktable but the companies have to keep the promise.
Also I'm not an expert on these things. Maybe there is an entity buying whatever bug you can find and giving generous money. Think governmental institution who might be interested with whatever backdoor they can get.
I always "joke" to people we are professional writers. Actually professional infosec consultants, but our life blood is coherently documenting everything we find and making pretty executive summaries. It is hard to overstate how much I have seen the reporting differentiate infosec consulting firms over then years. Which is not to say we don't value the hard technical skills, but writing is really, really important too.
I’m not sure how familiar you are with South America but Uruguay is one of the most developed countries in the Western Hemisphere, in the group right behind the US and Canada.
Yes, but I don’t think you find a ton of highly skilled 18 year old software engineers in the US selling RCEs in the black market for $200K, so I’m not sure you find that in Uruguay.
You don't find a ton of people of any age anywhere selling serverside RCEs in individual websites for $200k, or $20k, or --- I will go out on a limb here --- $2k.
The $2k limb is shaky because I guess in theory you could buy a GCE RCE for $2k and flip it to Google for their bounty payout, which will probably be at least $3,133.70.
There's been a bit of salary inflation here, but yes, it's a very good payout and probably a year's salary for a mid-level developer (I'm a developer in Uruguay).
He'll have to pay taxes on it though (if he has no other income it won't be that bad, maybe 20%).
I'm not familiar with South America at all really, just assumed that in a country with a GDP per capita a third of the US, the incentive to not do this the official way will likely be higher. It's certainly possible that I may have overestimated the wealth differential and its effect in this particular situation.
> the integrity to not sell something like this on the black market
Wouldn't the investigation lead to him? He noted he did not know, at the time, that it was an RCE. So he would need to research further (where he might trigger an alarm). He also noted he utilized staging environment, which he had access to due to previous found vulnerabilities (so Google had his personal details).
So, now imagine he found something and he had sold it. If it was used, Google would do a thorough investigation to find if given vulnerability was abused in the past. And they find this guy using it exploratory and nothing else. It's not hard to put 1+1 together after that.
While the idea of finding an exploit and selling it for hundreds of thousands of dollars on black market sounds exciting, it wouldn't be so easy in this case.
There are companies who buy vulnerabilities/exploits and sell them to the highest, supposedly non-criminal - to whatever extent that can be applied to governments, law enforcement and intelligence agencies - bidder. That's mostly what I had in mind and black market is a misleading term for it, but I can't edit the comment now.
> I am 18-year-old student at the University of the Republic [Uruguay] interested in computer security
Someone could say that he could have gotten even more money by selling his findings in the black market, very difficult but doable. However, as someone who understands how studying computer science in a 3rd-world country is, getting USD +36k in a legal way and from a company that is considered one of the best in the industry, it must have felt very good to get that mail.
Plus I imagine putting that on your resume/CV will be a big help. An amazing talking point for interviews. Projects a strong sense of integrity. It would help them get a remote job or visa.
That's a lot of money, even in a first world country. That could get you a new car, or a deposit on a house (depending on where you are), or pay off a substantial chunk of your student loan (or all of it, if you're not in the USA).
I'm on the fence about student loans, obviously American student loans are obscene, but smaller ones seem reasonable to me.
It's not fair on those who went directly into the workforce or became an apprentice that the university students get free money for living costs for 3 or 4 years.
The current system in New Zealand (simplified slightly) is that university is free, there are no fees, and you can get up to $180 per week for living costs as an interest free loan. I think it's a perfectly reasonable system. That amount isn't really enough to cover all your expenses unless you are especially frugal, so students still need to go out and get a job (or work in the summer), which I think is perfectly fair.
Every year spent studying you miss out on working at some entry level job for ~25k/y. After 5 years of study, you (in Europe) have accumulated a debt of ~$25k-$50k depending on whether you've had a side job or help from your parents. That means you've lived for ~$10k/y for 5 years, where someone in the workforce lives on significantly more, and at the end you're in debt as well.
Obviously the upside is that you get a job where it is reasonable to expect that you'll be able to repay the loan in about 10 years, but that's something you worked for, and something an ambitious person in the workforce might achieve as well.
Anyway, I don't think you can simply say it's unfair to aid students in their living expenses while they're studying. Obviously they're usually having a great time studying, but they also don't own a car, or a house and instead live in tiny apartments sharing privacy with other students. They usually delay their family building for the entire span of their studies, and often even until they've paid of (most of) their debt. But yeah, have them work some extra in the summer while the workforce is driving their caravans to southern France...
(Edit: I appear to be a broken record) I spent a spell on a bug bounty program. There are some beasts out there in the 16-20yo age range, it's pretty crazy.
What I appreciated most about it (or maybe this is just his writeup skills), but how not advanced each of the things were. It was just mostly him Googling things and making random guesses at the rabbit hole until he found something.
The skill, I guess, comes in piecing it all together and consistently making good guesses.
Yes it is. I do not want to be that stickler but this is the charts for that term:
1st world: US, UK, West Germany, essentially western countries
2nd world: Soviets, East Germany, communist countries
3rd world: everyone who doesn’t fit the Cold War theater.
Either way, a GDP per capita of $15k isn’t considered “first” at whatever category you want to assign it by. Unfortunately there are no “1st world” countries in Latin America.
"X World" has gone he way of "begs the question", "Literally vs. Figuratively", "One bad apple..", etc. Popular usage has destroyed the original meaning, and it's no use trying to get people on board with what those phrases actually mean.
While the terms may have been originally coined during the Cold War, the meaning and common usage has clearly changed. If you use 'third world countries' in a conversation hardly anyone will assume that you're including Switzerland in that. When enough people use a word "wrong" for a long enough time, they kinda stop being wrong.
This could be related to the fact that while you still hear a lot about "first-world" and "third-world" countries, you don't ever hear anyone use the term "second-world".
“Please stop exploring this further, as it seems you could easily break something” has got to be the best reply one can receive to a bug bounty report.
That's what struck me. From the well told story I feel like there were several points of "welp probably can't go further then this better move on to something else" but he kept going and going and sure enough he got somewhere. Hope to see more work published by him over the next few years!
I wish all companies had the internal connectedness to be able to react to realtime updates and coordinate the exact split second to hold up the friendly stop sign!
There have been few stories on here of bounty hunters who've either been scared to go any further than they have, obviously giving the remote end a good excuse to pay less; or who've taken things slightly too far, generating mildly put-out emails and stepped-on toes (which can't bode too well for future interactions).
So this the best thing you can hear from that standpoint as well.
Straightforward communication seems to be a holy grail, which is understandable: besides needing full internal access to all the areas affected by the full scope of vulnerability (either directly or via intermediaries who are good communicators) so you can figure out when to hit the stop button, you also need to have the resources to delicately balance management risk and developer egos.
That sounds like a "fun" job...
---
I guess this demonstrates the fundamental requirement of security: the recipient is the one who needs the non-broken understanding of security, both a) properly in and of itself, and b) how it relates to the domain they're in.
There are a thousand public examples of where this has been gotten hopelessly wrong; my bookmarks are a mess, so here's the (somewhat poorly related) one I can remember how to find via Google (it's as bad as all the others, wouldn't mind being reminded and learning about new ones). This one is not the same kind of vulnerability (there's no need to coordinate pushing a stop button, it's a data breach), it's just an example of grossly misunderstood security and utterly broken internal communication.
"When issuing the reward, we'll take into account what you could have achieved with this access" makes me laugh.
How scary must that be for the Google team? You know you've messed up so badly and the person who is investigating is doing so blindly with no knowledge or accountability if he breaks something. Yikes.
Kudos to everyone for doing the right things. And great bounty- the average yearly income in Uruguay is $2000-$3000 USD per household. This guy just got awarded more than ten times that.
Same; I worked on GAE in 2013 and it's so funny to read the story of someone exploring, discovering, and being so close to breaking something you know really well. There's a few moments in here where I thought "oh man, you could have done XXXX and that would have been so bad!". Definitely understand why they gave them the big bucks for this one.
I’ve been on the receiving end where hobbyists were trying (and eventually succeeded) to hack our DRM scheme.
It was really fun to read the forums and see how, day by day, they managed to get closer. Since it wasn’t really crucial IP to begin with, we were rooting for the little guys to see how close they would get, secure in the knowledge that our algorithm was solid. :-)
The final exploit that granted them access was due to a supplier who replaced an earlier validated random generator with something not quite as random, which enabled replay attacks.
I didn't work on GAE, but I know a fair amount about how it ran and, as I read, too, I could think of quite a few things I would have done, had I been a malicious actor. Denial of service is the first to come to mind. I'm sure a real security person could do a lot more.
* Grab nearly all of googles source code (no extra auth required for that, since so many libraries read config etc from the source code repo)
* Make the right requests to one endpoint he found and retrieve company financials, number of hits to every google service, the name of every application running in every datacenter, etc.
* With the above two things, you know the location of services and every RPC endpoint on them, and all access control configs. You can take your sweet time to audit the 10's of millions of lines of code to find vulnerabilities and get to attack as an authenticated (albeit low privilege) user. A lot of stuff is open to all authenticated internal users.
* For example, you could take down any google service by quitting all the application servers at the same time by calling the right debugging RPC. You'd be caught obviously tho.
Production code has no unauthenticated access to source. In all the cases I know, there was a data push for dynamic configurations from source to production, usually adding access control, auditing, validation, etc. Static configurations would be baked into the container image.
That said, it looks like in this case he would been able to turn on access to google3, through a special flag that is probably meant for internal GAE apps that work on source code. Presumably, the flag allows the app to use a GAE proxy that authenticates to Piper and provides a filesystem-like interface. It's not clear what would have happened at this point, because it is likely there exists a finer grained quota for source access than just all of GAE. These kinds of new RPC traffic would have been visible in Dapper and Census data, plus there are booby traps everywhere aka defense in depth.
At the end of the day, even a non-production GAE environment runs on Borg, talks to GFS and Bigtable, etc. They're not going to run the entire stack on bare metal. The environments will have different configurations, will be backed by different Borg jobs, hopefully will run under different Borg identities, etc.
The articles about Piper and co. have made it clear that it ran on top of Bigtable first and then Spanner, i.e. in the production network, spread over ten locations. There's file level auditing and access control. It would be very dumb of them not to have intrusion and anomaly detection in place.
Android is probably one of those markets that are more liquid than most for "black market" sources (as talked about elsewhere in the comments for this)
Android is wormable, and potentially not repairable by google.
For example, with a decent remote android exploit, I could distribute a patched Google Play Services to all vulnerable handsets which disables updates and then listens to my own command and control infrastructure for further actions.
I can now hold the phones hostage and extort google for money to regain control of them.
That would be pretty brutal and cause people to quit trusting android phones.
But I think the same could be accomplished with the access he had, or worse, but would have taken a lot more work. He also would have needed to avoid detection too. His access sounds more troubling than the Aroura attacks they had years ago.
Software architecture isn't like that. Nobody has a list of all the software running in every datacenter -- not in any large corporation anywhere in the world, and an API isn't going to change that.
And, you can't quit all application servers at the same time. These are distributed, self-healing, and highly-redundant -- meaning that there are thousands of copies of each service, and the system will bring up new copies to replace the ones you kill, without loosing service (though you could, potentially, affect quality of service).
Even the control mechanisms that allow all this magic to happen are run on the same platform, meaning that you can't affect the managers in this way either.
The sort of things you can do are: affect billing and reporting, find sensitive company data, and, potentially, execute code remotely (though that will probably be in a container, and not have access to much else).
Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions. The code is huge, so downloading it will take forever, and then you'd have to read it. Finally, it's written in dozens of programming languages, some, like Golang, are unreadable to anyone but experts.
Google API Auth/Access is extremely tedious, so I guess someone just didn't bother. Basically you have to get an token, format it, make a couple of http posts, and finally you will have a token to make an access-token-token ... Now image all the steps you would have to make to create a new type of access to some internal API, that should not have public access anyway. Probably saved six months work. And the libraries are probably hiding all the obscurity so no-one did notice until this guy started digging.
Sounds plausible. Now I’m curious how much extra work this finding has caused teams at Google. There are probably many other similar insecure paths that need to be cleaned up.
Another thing that is very admirable and bold is that he had no actual idea that he discovered a RCE vuln but went ahead and confidently contacted google.
How many would stop at "Eh I managed to fire requests to a hidden RPC service in google, but couldn't figure out how to make it do anything useful to qualify".
Put yourselves and your work/findings out there people!
Yes, if he had known more about Google infrastructure, he could have done some damage, both active and passive. He had access to a lot of internals. I was surprised by them at first, too, but they all make sense. On the other hand, he would have probably been caught fairly rapidly.
One thing to note is that Google, like any responsible organization should, has layered security and threat models that include insiders as a potential threat; hopefully, while he could make RPC calls to internal services (which is itself a serious problem, hence the giant bounty) he hopefully could not authenticate to do anything any serious damage or access any sensitive information.
I agree that there are multiple layers of security in place, which is why I said he would have been caught fairly quickly. I'm not going to go into details, but, at the very least, he had access to quite a bit of proprietary information, i.e. private to Google, not necessarily user data. I talked to another former Googler and we agreed that this is at least as bad as a previous disclosure: https://packetstormsecurity.com/files/129406/Google-App-Engi...
That's probably not quite true. "google3 file access" most likely means the ability to open paths with the "google3 file library", which lets you use paths like "/gfs/<cluster>/...", and similar for all the different storage services. But you'd also need an identity with permissions to read whatever you're reading. Without an identity that's in ACLs and stuff, you'd only be able to see "world-readable" data, which is still quite serious because who knows what's lying around, but probably nothing sensitive like source code.
You want the part of the write-up where it mentions he gets FILE_GOOGLE3_ACCESS.
I'm not sure what access that actually gives, but that email thread showing people getting +cc'd on the ticket speaks pretty obviously to the severity and magnitude of the problem.
This is not an inconsequential amount of cash, for sure. Especially at 18! Congrats! And a great write-up to boot. Just awesome.
All that said an honest question: why would a company like Google not pay insane amounts of money for these kinds of bug finds? What would they pay their own people to find them? Seems like RCE on App Engine should be worth 100K+ and then some on top for giggles just because they can.
Obviously having a standard policy makes sense so that your community understands what to expect but as Google, what's your operational impact if you triple / quadruple vs. market value of the exploits?
They pay their own people a salary - having a dependable living (at a very, very nice rate) is a pretty awesome thing. Sometimes there are bonuses involved, but they're usually not on the order of magnitude of external bug bounties.
My understanding of the pricing is that it's designed to make ethical behavior profitable enough that fewer people are tempted to sell the exploits on the black market, not necessarily to out-compete the black market entirely. I think this person's find is a great example - it's a resume-booster, a great experience, a very nice cash infusion, and helps, instead of hindering, their job prospects with future employers.
Anecdotal, but I got my job at Google through participating in their bug bounty program. The first set of interviews you have that ask general CS questions might not care what you did last week or last year, but when you talk with the team who wants to hire you they certainly do care.
Huh, so you can run binaries in GAE by downloading a statically linked app to /tmp, chmod'ing & executing it? And there would be no limits on how it's run? That's crazy & pretty cool!
The Java 8, Node.js (just announced at I/O) and GCF environments use a new sandbox that should allow you to run any binary. Of course there are limits, and you have to pay to run things on App Engine past the free tier, but most things should run just fine.
The older GAE sandboxes didn't let you do this (and had a ton of other limitations as well)
But that's would be counter to their interests. They want to hire this kid when he graduates. If they paid 10x their current bounty rates they'd have paid over $400,000 to him in the last couple of years of his free time. That's a great way to never be able to hire him.
Dude has cashed out a $10k and $30k bug bounty at the age of 18. Either he's lucky or he's very good. If he's the latter that $400k turns into an annual bounty.
And very, very few Google engineers make that kind of money.
> And very, very few Google engineers make that kind of money.
You'd be surprised. In Mountain View, everyone level 6 or above makes at least that amount, and most level 5s probably do as well. I'd guess that probably 20% of engineers are T5 or above, which is a ton of people when you multiply by tens of thousands of engineers.
Very few as a ratio. There aren't that many T6's and I'm not sold on the idea that "most" T5's are making $400,000 when the average is around $350,000 according to levels.fyi.
I don't think it's so easy to sell a vulnerability on the black market. If you send the code first they will have no incentive to send the money, if you get the money, you might not send the code.
I have almost 15 years on the OP and aren't even half as talented. It looks like they've received almost $60k from Google across five bug bounties, very impressive.
In Australia, only if you are engaged in a business activity. This means that if you come across some small bug and get a bounty of $1000, then it's probably not even taxable income.
Of course it's taxable. Unless its a gift which, a payout from a company for completing work which the ATO will say is not a gift, then it's income and it'll get added onto your taxable income and tax appropriately. For something like a bug bounty payout, you'll have to declare it yourself as other income.
Those skills at 18, the integrity to not sell something like this on the black market (assuming here that an 18 year old in Uruguay isn't exactly swimming in money), and a bounty from Google under his belt - he won't have trouble finding work. If I was considering hiring him, the creative bit of guerilla marketing for The Expanse he threw in there wouldn't hurt his chances either.
There's some backstory in his Google Code-In article (Summer of Code for younger students) https://opensource.googleblog.com/2016/11/stories-from-googl...
> a student from Uruguay who worked with Sugar Labs. Sugar Labs is the organization behind Sugar, the operating system for the [One Laptop per Child] XO-1 which the Uruguayan government has distributed to public primary schools. The XO-1 was Ezequiel’s first computer.
> Ezequiel’s curiosity in computer science was piqued when a technician came to his school to solve a simple bug that was affecting most XO’s. The technician used the command line which, up to that point, Ezequiel thought was useless. Realizing that the command line offered him a lot of power, Ezequiel began his exploration.
Let's try a thought experiment. To make things easier, imagine you're 21 years old, not 18, and have made up those 3 years working in the industry. You found this vulnerability, and have decided not to submit it for a bounty, but rather to the black market.
Who do you sell it to? I assume your answer will involve putting it up on some darknet version of Craigslist. That's fine, but then tell me: who's paying for it? What price do they assign to it? For instance: if you think you can sell it for $50k, who's paying that, and for what purpose?
Finally, what are the steps you take to safely complete the transaction?
(This is intended only to clarify arguments about the market for vulnerabilities like these, and not to suggest that the finding and the writeup aren't excellent, which they sure appear to be.)
Given the way this exploit was developed, I honestly think he got something on the order of the ceiling that someone who is not already known/connected on the black market could hope for. The act of advertising it in any way that could convey its potential value would likely burn it. Any pitch on the black market would tip your hand to people who had access to service logs for the past several weeks.
The only way you could sell it is if you already had connections directly to people who were known to need an exploit for this service and had a channel to approach them privately, IMO.
For someone in the author's situation, I think he got a very good deal and almost certainly made some great contacts that he will be happy to have in the future.
(That's a long way of saying that when I run that particular thought experiment, in the context of a vulnerability where discovery provides a certain amount of disclosure should anyone go back and check logs, I have a hard time seeing a more lucrative black market. I think the math would be different in less centralized cases.)
You don't have to think about it too hard, there's companies that will help you with the transaction. https://www.zerodium.com/
First, you can just go look at Zerodium's website and see what they'll buy. Notice that one-off vulnerabilities aren't there at all: there are no vulnerability types on their rate sheet that a single vendor can instantaneously fix worldwide with a single patch.
Notice also that with just a couple exceptions, RCEs in extremely widespread serverside web components are valued at $10k (if you believe their price list; I'm skeptical of it). Those are vulnerabilities that all have half-lives after patches are issued --- that's a ceiling for what anything like this could be worth.
Second, Zerodium isn't "the black market".
I was in fact thinking of exploit brokers as well, so my wording was unclear. Let's call it the grey/black market.
In the scenario you described, without any other contacts and/or experience with transactions like this, I would approach an exploit broker. As for the payout - I assumed that any RCE vulnerability that qualifies for Google's highest bounty is likely to fetch a higher price elsewhere.
My experience with the field is limited and considering yours, if you are suggesting that this particular exploit would not fetch a significantly higher price, I shall stand corrected.
It actually turns out to be not that simple to approach organized crime for an one-off transaction. If that would be simple for you, then it would be exceedingly easy for LEO to get to these players as well.
Clearly. As pointed out/clarified in another comment I was mostly thinking of the grey area companies who buy 0-days and sell them to governments, law enforcement and god knows who.
What's an "exploit broker"? Where would you find them? What price would you ask for this vulnerability?
I was referring to companies like Zerodium.
If you look at Zerodium's FAQ, they explicitly say that they don't buy one-off vulnerabilities like this. Have you found a firm that does?
I have not and I haven't been looking for one either.
I'm not quite sure what you're getting at here. If you're trying to point out that I haven't done my homework on this and that I don't have a sufficiently specific/workable plan how to approach it - that is accurate. I don't have exploits to sell.
In my previous comment I already stated the assumption that I made, if you feel it's incorrect, which clearly you do, feel free to correct/fill in whatever you think is missing. I'm not getting into a debate about something with which I don't have in-depth experience with someone who does.
Sorry, I'm interested in anyone's response to this, since the HN community reaction to any price paid in a bug bounty by a big company always seems to be "people can make more money on the black market". Rather than recapitulating all the previous debates about why that's not true, I'm interested in seeing someone --- doesn't have to be you --- work their way to an educated guess at a black market price for a bug like this one.
I think a lot of HN'ers believe that there's a market for high-severity bugs of any ilk, when, in reality, there's really only a liquid market for a pretty specific subset of those bugs.
I understand. I did in fact believe that this applies to any highest reward RCE vulnerability, thanks for pointing out that this may not be the case.
As for the black market price - I don't consider my security background sufficient for my guess to be anywhere near educated enough, so I'm bowing out.
No problem. You've been a good sport, thanks! (I'm still interested in seeing someone take a crack at this.)
Hah. As soon as I saw your initial setup I knew it was Wargames. The only winning move is not to play. I think you are right. When you add in all of the conditions required to sell a bug like this it becomes obvious (to me) that Google is offering more than you can get anywhere else without a lot of effort and or risk on the sellers part.
And not to spoil the game, but the subset of vulns that fetch good money has only narrowed in the last years as exploit mitigation has improved. The true unicorn 0days of yesteryear are almost always multiple hard earned bugs these days. Bugs in one vendor's project, even Google, it is cool they have such a high end reward, let alone 36k. Unless you crossed a line and exfiltrated data (high risk), I can't imagine getting this much money anywhere else.
Amusingly, 36k does look very similar to ~3wks of boutique infosec consulting, though, so for Google the price while generous probably makes sense.
Why to vendors pay bug bounties, if not to defend against financially motivated attackers? To defend against "digital vandals" who would damage systems but not profit from them? To defend against widespread grassroots attacks if an attack is published publicly?
To encourage people to report bugs they've found and to research bugs and then report them.
Just to trigger the proverbial trap because I'm curious, what's wrong with, "darknet Craigslist, paid in some crypto coin"?
I think there's a trust problem with that. How do you arrange the transaction? If I offer to sell you a zero-day for X BTC or something, how do you know that my zero-day is real and exploitable, and that I will actually give it to you and nobody else? How do I know you will actually send me the BTC? How do I demo it in a way that at least proves that it works without giving away enough info to recreate it?
I guess Zerodium has enough reputation that you can be reasonably sure they'll fork over the cash after you show them your bug. Building that kind of reputation on the darknet with cryptocurrency isn't going to be easy. If there was anyone like that out there, we'd already know about them.
Nothing, but to make a complete argument you have to provide a hypothetical buyer, their price, and why they'd pay it.
Here is an angle that could work.
Known security person buys this in order to turn it in themselves for a higher bounty and increased rep.
The premise is you somehow making more than Google will pay.
Genuinely curious, what qualifies as one-off in these circumstances? Your phrasing-- "vulnerability...that a single vendor can instantaneously fix worldwide with a single patch" seems to me (non-professional but security interested techizen) to describe what a lot of vulnerabilities are. How would you (or a company like Zerodium) differentiate between a "one off" RCE exploit vs. the kind they would pay out for?
BTW -- not to fan out too hard, but I put together the pieces recently and realized in addition to Latacora you worked on Cryptopals and Microcorruption. As someone interested in exactly those areas with little knowledge of how to break in, thank you! I've found both great learning resources (although I'm embarrassingly early in the crypto challenges still..)
> How would you (or a company like Zerodium) differentiate between a "one off" RCE exploit vs. the kind they would pay out for?
In the GAE case, Google can patch all deployments of their service on the same day, which is an "instantaneous fix".
If it were an RCE with a webserver, OS or some other popular software, that can't be patched "instantaneously" "worldwide". Each entity who maintains their own (i.e. nginx, Drupal, Redis, etc.) instance needs to be alerted to the patch and then apply it. That's an incredibly long tail.
Also, if you look at the Zerodium payout chart --- again, I'm skeptical of it, but not of this observation --- you'll notice the serverside RCE stuff is at the bottom of the rate sheet. What buyers really want are clients, not servers. This despite the fact that clients are more efficiently patched than servers (the software targets that command the highest payouts are all auto-updated).
This also seems backwards to me? If you can get RCE or privilege escalation on a server, isn't that much worse? Or is it the difference in purpose between targets/adversaries? E.g., a server side vulnerability maybe you get to dump a companies records but something like no-click jailbreak + a high value individual => all their personal information?
Or else, why do buyers want clients?
It should now be starting to click that the extant markets for vulnerabilities don't value "severity" (or any other abstract scale message board nerds want to apply to vulnerabilities), but merely utility.
The hypothesis I like to come back to --- I'm pretty sure it's true --- is that vulnerabilities have value on the black market only if they fit into an existing business model, such that they can be dropped in and immediately be used to make money. People have to already be using some other vulnerability to do the exact same thing, and reliably making money with it.
People on HN like to tell stories about how a master criminal could make money with everything from Facebook CSRFs to serverside RCEs. But none of those kinds of exploits support current ongoing business concerns; they're all one-of-a-kind. Nobody buys a vulnerability speculatively to see if they might make a go of it --- they especially don't do that for a vulnerability that could be extinguished universally in moments by Google or Facebook's security team.
Aren't we seeing lots instances of the server-side model playing out? e.g., Company XYZ announces they've been breached and leaked XXX million user's data?
I could see an argument here towards the market price of the information -- breaching a company in the above scenario might get you zilch. Breaking an iOS device of an individual of interest gets you a lot of value.
Circling back though -- you're saying that this model isn't one in use. Do you agree with my real-life counterpoint? If so, why wouldn't there be a market? Or, is the market there and the low payouts from the likes of Zerodium reflect the actual low value of the product (and by extension, business model)?
There isn't a "low" payout on Zerodium for these bugs; there is no payout for them. Zerodium explicitly will not buy bugs in individual websites. Every vulnerability Zerodium will buy has a half-life.
Thanks, this makes sense. I was thinking perhaps a bit too narrowly from the perspective of someone producing the patch, rather than about that long tail of users on the vulnerable version.
Why wouldn't you just do both? You can get Zerodium to pay you and then go to the 'target' and submit the vuln through their channels. It seems plausible.
It seems plausible to you that you can get Zerodium to pay you for a vulnerability that their FAQ specifically says they're not interested in buying?
Any third parties reading this should take anything tptacek says with a grain of salt -- he's a well-known security contractor and his business model specifically consists of convincing his clients that they can't protect themselves by offering bug bounties.
Wow, a digital arms dealer. How is that they have not been destroyed or captured by someone's military?
There are dozens of firms like this, and have been for something like a decade now. The ostensibly totally above-board ones, like Zerodium, aren't "arms dealers"; they're controlled disclosure venues, which capture a premium from big companies and government buyers for access to threat intelligence information before they sit in vendor bug tracking systems for 3-9 months waiting for patches.
There are, of course, real brokers who will buy zero-day vulnerabilities for use by national IC's and LEO's. Their names don't get around like Zerodium's --- Zerodium sponsors conferences --- but they're not hard to find.
If you've got the kind of bug that these firms buy --- essentially, clientside RCE in hugely popular platforms --- you can probably do better selling to them than you can by collecting bounties from the vendors directly. It takes some moral flexibility, though, since really what you're doing is profiting from other people's exposure, and, especially with mobile clientside RCE, what you're really really doing is getting dissidents in Western-friendly dictatorships imprisoned. But you can do that.
But none of these firms (that I know) buy one-off vulnerabilities like a GCE RCE. All the vulnerabilities with high market values have half-lives, which is to say that even after they're patched, it will take weeks, months, or sometimes even years to see them eradicated, which gives them the residual value that props up their market price. In contrast to that, a GCE RCE that was actually exploited would be detected pretty quickly, and shut down with finality in a matter of hours.
It's not morally offensive to profit from the mistakes of a commercial entity. I'd guess that a large portions of companies are ultimately in this class.
And, really, dictators don't need help imprisoning dissidents -- it's sort of in the jobspec. It's not likely that a vuln would cause this to happen unless it was in systems created and secured specifically for sedition, in which case a researcher should be considerate of the potential damage.
I don't know what to tell you. Maybe you'd have an easier time selling vulnerabilities than I would.
"It's not morally offensive to profit from the mistakes of a commercial entity. I'd guess that a large portions of companies are ultimately in this class."
I would find it morally offensive but I'm a naive country bumpkin I guess. I have pointed out something that was mispriced rather than profit from their error. It cost me something like ~$50-100 extra but I slept better.
I guess IC means "Intelligence Community", but what is a LEO? A Google search didn't provide meaningful results.
Probably Law-Enforcement Organization?
Law Enforcement Officers.
The same way arms dealers stay in business, by being needed.
I don't think you even need to go to the darknet. There are plenty of legal avenues open to you for selling exploits.
Name one that buys vulnerabilities like these.
http://zerodium.com
If I had to sell a random exploit, I'd try the NSA. Email random NSA types or political appointees with the offer until someone got back to me. I'd give them the exploit first and ask for payment greater than what I estimated it was worth from disclosing it.
If the NSA failed to pay, I would disclose it and try to collect that way. I doubt the NSA would reveal I went to them first as that would reveal that they didn't pay for exploits.
Government organizations should have the money and an incentive to pay.
That will be the equivalent of a 21 year old with no backgrounds robbing a bank. He'll just get caught, somehow. Either through financial transactions, taxes, leaves a trace somewhere, get involved with that criminal organization, etc... He'll eventually mess up.
But back to your question. If I remember correctly, the big companies started giving higher bounties because they noticed that some of the bugs are being sold on the dark market. Some of these bugs are certainly not unmarktable but the companies have to keep the promise.
Also I'm not an expert on these things. Maybe there is an entity buying whatever bug you can find and giving generous money. Think governmental institution who might be interested with whatever backdoor they can get.
As someone who worked in a bug bounty program, the skill and age of this individual isn't what sets them apart. It's the write up.
As someone who has been on the other end of receiving incoherent and inaccurate bug bounty reports, this!
To find the bug is impressive. To write about it so well is truly exceptional.
I always "joke" to people we are professional writers. Actually professional infosec consultants, but our life blood is coherently documenting everything we find and making pretty executive summaries. It is hard to overstate how much I have seen the reporting differentiate infosec consulting firms over then years. Which is not to say we don't value the hard technical skills, but writing is really, really important too.
The best part is when your kids complain about having to write a 1500 word essay and you show them what's up.
I’m not sure how familiar you are with South America but Uruguay is one of the most developed countries in the Western Hemisphere, in the group right behind the US and Canada.
His bounty is equivalent to a year's salary for a very good senior developer in here. So I believe his point stands.
Yes, but I don’t think you find a ton of highly skilled 18 year old software engineers in the US selling RCEs in the black market for $200K, so I’m not sure you find that in Uruguay.
You don't find a ton of people of any age anywhere selling serverside RCEs in individual websites for $200k, or $20k, or --- I will go out on a limb here --- $2k.
The $2k limb is shaky because I guess in theory you could buy a GCE RCE for $2k and flip it to Google for their bounty payout, which will probably be at least $3,133.70.
There's been a bit of salary inflation here, but yes, it's a very good payout and probably a year's salary for a mid-level developer (I'm a developer in Uruguay).
He'll have to pay taxes on it though (if he has no other income it won't be that bad, maybe 20%).
I'm not familiar with South America at all really, just assumed that in a country with a GDP per capita a third of the US, the incentive to not do this the official way will likely be higher. It's certainly possible that I may have overestimated the wealth differential and its effect in this particular situation.
> the integrity to not sell something like this on the black market
Wouldn't the investigation lead to him? He noted he did not know, at the time, that it was an RCE. So he would need to research further (where he might trigger an alarm). He also noted he utilized staging environment, which he had access to due to previous found vulnerabilities (so Google had his personal details).
So, now imagine he found something and he had sold it. If it was used, Google would do a thorough investigation to find if given vulnerability was abused in the past. And they find this guy using it exploratory and nothing else. It's not hard to put 1+1 together after that.
While the idea of finding an exploit and selling it for hundreds of thousands of dollars on black market sounds exciting, it wouldn't be so easy in this case.
There are companies who buy vulnerabilities/exploits and sell them to the highest, supposedly non-criminal - to whatever extent that can be applied to governments, law enforcement and intelligence agencies - bidder. That's mostly what I had in mind and black market is a misleading term for it, but I can't edit the comment now.
See also tptacek's comments in this thread.
> I am 18-year-old student at the University of the Republic [Uruguay] interested in computer security
Someone could say that he could have gotten even more money by selling his findings in the black market, very difficult but doable. However, as someone who understands how studying computer science in a 3rd-world country is, getting USD +36k in a legal way and from a company that is considered one of the best in the industry, it must have felt very good to get that mail.
Congratulations, and keep the good work.
Plus I imagine putting that on your resume/CV will be a big help. An amazing talking point for interviews. Projects a strong sense of integrity. It would help them get a remote job or visa.
I remember crying/laughing/screaming when I won three GameBoy Advance games from some kids magazine. I would say he's pretty happy with himself.
That's a lot of money, even in a first world country. That could get you a new car, or a deposit on a house (depending on where you are), or pay off a substantial chunk of your student loan (or all of it, if you're not in the USA).
OT: Luckily for him, he studies at a public university so much likely he doesn't even need a student loan.
Students shouldn't need loans. It's an aberration.
I'm on the fence about student loans, obviously American student loans are obscene, but smaller ones seem reasonable to me.
It's not fair on those who went directly into the workforce or became an apprentice that the university students get free money for living costs for 3 or 4 years.
The current system in New Zealand (simplified slightly) is that university is free, there are no fees, and you can get up to $180 per week for living costs as an interest free loan. I think it's a perfectly reasonable system. That amount isn't really enough to cover all your expenses unless you are especially frugal, so students still need to go out and get a job (or work in the summer), which I think is perfectly fair.
Every year spent studying you miss out on working at some entry level job for ~25k/y. After 5 years of study, you (in Europe) have accumulated a debt of ~$25k-$50k depending on whether you've had a side job or help from your parents. That means you've lived for ~$10k/y for 5 years, where someone in the workforce lives on significantly more, and at the end you're in debt as well.
Obviously the upside is that you get a job where it is reasonable to expect that you'll be able to repay the loan in about 10 years, but that's something you worked for, and something an ambitious person in the workforce might achieve as well.
Anyway, I don't think you can simply say it's unfair to aid students in their living expenses while they're studying. Obviously they're usually having a great time studying, but they also don't own a car, or a house and instead live in tiny apartments sharing privacy with other students. They usually delay their family building for the entire span of their studies, and often even until they've paid of (most of) their debt. But yeah, have them work some extra in the summer while the workforce is driving their caravans to southern France...
That's pretty advanced stuff for an 18y/o, even for most senior developers. No doubt Google will try to recruit him after he finishes university.
(Edit: I appear to be a broken record) I spent a spell on a bug bounty program. There are some beasts out there in the 16-20yo age range, it's pretty crazy.
Their minds probably haven't been numbed by years of fixing shoddy code and writing CRUD apps yet.
What I appreciated most about it (or maybe this is just his writeup skills), but how not advanced each of the things were. It was just mostly him Googling things and making random guesses at the rabbit hole until he found something.
The skill, I guess, comes in piecing it all together and consistently making good guesses.
The tenacity is as important as the skill. He knew there was a way, he just had to find it.
Selling and getting paid in the black market is probably not very easy unless you already have quite a lot of contacts.
That's not necessarily true, there are brokers that are relatively easy to find. It's more risky at a personal level to go into crime though.
Uruguay is not third world country.
According to Wikipedia, it is a developing country, which is basically the modern way of saying third world.
https://en.wikipedia.org/wiki/Developing_country#Country_lis...
Yes it is. I do not want to be that stickler but this is the charts for that term:
1st world: US, UK, West Germany, essentially western countries
2nd world: Soviets, East Germany, communist countries
3rd world: everyone who doesn’t fit the Cold War theater.
Either way, a GDP per capita of $15k isn’t considered “first” at whatever category you want to assign it by. Unfortunately there are no “1st world” countries in Latin America.
Come on you guys. There's no need to descend into one of these circles of hell.
> I do not want to be that stickler
Really?! You do but you don't but you do want to be a stickler? ....
PS West Germany and East Germany are not separate countries. :)
They were 30 years ago. The term “1st world” and “2nd world” came to existence in the second half of the 20th century.
I’m only mentioning this as people usually attribute it to only economic power.
"X World" has gone he way of "begs the question", "Literally vs. Figuratively", "One bad apple..", etc. Popular usage has destroyed the original meaning, and it's no use trying to get people on board with what those phrases actually mean.
this isn’t the way the OP used the term. they used it to make a statement about a lack of resources.
“studying CS in 3rd world country 35k is a lot of money”
Soviets, East Germany... in 2018?
The terms first, second and third world were coined during the cold war, during which those countries did exist.
First world were countries allied with the US. Second world were countries allied with the USSR. Third world were neutral countries.
So Switzerland, Ireland, and Sweden are third world countries.
While the terms may have been originally coined during the Cold War, the meaning and common usage has clearly changed. If you use 'third world countries' in a conversation hardly anyone will assume that you're including Switzerland in that. When enough people use a word "wrong" for a long enough time, they kinda stop being wrong.
You're getting downvoted, but you're right.
A "third-world country" was a country that wasn't aligned to either the Western countries (first world) or the Communist Bloc (second world).
These terms have become outdated, and "developing" countries is more commonly used now by, e.g., the press.
You really should read a couple of newspapers every now and then, East Germany no longer exists and most communist countries don't either.
This could be related to the fact that while you still hear a lot about "first-world" and "third-world" countries, you don't ever hear anyone use the term "second-world".
This report showcases a ton of tenacity and thoroughness. Not his first time as well: https://sites.google.com/site/testsitehacking/10k-host-heade.... Very impressive.
“Please stop exploring this further, as it seems you could easily break something” has got to be the best reply one can receive to a bug bounty report.
That's what struck me. From the well told story I feel like there were several points of "welp probably can't go further then this better move on to something else" but he kept going and going and sure enough he got somewhere. Hope to see more work published by him over the next few years!
I wish all companies had the internal connectedness to be able to react to realtime updates and coordinate the exact split second to hold up the friendly stop sign!
There have been few stories on here of bounty hunters who've either been scared to go any further than they have, obviously giving the remote end a good excuse to pay less; or who've taken things slightly too far, generating mildly put-out emails and stepped-on toes (which can't bode too well for future interactions).
So this the best thing you can hear from that standpoint as well.
Straightforward communication seems to be a holy grail, which is understandable: besides needing full internal access to all the areas affected by the full scope of vulnerability (either directly or via intermediaries who are good communicators) so you can figure out when to hit the stop button, you also need to have the resources to delicately balance management risk and developer egos.
That sounds like a "fun" job...
---
I guess this demonstrates the fundamental requirement of security: the recipient is the one who needs the non-broken understanding of security, both a) properly in and of itself, and b) how it relates to the domain they're in.
There are a thousand public examples of where this has been gotten hopelessly wrong; my bookmarks are a mess, so here's the (somewhat poorly related) one I can remember how to find via Google (it's as bad as all the others, wouldn't mind being reminded and learning about new ones). This one is not the same kind of vulnerability (there's no need to coordinate pushing a stop button, it's a data breach), it's just an example of grossly misunderstood security and utterly broken internal communication.
- https://arstechnica.com/information-technology/2018/04/paner...
- https://news.ycombinator.com/item?id=16739753 / https://news.ycombinator.com/item?id=16741391 / https://news.ycombinator.com/item?id=16737583
"When issuing the reward, we'll take into account what you could have achieved with this access" makes me laugh.
How scary must that be for the Google team? You know you've messed up so badly and the person who is investigating is doing so blindly with no knowledge or accountability if he breaks something. Yikes.
Kudos to everyone for doing the right things. And great bounty- the average yearly income in Uruguay is $2000-$3000 USD per household. This guy just got awarded more than ten times that.
Are you sure that's accurate? According to this [1] it's about 10 000 USD per capita.
[1] https://www.ceicdata.com/en/indicator/uruguay/annual-househo...
Ah you may be right- I did a cursory Google search only. Still, multiple years of income in one bug isn't bad at all.
I used to work support for GAE and recognize all of this. This is really impressive, congrats on the great work and huge bounty. Keep it up!
Same; I worked on GAE in 2013 and it's so funny to read the story of someone exploring, discovering, and being so close to breaking something you know really well. There's a few moments in here where I thought "oh man, you could have done XXXX and that would have been so bad!". Definitely understand why they gave them the big bucks for this one.
I’ve been on the receiving end where hobbyists were trying (and eventually succeeded) to hack our DRM scheme.
It was really fun to read the forums and see how, day by day, they managed to get closer. Since it wasn’t really crucial IP to begin with, we were rooting for the little guys to see how close they would get, secure in the knowledge that our algorithm was solid. :-)
The final exploit that granted them access was due to a supplier who replaced an earlier validated random generator with something not quite as random, which enabled replay attacks.
I didn't work on GAE, but I know a fair amount about how it ran and, as I read, too, I could think of quite a few things I would have done, had I been a malicious actor. Denial of service is the first to come to mind. I'm sure a real security person could do a lot more.
XXXX == what types of things?
I’m curious why there was no auth required for his calls.
* Grab nearly all of googles source code (no extra auth required for that, since so many libraries read config etc from the source code repo)
* Make the right requests to one endpoint he found and retrieve company financials, number of hits to every google service, the name of every application running in every datacenter, etc.
* With the above two things, you know the location of services and every RPC endpoint on them, and all access control configs. You can take your sweet time to audit the 10's of millions of lines of code to find vulnerabilities and get to attack as an authenticated (albeit low privilege) user. A lot of stuff is open to all authenticated internal users.
* For example, you could take down any google service by quitting all the application servers at the same time by calling the right debugging RPC. You'd be caught obviously tho.
Production code has no unauthenticated access to source. In all the cases I know, there was a data push for dynamic configurations from source to production, usually adding access control, auditing, validation, etc. Static configurations would be baked into the container image.
That said, it looks like in this case he would been able to turn on access to google3, through a special flag that is probably meant for internal GAE apps that work on source code. Presumably, the flag allows the app to use a GAE proxy that authenticates to Piper and provides a filesystem-like interface. It's not clear what would have happened at this point, because it is likely there exists a finer grained quota for source access than just all of GAE. These kinds of new RPC traffic would have been visible in Dapper and Census data, plus there are booby traps everywhere aka defense in depth.
Right, but he wasn’t doing this in a production environment, so I wonder if he could’ve got access to source still.. thoughts?
At the end of the day, even a non-production GAE environment runs on Borg, talks to GFS and Bigtable, etc. They're not going to run the entire stack on bare metal. The environments will have different configurations, will be backed by different Borg jobs, hopefully will run under different Borg identities, etc.
The articles about Piper and co. have made it clear that it ran on top of Bigtable first and then Spanner, i.e. in the production network, spread over ten locations. There's file level auditing and access control. It would be very dumb of them not to have intrusion and anomaly detection in place.
Wow, that is quite significant.
36k is not a small bounty for an RCE, but I feel like this is more critical to Google than the highest Android payout, for which they pay up to 200k for: https://www.google.com/about/appsecurity/android-rewards/
Android is probably one of those markets that are more liquid than most for "black market" sources (as talked about elsewhere in the comments for this)
Android is wormable, and potentially not repairable by google.
For example, with a decent remote android exploit, I could distribute a patched Google Play Services to all vulnerable handsets which disables updates and then listens to my own command and control infrastructure for further actions.
I can now hold the phones hostage and extort google for money to regain control of them.
That would be pretty brutal and cause people to quit trusting android phones.
But I think the same could be accomplished with the access he had, or worse, but would have taken a lot more work. He also would have needed to avoid detection too. His access sounds more troubling than the Aroura attacks they had years ago.
Software architecture isn't like that. Nobody has a list of all the software running in every datacenter -- not in any large corporation anywhere in the world, and an API isn't going to change that.
And, you can't quit all application servers at the same time. These are distributed, self-healing, and highly-redundant -- meaning that there are thousands of copies of each service, and the system will bring up new copies to replace the ones you kill, without loosing service (though you could, potentially, affect quality of service).
Even the control mechanisms that allow all this magic to happen are run on the same platform, meaning that you can't affect the managers in this way either.
The sort of things you can do are: affect billing and reporting, find sensitive company data, and, potentially, execute code remotely (though that will probably be in a container, and not have access to much else).
Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions. The code is huge, so downloading it will take forever, and then you'd have to read it. Finally, it's written in dozens of programming languages, some, like Golang, are unreadable to anyone but experts.
> Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions.
Google is using a monorepo.
Google API Auth/Access is extremely tedious, so I guess someone just didn't bother. Basically you have to get an token, format it, make a couple of http posts, and finally you will have a token to make an access-token-token ... Now image all the steps you would have to make to create a new type of access to some internal API, that should not have public access anyway. Probably saved six months work. And the libraries are probably hiding all the obscurity so no-one did notice until this guy started digging.
Sounds plausible. Now I’m curious how much extra work this finding has caused teams at Google. There are probably many other similar insecure paths that need to be cleaned up.
Another thing that is very admirable and bold is that he had no actual idea that he discovered a RCE vuln but went ahead and confidently contacted google.
How many would stop at "Eh I managed to fire requests to a hidden RPC service in google, but couldn't figure out how to make it do anything useful to qualify".
Put yourselves and your work/findings out there people!
Yes, if he had known more about Google infrastructure, he could have done some damage, both active and passive. He had access to a lot of internals. I was surprised by them at first, too, but they all make sense. On the other hand, he would have probably been caught fairly rapidly.
One thing to note is that Google, like any responsible organization should, has layered security and threat models that include insiders as a potential threat; hopefully, while he could make RPC calls to internal services (which is itself a serious problem, hence the giant bounty) he hopefully could not authenticate to do anything any serious damage or access any sensitive information.
I agree that there are multiple layers of security in place, which is why I said he would have been caught fairly quickly. I'm not going to go into details, but, at the very least, he had access to quite a bit of proprietary information, i.e. private to Google, not necessarily user data. I talked to another former Googler and we agreed that this is at least as bad as a previous disclosure: https://packetstormsecurity.com/files/129406/Google-App-Engi...
He was about 2 API calls from being able to grab nearly all of googles source code from Google3 there...
That's probably not quite true. "google3 file access" most likely means the ability to open paths with the "google3 file library", which lets you use paths like "/gfs/<cluster>/...", and similar for all the different storage services. But you'd also need an identity with permissions to read whatever you're reading. Without an identity that's in ACLs and stuff, you'd only be able to see "world-readable" data, which is still quite serious because who knows what's lying around, but probably nothing sensitive like source code.
What is "Google3"? Incidentally, Google isn't helpful in finding out - all I get as results is garbage and some node.js package placeholders.
It's the Google monorepo: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
Yeah, I wonder if the google3 access may have been the real reason they asked him to stop poking around.
Source? Is there really no authentication around it?
He had read access to G3. Exfiltrating it all would have been hard, but he could have gotten a chunk of it for sure.
You want the part of the write-up where it mentions he gets FILE_GOOGLE3_ACCESS.
I'm not sure what access that actually gives, but that email thread showing people getting +cc'd on the ticket speaks pretty obviously to the severity and magnitude of the problem.
(Am Googler.)
Cute base amount, $31337... :)
For anyone not informed like me...
https://www.urbandictionary.com/define.php?term=31337
Took me a minute to realize why the strange payout number.
This is not an inconsequential amount of cash, for sure. Especially at 18! Congrats! And a great write-up to boot. Just awesome.
All that said an honest question: why would a company like Google not pay insane amounts of money for these kinds of bug finds? What would they pay their own people to find them? Seems like RCE on App Engine should be worth 100K+ and then some on top for giggles just because they can.
Obviously having a standard policy makes sense so that your community understands what to expect but as Google, what's your operational impact if you triple / quadruple vs. market value of the exploits?
They pay their own people a salary - having a dependable living (at a very, very nice rate) is a pretty awesome thing. Sometimes there are bonuses involved, but they're usually not on the order of magnitude of external bug bounties.
My understanding of the pricing is that it's designed to make ethical behavior profitable enough that fewer people are tempted to sell the exploits on the black market, not necessarily to out-compete the black market entirely. I think this person's find is a great example - it's a resume-booster, a great experience, a very nice cash infusion, and helps, instead of hindering, their job prospects with future employers.
Looks like the "Hall of Fame" link in the bounty confirmation email is broken / not rendering:
https://www.google.com/about/appsecurity/hall-of-fame/
It appears to be hosted at https://bughunter.withgoogle.com/rank/hof
(From following this error: "Refused to display 'https://bughunter.withgoogle.com/0x0A?embed=1' in a frame because it set 'X-Frame-Options' to 'deny'.")
Google should send this guy a request to be hired. Clearly he's as good as their internal engineering team and his write up was great.
As has been discussed to death here, Google’s hiring process doesn’t care what you did last week, or last year (eg Max Howell).
This would not improve his odds.
Anecdotal, but I got my job at Google through participating in their bug bounty program. The first set of interviews you have that ask general CS questions might not care what you did last week or last year, but when you talk with the team who wants to hire you they certainly do care.
That's the thing, he might get rejected from the screening questions...
I sure as hell wasnt this competent at 18 let alone now. Kudos to him.
Huh, so you can run binaries in GAE by downloading a statically linked app to /tmp, chmod'ing & executing it? And there would be no limits on how it's run? That's crazy & pretty cool!
GCP Developer Advocate here:
The Java 8, Node.js (just announced at I/O) and GCF environments use a new sandbox that should allow you to run any binary. Of course there are limits, and you have to pay to run things on App Engine past the free tier, but most things should run just fine.
The older GAE sandboxes didn't let you do this (and had a ton of other limitations as well)
It would be no skin of Google’s back to multiply these bug bounties by 10, and they should.
But that's would be counter to their interests. They want to hire this kid when he graduates. If they paid 10x their current bounty rates they'd have paid over $400,000 to him in the last couple of years of his free time. That's a great way to never be able to hire him.
> That's a great way to never be able to hire him.
Why's that? It's not retirement money. 400k (salary+stock) is one year of compensation for some Google engineers.
Dude has cashed out a $10k and $30k bug bounty at the age of 18. Either he's lucky or he's very good. If he's the latter that $400k turns into an annual bounty.
And very, very few Google engineers make that kind of money.
> And very, very few Google engineers make that kind of money.
You'd be surprised. In Mountain View, everyone level 6 or above makes at least that amount, and most level 5s probably do as well. I'd guess that probably 20% of engineers are T5 or above, which is a ton of people when you multiply by tens of thousands of engineers.
Very few as a ratio. There aren't that many T6's and I'm not sold on the idea that "most" T5's are making $400,000 when the average is around $350,000 according to levels.fyi.
I suspect the bug bounties are much more about deterring the sale of exploits to bad actors than about recruiting employees.
It's both. You could do the former by throwing huge sums of money at the problem.
What's your definition of huge?
I don't think it's so easy to sell a vulnerability on the black market. If you send the code first they will have no incentive to send the money, if you get the money, you might not send the code.
You don't have to send the quote, you can demonstrate it, e.g. on a dummy shared account where you bypass all quotas.
I have almost 15 years on the OP and aren't even half as talented. It looks like they've received almost $60k from Google across five bug bounties, very impressive.
Any idea how such rewards get taxed ?
On the receiving end if you're non-US-based? You're supposed to figure it out yourself (ie. pay local applicable income tax), they just want a W8BEN.
Source: got a Google VRP reward.
so US government does not take an cut ?
The US govt doesn't take a cut if you pay an overseas contractor X for Y.
I would expect that in most jurisdictions (well, if they're like Australia) that income is income, and it just goes onto your taxable income.
In Australia, only if you are engaged in a business activity. This means that if you come across some small bug and get a bounty of $1000, then it's probably not even taxable income.
Of course it's taxable. Unless its a gift which, a payout from a company for completing work which the ATO will say is not a gift, then it's income and it'll get added onto your taxable income and tax appropriately. For something like a bug bounty payout, you'll have to declare it yourself as other income.
Mention of #savetheexpanse in article is nice dedication of sorts :)