“I believe that the clue to his mind is to be found in his unusual powers of continuous concentrated
introspection. A case can be made out, as it also can with Descartes, for regarding him as an accomplished
experimentalist. Nothing can be more charming than the tales of his mechanical contrivances when he was a
boy. There are his telescopes and his optical experiments, These were essential accomplishments, part of his
unequalled all-round technique, but not, I am sure, his peculiar gift, especially amongst his contemporaries.
His peculiar gift was the power of holding continuously in his mind a purely mental problem until he had
seen straight through it. I fancy his pre-eminence is due to his muscles of intuition being the strongest and
most enduring with which a man has ever been gifted. Anyone who has ever attempted pure scientific or
philosophical thought knows how one can hold a problem momentarily in one's mind and apply all one's
powers of concentration to piercing through it, and how it will dissolve and escape and you find that what
you are surveying is a blank. I believe that Newton could hold a problem in his mind for hours and days and
weeks until it surrendered to him its secret. Then being a supreme mathematical technician he could dress it
up, how you will, for purposes of exposition, but it was his intuition which was pre-eminently extraordinary
- 'so happy in his conjectures', said De Morgan, 'as to seem to know more than he could possibly have any
means of proving'. The proofs, for what they are worth, were, as I have said, dressed up afterwards - they
were not the instrument of discovery.”
This ability to see -- an ability honed by the product of relentless determination combined with a mind possessed with an almost singular desire to seek and see true -- is a master key found among many of the most penetrating insights. Feynman said it this way:
You have to keep a dozen of your favorite problems
constantly present in your mind, although by and large
they will lay in a dormant state. Every time you hear
or read a new trick or a new result, test it against
each of your twelve problems to see whether it helps.
Every once in a while there will be a hit, and people
will say, “How did he do it? He must be a genius!" [1]
This reminds me of G. Polya's book "How to Solve It" [0], there he talks about the process that teachers can go through to get students to think through problems. But more than that it's trying to build in a habit, so that you tackle problems in the same way each time and can train your brain pathways to hold more of the process in your head.
That's a very good book. I first read it while in college or high school, IIRC. Should buy a copy again, it's worth it.
BTW, for others who might not know, it is not applicable just to math (although Polya was a mathematician). It's good for general problem solving.
And there is a book that was inspired by it, called How to Solve it by Computer. I had and read that one too. It is very good too. See descriptions of both books below.
That is a lot like what Nikola Tesla describes himself doing in his autobiography when inventing the alternating current electric motor – he could, in his mind’s eye, see the motor, turn it off and on and make it go backwards and forwards, and even momentarily forgot that the other people who were with him could not also see it.
EDIT: Quote:
I started by first picturing in my mind a direct-current machine, running it and following the
changing flow of the currents in the armature. Then I would imagine an alternator and investigate
the processes taking place in a similar manner. Next I would visualize systems comprising
motors and generators and operate them in various ways. The images I saw were to me
perfectly real and tangible.
[…]
The images I saw were wonderfully sharp and clear and had the solidity of metal and
stone, so much so that I told him: “See my motor here; watch me reverse it.”
[…]
The
pieces of apparatus I conceived were to me absolutely real and tangible in every
detail, even to
the minute marks and signs of wear. I delighted in imagining the motors constantly running, for in
this way they presented to mind’s eye a more fascinating sight.
I achieved this state once so far in my life, scary and just great at the same time. Back then I basically had the supply chain I was working on running in my head almost 24/7. Not quantitiv but qualitiv, computers are there for the exact numbers. I can just assume I drove most of my colleagues crazy. Still it made work relativly easy and fun, also exhausting.
He's close to how I think about it, but not quite there.
On regularity:
Take the handling of what is false-y. In Ruby only nil and false are false-y. If you use an && the only way it can come out false is if one of the two sides is nil or false. Python, on the other hand, tries to treat emptiness as false-y. It leads to irregularities because assumptions are broken. The clearest example was that the exact second of midnight was considered false-y. Fixed in 3.5 if memory serves.
On restrictiveness:
I agree with PG that a pro working on a project with a small set of other pros prizes unrestrictive languages. On large projects it is usually the opposite. This is why Go is great for Google, but Ruby is great for startups.
On conciseness vs readability:
Regex is concise, but it is shit at readability. The reason conciseness is less important than readability is that conciseness is about the character count while readability is about the time it takes to understand. They're related, but only to a point. Ruby is much more concise than assembly, and that helps its readability, but further gains in conciseness don't really impact speed to understanding.
Regex is fine for a grep of your own filesystem because you're the one writing it so you know you understand it. But for complex form validation it's usually not the first tool I reach for because I tend to assume that the person that comes in behind me might not know more than the basics. Even when I do use them, I break them up as strings into variables that name what they do before concatenating them.
Nontrivial regexes should be treated as their own language. I.e., either the team agrees to use them freely, in which case, team members are obligated to actually learn them. Or, if not, then team members agree not to use them.
If you were working on a Python-only codebase and someone opened a merge request to add a C++ file, you'd smack them down. Complex regexes should be no different.
I disagree. Regexes can can be done in a neat and modular way. I can write a 3 line function IsValidEmailAddress() with a nasty regex in it, and everyone on the team who doesn't know regexes will understand immediately what I'm up to when I call IsValidEmailAddress() and program around it.
The alternative is to write a function hundreds or thousands of lines long that verifies one part of an email string at a time. Of course, that function will be too long and will have lots of duplicated code, so it will be refactored into smaller and more generic functions over and over until I've got a half baked text matching library on my hands. And then I'll have to document the library, and write unit tests for it, and if there is a bug or assumptions change about what to accept as an email, someone will have to read and understand my pages of code. And then someone will want to find email addresses in a file, and there won't be an efficient way to do it.
If it's not trivial to do a pattern match with regexes, it's probably not trivial to do it without. The best you can do is keep the ugly implementation hidden in some well named functions that make your intent obvious.
1) they are abused by people who have never written a parser but need to write a parser, often to parse languages that are highly nonregular, and
2) they get thrown at problems where the underlying grammar is actually unknown and iterated on until they're uneditable black boxes.
The first problem already has a really good solution: hire people who know the standard undergraduate CS material and you'll never end up with a regex-from-hell when a simple and easy recursive descent parser would've done the trick with ease.
The second problem is, of course, more difficult. I've only seen regexes used successfully for a problem like that once. And that's only because the structure was very easy to think about as an NFA. By itself, the regex was a freaking nightmare to look at. But the documentation included a picture of the automata and a great description of what was going on. From there, the monstrous regex was easy to reason about and modify.
hire people who know the standard undergraduate CS material and you'll never end up with a regex-from-hell when a simple and easy recursive descent parser would've done the trick with ease
The number of programmers whom I've personally worked with that actually applied proper parsing principles in their work are exceedingly few--count on one hand, few; too few to make generalizations. But suffice it to say not all had CS degrees, nor even a degree of any kind.
By contrast, most of the people I've worked with professionally have had a CS degree, and regex abuse is rife. I'm sure the reasons are varied and complex (never really grasped it, unable to recognize appropriate situations, can't apply it in larger code bases, or are simply unable to resist easy fixes when under pressure), but the end result is the same.
> But suffice it to say not all had CS degrees, nor even a degree of any kind.
You will notice that I very carefully worded that post. The word degree wasn't used.
Among all the coding quiz questions that get asked in interviews, the parsing-related questions make a ton of sense to me. Being able to bang out a reasonable parser -- and more importantly, knowing which type is needed for which problems -- is an actually useful skill and an even better indicator.
> most of the people I've worked with professionally have had a CS degree, and regex abuse is rife.
"> But suffice it to say not all had CS degrees, nor even a degree of any kind.
You will notice that I very carefully worded that post. The word degree wasn't used."
I sympathize with those shaking their heads at people using regexes to parse XML, for instance. I have fought the battle of trying to get people to do it right. However, in (my version of) the real world, people will give you data that is invalid according to a canonical parser or the language standard, so sometimes you might as well use regexes anyway. A lot of professional programming is more like being an auto mechanic than being an automotive engineer. HN gravitates towards the latter mindset.
> However, in (my version of) the real world, people will give you data that is invalid according to a canonical parser or the language standard, so sometimes you might as well use regexes anyway.
This is very true, and I mention this in my top-level reply:
> Regexes get a bad rap mostly for two reasons: ... 2) they get thrown at problems where the underlying grammar is actually unknown and iterated on until they're uneditable black boxes.
When the (non-fuzzy) version of the input language is regular, then robustifying a regex can be the right way to go. And blaming the person who chose regexes on the basis that they used regexes doesn't make sense if the robustification isn't good enough. The failure to sufficiently robustify is a failure of domain knowledge about that particular fuzzy parsing task, not a failure of training.
However, if a developer is using regexes to parse XML then they're Doing It Wrong (TM), even if the XML files they're parsing are not always valid XML. It's much easier to robustify the right solution to the idealized problem than it is to robustify the wrong solution to the idealized problem.
> A lot of professional programming is more like being an auto mechanic than being an automotive engineer. HN gravitates towards the latter mindset.
That's true. But also, a lot of professional programming is more like being an auto mechanic even though it should be like being an automotive engineer. Including, unfortunately for a hundred deceased Toyota vehicle owners, automotive software engineering.
I agree fully. Regexes get misused and people who don't speak regex blame regex rather than the abusers.
Regexes don't even need to be terse: Perl doesn't have too many fans these days but its implementation of regexes was not only exceptionally advanced but it also allowed you to format regexes across multiple lines like code, and add inline comments to it, which meant you could make them very readable.
I mean sure, you could also add Perl statements that were executed inside the regex, but that's another case of abuse!
> I can write a 3 line function IsValidEmailAddress() with a nasty regex in it, and everyone on the team who doesn't know regexes will understand immediately what I'm up to when I call IsValidEmailAddress() and program around it.
You can do that, sure.
By the way, here is an actual RFC-822 compliant PCRE regex for validating email addresses (with extraneous line breaks; remove these for actual usage):
Hmm. Well. I understand where you are coming from, but I don't think it is that 100% that simple. Sometimes a project is 99.99% regex free, but a regex does the exact right thing.
For example, say you're working on a project where homomorphic confusion between characters would be a security concern so the charset is restricted but in certain contexts, say, Japanese is expected. A simple:
japanese = '([\p{Hiragana}\p{Katakana}]+)'
Is all you really need. Concat it with the rest of the strings for the regex and everyone can understand it.
Adding that kind of thing to a project requires a calm cost-benefit analysis and very likely can not be done without the input of everybody working on the project.
Imagine you need to check if an email address meets RFC822. You can:
a) read the 47 page standard and implement every single requirement, one if statement at a time, and end up writing 10s of thousands of lines of code that are nearly unverifiable.
b) copy that one regex from the standard, put it in a nice function with a comment on where to find the standard, and move on with your life.
Option b will not only save you a week, but also make it much less likely that you screwed up.
Actually, both A and B are incorrect choices. Option A is a complete waste of time. Option B uses a regex that is no longer standards compliant and will tell users that their entirely valid email address is invalid (this is usually a no-no...).
You're skipping entirely over option C:
c) Send the user an email asking them to validate their email.
You'll notice option C is actually the preferred method of almost every company that needs your email to be valid. Option C also does not use a regex at all.
What are you building? Almost every time you don't have to do a, you also do not need full email validation. Besides, the regex is wrong, like I said on my comment, so now what do you do?
But that's a diversion. You can import a library that will do the same just as well as you can import that regex. Both will have comparable levels of gains and problems, but of very different nature.
Of the two, the regexp from the spec should be preferable, because it is exactly what is in the spec. Some other library would be a once-removed version of it and could actually bring flaws of its own.
Yeah, I agree with you regarding email validators. My overarching point was that I don't think it is an all-or-nothing decision to include regexes. I shy away from them for most projects, but sometimes they're necessary.
I think of regexes like I think of code modules that hide abstractions: they should be described via documentation, interfaces, and tests. You generally should not need to read/parse the regex to understand what it does.
I think regular expressions are very useful, but trying to push too much parsing work into them is how you end up with unreadable code.
I wish there was a way to send this to my programmer coworker without seeming like a jerk. He insists we "work" together, meaning, we're in the same physical space, working on two different projects, and he interrupts my train of thought every 2 minutes with an arbitrary question about the thing he's working on. I'm close to a snapping point and don't know how to address it tactfully. What I want to say: "us working together may enhance your productivity by 50% but slows mine down by 80%."
The next time it happens, have a conversation with him about time-boxing. That is, say something like this:
Him: <some question>
You: "Hey, that's an interesting question. Time-box an investigation into that to 20 minutes, and then we'll dig in after that if you're still not sure about the solution."
If he asks what time-boxing is, then you can have the discussion about actively investigating a solution for a specific amount of time. If, at the end of that time, you don't have a solid lead on a solution, then you reach out to your peers. That means, you have input to answer questions if your peers have any, and you know some simple solutions that don't work, and why.
That's what I've done with jr engineers for a number of years now and it works great. You have to do it a couple of times before they turn it into a habit, and internalize it, but eventually they bother you less and become better engineers in the process.
Thanks. Admittedly, I've been giving positive re-enforcement to his questions by answering them immediately. Which I realize is doing us both a disservice. It's now easier for him to ask me than to spend time investigating, and it's easier for me to answer quickly to get back to my work. But it's just making the problem worse. It's become programming by proxy.
Just be honest with him. Tell him you want to help but at the same time, when you break off you are losing track and it's making it hard to do deep work. Tell him you want him to get the help he needs but you also need to complete your own work, and open up a dialog for ways you can both strike a balance.
If he's asking questions out of laziness, using you as a replacement for Google, next time he asks tell him to do some investigating and that you can't always give him the easy answer. It's a key programmer skill.
Do you work with my coworker? Does he make little noises (sighs, hmmmms, Ohhhhs) when he wants to talk to you? Does he make the noises more frequently and louder when you ignore him? Does he stare at you, trying to catch your eye while you studiously stare at your monitor?
You could try a bit of roleplay with your friends, script out some different ways to get the point across and so on. The most important thing is that you talk to this person sooner rather than later, when you’ll be unable to control your emotions because you’re ready to snap. Be polite, be direct, be firm, and then stick by it.
Could you elaborate a little further in this context? I don't want to ostracize because, in the end, we're both working towards the same goals. It's just our methods are becoming unhealthy. I'm looking to make them more healthy.
What I mean is that it's perfectly fine to politely but firmly reject people from activities you take part in with no special reason given. It isn't "being a jerk" to just tell someone you don't want to pair program with them, and you don't need an excuse for it.
I think being able to be productive without having the whole thing in your head is one of the important skills of working at BigCo. It's definitely more awkward and I'm sure leads to an increased number of bugs, but it lets shifting teams and priorities move forward. Some people struggle a lot with this though, especially if they've only ever worked on their own programs or programs designed in a small and tight group (e.g school projects). It's definitely a skill.
After an intern experience I changed up one of my phone screen sections -- we have to do a phone screen (I modeled my favorite variation on Yegge's), I try to change it up from time to time though to make it somewhat interesting for both people, test different ideas of what questions produce what signals and what signals are valuable, and at least never make it a whiteboard or accurately-code-well-known-freshmen-problems-like-sorting-algos hazing. I used to reserve a section for discussing program design in a hypothetical board game program, I took that out to instead give the person a description of a simple web app with an accompanying ~100 LoC describing the architecture and component hooks (simplified and mocked a bit from our actual code, it has a startup that creates top level components like the PageBody component and the Header component, the Header component creates a Navigation component and whatever else, the Navigation component creates navigation elements...) and a problem statement that involves threading some new Component data from the root to a leaf using the existing framework in order to make a proposed code change work. So it's not that hard objectively, it's mainly adding a new function param in a couple component initializing functions and updating call sites, but it requires being able to read a chunk of code, ignoring most of it, but understanding just enough of it in context to make the change. The new intern who got through this (took ~20-25 mins) ended up being pretty solid, and I think the test would have negatively differentiated the prior intern with other candidates we had thought of as about equivalent...
I'm a talented programmer but wasn't always so. The most important thing I ever learned was how to plan a program into abstract components that are as isolated as possible BEFORE ever touching any code.
If you plan an abstract hierarchy correctly you don't have to try and keep the entire swarm in your head. You can focus on the individual components and when they are done, forget about the implementation and refocus on the next level in the hierarchy. If you run into issues step back out into the planning phase. Never write code without a plan.
When I was younger I could barely get 1k lines of code down without it being a mess. Now I can build complex applications with multiple libraries in the 20k-40k SLOC range without much mental effort. It is almost like entering and leaving functions in your brain.
I think that's what he was referring to by "bottom-up programming". Essentially, make your abstractions air-tight and grok-able and you never have to worry about what the abstraction contains again.
Better put you build the components with the least dependencies first after your software architecture is decided, up to and including pseudo code, to the point you are confident in the design and are not required to think about it while implementing it in real code. If a design issue comes up stop programming and fix the architecture. I'm a graph paper guy myself.
Say a racetrack library. We have a 'racetrack', it has a 'list' of 'cars', they have a 'list' of 'tires', and 'list' is a custom class and we know how it is all going to fit.
Now normally you might start with a 'racetrack' class, start a car class, start the list class so you can have the cars on the racetrack, start the tires class. Add functionality to the list class to get something to work. You leave a trail behind you that if you try to keep in your head can get pretty awful in big projects.
If you start with the objects with least dependencies you can forget about them once you are done. 'list' has no dependencies, you can make the class (and all test code if desired) and forget about it. Now you can focus entirely on a tire (and all test code if desired), and forget about it. Then you do the car class fully, then the racetrack. At every level you have everything below it done so you can fully complete the implementation in one shot instead of bouncing all over the place . It isn't always perfect but it really cuts down on the brainpower you have to expend.
This is best suited with unit testing. At first it can feel ungratifying to program this way, even scary. You could be 85% done with your planned architecture and have almost no visible behavior of your final product, with that last 15% being the actual high level abstractions that really tie everything together and the project just seems to pop into completion.
A couple months ago I quit my software job to incubate a startup idea. I’m working out of home now and this definitely rings true. I’ll add a couple additional thoughts. One is that “whole program” thinking is tiring, and I can only muster about 6 hours a day. After that, my brain resists thinking deeply until the next day. Naps help somewhat, but I’ve learned to just switch gears and either enjoy some entertainment, or work on less demanding tasks. I keep a queue of lite tasks now for this purpose. Another discovery is how helpful a warm shower is to creative thought (which includes refactoring IMO). It sounds rediculous, but when I’m working on a tough problem now, I may take 3 or more showers a day. And it works! Like magic, I don’t think I’ve ever not come out of a shower with the problem solved. There’s nobody to judge me :) But if I do create a company someday, I will have onsite personal showers and encourage their use.
I find nearly anything that gets me away from the screen can help with creativity. For me, gardening is one of the best.
An interesting property of the shower is that it’s one of the few places where hardly anyone takes electronics. Even if you’re someone who reaches for a smartphone without a second thought while walking, the chances are you put it down before showering (the garden has a similar property —- I don’t want to handle electronics with muddy hands or gloves...)
> Oddly enough, scheduled distractions may be worse than unscheduled ones. If you know you have a meeting in an hour, you don't even start working on something hard.
This hit a little too close to home, since I'm procrastinating working on something right now because lunch is scheduled to be in ten minutes and won't be able to start anything meaningful…
One thing that this list of prescriptions would probably contain if this were written today is: write tests. This lets you make a lot of classes of change without actually having to hold the whole thing in your head. You can make an educated guess as to what needs to change, write a test for it, and confirm that you made the right change. More importantly, if the tests you wrote months ago are complete enough, you can be confident that your recent changes didn't fuck up anything up that you didn't intend.
Depends on how you write the tests. If your tests are very low-level, making cross-cutting changes can be extremely painful, to the point that you'll avoid doing them b/c of the cost of fixing the tests.
Unfortunately, such low-level tests are becoming increasingly common as managers and engineers focus on optimizing metrics like code coverage, and those kinds of tests are very easy to write to improve code coverage.
As a manager, I haven't bothered to look at code coverage for years.
What I care about is functional coverage for the boundaries and exception cases of our use cases or APIs. Run them CD in your pipeline.
Cover customer expectations at the highest level of reasonable abstraction (I also tend to avoid ui testing, and frontend JavaScript too).
Low level tests or unit tests only matter insofar as they make it easier to trust your dependecies. Breaking abstraction or encapsulation to test is a hard stop NO.
That is also how I work when I am at my most effective. As a Lisp programmer, I especially like: “You can magnify the effect of a powerful language by using a style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above. If you do this right, you only have to keep the topmost layer in your head.”
I work for a company right now in an office and the only way I get uninterrupted time slots is to either start work very early in the morning or occasionally go home for two hours when I need quiet heads down work time. When I work at home as a remote consultant I usually do 4 to 5 hour sprints and my wife and my pet parrot usually don’t interrupt me. A five hour sprint starting early in the morning followed by four hours of exercise and relaxation, then another hour or two of work is just about perfect for me.
This is certainly true when solving hard problems.
A counterpoint would be that most software used to run businesses is complicated, but not hard (or the hard parts are very localized.) And for these problems it's best to factor your codebase such that it is not required to hold the whole thing in your head.
Because if you do, at some point in the project's history, someone is going to blunder in and make a change that fractures your crystal palace without you even realizing it.
> You can magnify the effect of a powerful language by using a
style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above. If you do this right, you only have to keep the topmost layer in your head.
This is pretty much how you should write programs regardless of the programming language. Otherwise the program would become virtually impossible to understand and maintain as it grows.
I think the fundamental is the description of software as an idea.
It is very important to understand, because it helps explains that the essence/idea part of software writing will never be automated and that any repetitive part in software must be automated to make progress.
This is why the concept of software sweat shops/and some bad flavors of pair coding/agile development are stupid, it's all built around systematizing what should have been expressed in higher-order software anyways. They're only effective at still making money for the shop while the market hasn't caught up.
This is wrong or at least worded incorrectly IMHO. I believe the author knew it wasn't quite right when they wrote it, since the second paragraph is an attempt to qualify it away.
The rule would perhaps be better stated as "Think in a distraction free environment".
Plenty of productive programming is done at rest, and long stretches of "work" produce subpar code.
I do think working in long stretches is effective, but I'm not talking about a single "butt-in-seat" marathon coding session. I think of it more as a focus on solely one project for a number of days, without multi-tasking on 3+ projects at once giving each one only a couple hours per day.
The processing that you do when not sitting at a desk, and those eureka moments that come along with it happen more often when you are working one thing at a time. Or, at least they do for me.
Yep. The tricky thing is that "help a colleague on some little problem" can end up being an expensive context switch, but can also be pretty valuable. A degree of asynchrony, and perhaps some kind of "office hours" type mechanism to handle questions in batch mode seem like good solutions. But that goes against processes which treat "resolving blockers" as top priority.
As someone who does this with a lot of programs I've written, I'll say that avoiding abstractions is just as important as distractions --- if doing the simplest of tasks takes half a dozen layers of abstraction and indirection, it makes it all the more difficult to see the whole program at once. "Too many moving pieces makes it hard to see what's going on", as the saying goes.
The amount of "fluff" introduced by "modern" languages and their culture, like Java and C#, is certainly an obstacle to this; it seems like they encourage overcomplexity and the creation of huge complex software, so that all programs written in them naturally take on that style, no matter how trivial they actually should/need to be.
The languages themselves sort of forces people's hands depending on how rigid they are and how easy it is to change your mind afterwards.
Take Python for example. Firstly they have a culture of "we're all adults here" and don't really bother with public/private distinctions in classes, and secondly the language makes it easy to turn a variable into a property with the @property decorator.
I'm not that familiar with Java or C#, but I guess turning a variable into a property after the fact is non-trivial. In Python all the calling code remains unchanged (`val = object.field`)
I manage to keep my main side project (~20,000 lines of code) in my head despite not meeting conditions #1 (I'm always distracted), #2 (I work in my free time at home), and #3 (C++, woo) because I have extensive documentation about the why of code, and this helps me put myself in the same state of mind I was in when I wrote it.
It does help that all the others apply to me, though. I've rewritten it kinda twice, I already mentioned my comments, I work basically alone on it, so nobody else edits anything, and it started out very small.
That's a wonderful article describing the work of a programmer, why bigger companies cause them trouble and how good programmers reason about code which is not even yet written. Paul Graham, which is a notable figure in the software engineering world if you didn't hear his name yet, names 8 steps that make you able to load a whole program into your head. I'm proud to say that I have mastered all 8 steps multiple times, though from my experience in my current engineering team I struggle with step 7. I do actually believe that what he describes shouldn't be done, is actually feasible - through pair programming.
I'll leave this here
“I believe that the clue to his mind is to be found in his unusual powers of continuous concentrated introspection. A case can be made out, as it also can with Descartes, for regarding him as an accomplished experimentalist. Nothing can be more charming than the tales of his mechanical contrivances when he was a boy. There are his telescopes and his optical experiments, These were essential accomplishments, part of his unequalled all-round technique, but not, I am sure, his peculiar gift, especially amongst his contemporaries. His peculiar gift was the power of holding continuously in his mind a purely mental problem until he had seen straight through it. I fancy his pre-eminence is due to his muscles of intuition being the strongest and most enduring with which a man has ever been gifted. Anyone who has ever attempted pure scientific or philosophical thought knows how one can hold a problem momentarily in one's mind and apply all one's powers of concentration to piercing through it, and how it will dissolve and escape and you find that what you are surveying is a blank. I believe that Newton could hold a problem in his mind for hours and days and weeks until it surrendered to him its secret. Then being a supreme mathematical technician he could dress it up, how you will, for purposes of exposition, but it was his intuition which was pre-eminently extraordinary - 'so happy in his conjectures', said De Morgan, 'as to seem to know more than he could possibly have any means of proving'. The proofs, for what they are worth, were, as I have said, dressed up afterwards - they were not the instrument of discovery.”
― John Maynard Keynes
This ability to see -- an ability honed by the product of relentless determination combined with a mind possessed with an almost singular desire to seek and see true -- is a master key found among many of the most penetrating insights. Feynman said it this way:
[1] http://alumni.media.mit.edu/~cahn/life/gian-carlo-rota-10-le...[1] http://wiki.c2.com/?FeynmanAlgorithm
This reminds me of G. Polya's book "How to Solve It" [0], there he talks about the process that teachers can go through to get students to think through problems. But more than that it's trying to build in a habit, so that you tackle problems in the same way each time and can train your brain pathways to hold more of the process in your head.
[0]: https://www.math.utah.edu/~alfeld/math/polya.html
That's a very good book. I first read it while in college or high school, IIRC. Should buy a copy again, it's worth it.
BTW, for others who might not know, it is not applicable just to math (although Polya was a mathematician). It's good for general problem solving.
And there is a book that was inspired by it, called How to Solve it by Computer. I had and read that one too. It is very good too. See descriptions of both books below.
https://en.wikipedia.org/wiki/How_to_Solve_It
https://en.wikipedia.org/wiki/How_to_Solve_It#Influence
https://en.wikipedia.org/wiki/How_to_Solve_it_by_Computer
That is a lot like what Nikola Tesla describes himself doing in his autobiography when inventing the alternating current electric motor – he could, in his mind’s eye, see the motor, turn it off and on and make it go backwards and forwards, and even momentarily forgot that the other people who were with him could not also see it.
EDIT: Quote:
I started by first picturing in my mind a direct-current machine, running it and following the changing flow of the currents in the armature. Then I would imagine an alternator and investigate the processes taking place in a similar manner. Next I would visualize systems comprising motors and generators and operate them in various ways. The images I saw were to me perfectly real and tangible.
[…]
The images I saw were wonderfully sharp and clear and had the solidity of metal and stone, so much so that I told him: “See my motor here; watch me reverse it.”
[…]
The pieces of apparatus I conceived were to me absolutely real and tangible in every detail, even to the minute marks and signs of wear. I delighted in imagining the motors constantly running, for in this way they presented to mind’s eye a more fascinating sight.
I achieved this state once so far in my life, scary and just great at the same time. Back then I basically had the supply chain I was working on running in my head almost 24/7. Not quantitiv but qualitiv, computers are there for the exact numbers. I can just assume I drove most of my colleagues crazy. Still it made work relativly easy and fun, also exhausting.
Not nearly so grand but I once read a quote describing the fragility of holding state in one's head as "juggling priceless eggs in variable gravity."
That quotation is actually from Jerry Pournelle and is uttered by one of the Moties in the book "The Mote in God's Eye". In full:
“We juggle priceless eggs in variable gravity. I am afraid. I will taste fear until I die.”
The book is well-known to many programmers and the quotation is very likely to have been lifted and re-purposed.
He's close to how I think about it, but not quite there.
On regularity:
Take the handling of what is false-y. In Ruby only nil and false are false-y. If you use an && the only way it can come out false is if one of the two sides is nil or false. Python, on the other hand, tries to treat emptiness as false-y. It leads to irregularities because assumptions are broken. The clearest example was that the exact second of midnight was considered false-y. Fixed in 3.5 if memory serves.
On restrictiveness:
I agree with PG that a pro working on a project with a small set of other pros prizes unrestrictive languages. On large projects it is usually the opposite. This is why Go is great for Google, but Ruby is great for startups.
On conciseness vs readability:
Regex is concise, but it is shit at readability. The reason conciseness is less important than readability is that conciseness is about the character count while readability is about the time it takes to understand. They're related, but only to a point. Ruby is much more concise than assembly, and that helps its readability, but further gains in conciseness don't really impact speed to understanding.
Regex is fine for a grep of your own filesystem because you're the one writing it so you know you understand it. But for complex form validation it's usually not the first tool I reach for because I tend to assume that the person that comes in behind me might not know more than the basics. Even when I do use them, I break them up as strings into variables that name what they do before concatenating them.
Nontrivial regexes should be treated as their own language. I.e., either the team agrees to use them freely, in which case, team members are obligated to actually learn them. Or, if not, then team members agree not to use them.
If you were working on a Python-only codebase and someone opened a merge request to add a C++ file, you'd smack them down. Complex regexes should be no different.
I disagree. Regexes can can be done in a neat and modular way. I can write a 3 line function IsValidEmailAddress() with a nasty regex in it, and everyone on the team who doesn't know regexes will understand immediately what I'm up to when I call IsValidEmailAddress() and program around it.
The alternative is to write a function hundreds or thousands of lines long that verifies one part of an email string at a time. Of course, that function will be too long and will have lots of duplicated code, so it will be refactored into smaller and more generic functions over and over until I've got a half baked text matching library on my hands. And then I'll have to document the library, and write unit tests for it, and if there is a bug or assumptions change about what to accept as an email, someone will have to read and understand my pages of code. And then someone will want to find email addresses in a file, and there won't be an efficient way to do it.
If it's not trivial to do a pattern match with regexes, it's probably not trivial to do it without. The best you can do is keep the ugly implementation hidden in some well named functions that make your intent obvious.
Regexes get a bad rap mostly for two reasons:
1) they are abused by people who have never written a parser but need to write a parser, often to parse languages that are highly nonregular, and
2) they get thrown at problems where the underlying grammar is actually unknown and iterated on until they're uneditable black boxes.
The first problem already has a really good solution: hire people who know the standard undergraduate CS material and you'll never end up with a regex-from-hell when a simple and easy recursive descent parser would've done the trick with ease.
The second problem is, of course, more difficult. I've only seen regexes used successfully for a problem like that once. And that's only because the structure was very easy to think about as an NFA. By itself, the regex was a freaking nightmare to look at. But the documentation included a picture of the automata and a great description of what was going on. From there, the monstrous regex was easy to reason about and modify.
By contrast, most of the people I've worked with professionally have had a CS degree, and regex abuse is rife. I'm sure the reasons are varied and complex (never really grasped it, unable to recognize appropriate situations, can't apply it in larger code bases, or are simply unable to resist easy fixes when under pressure), but the end result is the same.
> But suffice it to say not all had CS degrees, nor even a degree of any kind.
You will notice that I very carefully worded that post. The word degree wasn't used.
Among all the coding quiz questions that get asked in interviews, the parsing-related questions make a ton of sense to me. Being able to bang out a reasonable parser -- and more importantly, knowing which type is needed for which problems -- is an actually useful skill and an even better indicator.
> most of the people I've worked with professionally have had a CS degree, and regex abuse is rife.
IME this can be shortened to simply:
> regex abuse is rife.
"> But suffice it to say not all had CS degrees, nor even a degree of any kind.
You will notice that I very carefully worded that post. The word degree wasn't used."
I sympathize with those shaking their heads at people using regexes to parse XML, for instance. I have fought the battle of trying to get people to do it right. However, in (my version of) the real world, people will give you data that is invalid according to a canonical parser or the language standard, so sometimes you might as well use regexes anyway. A lot of professional programming is more like being an auto mechanic than being an automotive engineer. HN gravitates towards the latter mindset.
> However, in (my version of) the real world, people will give you data that is invalid according to a canonical parser or the language standard, so sometimes you might as well use regexes anyway.
This is very true, and I mention this in my top-level reply:
> Regexes get a bad rap mostly for two reasons: ... 2) they get thrown at problems where the underlying grammar is actually unknown and iterated on until they're uneditable black boxes.
When the (non-fuzzy) version of the input language is regular, then robustifying a regex can be the right way to go. And blaming the person who chose regexes on the basis that they used regexes doesn't make sense if the robustification isn't good enough. The failure to sufficiently robustify is a failure of domain knowledge about that particular fuzzy parsing task, not a failure of training.
However, if a developer is using regexes to parse XML then they're Doing It Wrong (TM), even if the XML files they're parsing are not always valid XML. It's much easier to robustify the right solution to the idealized problem than it is to robustify the wrong solution to the idealized problem.
> A lot of professional programming is more like being an auto mechanic than being an automotive engineer. HN gravitates towards the latter mindset.
That's true. But also, a lot of professional programming is more like being an auto mechanic even though it should be like being an automotive engineer. Including, unfortunately for a hundred deceased Toyota vehicle owners, automotive software engineering.
I agree fully. Regexes get misused and people who don't speak regex blame regex rather than the abusers.
Regexes don't even need to be terse: Perl doesn't have too many fans these days but its implementation of regexes was not only exceptionally advanced but it also allowed you to format regexes across multiple lines like code, and add inline comments to it, which meant you could make them very readable.
I mean sure, you could also add Perl statements that were executed inside the regex, but that's another case of abuse!
Your hidden away nasty regex that no one else understand works until it needs to be modified.
Then it's better to have the 15 lines of regular code that regular programmers can modify.
> I can write a 3 line function IsValidEmailAddress() with a nasty regex in it, and everyone on the team who doesn't know regexes will understand immediately what I'm up to when I call IsValidEmailAddress() and program around it.
You can do that, sure.
By the way, here is an actual RFC-822 compliant PCRE regex for validating email addresses (with extraneous line breaks; remove these for actual usage):
(per http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html)Sigh. Look at that thing. Regexes are write-only!
Hmm. Well. I understand where you are coming from, but I don't think it is that 100% that simple. Sometimes a project is 99.99% regex free, but a regex does the exact right thing.
For example, say you're working on a project where homomorphic confusion between characters would be a security concern so the charset is restricted but in certain contexts, say, Japanese is expected. A simple:
japanese = '([\p{Hiragana}\p{Katakana}]+)'
Is all you really need. Concat it with the rest of the strings for the regex and everyone can understand it.
This is one complex regex for you to get the context (it doesn't work anymore, as email validation changed since then):
http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html
Adding that kind of thing to a project requires a calm cost-benefit analysis and very likely can not be done without the input of everybody working on the project.
Imagine you need to check if an email address meets RFC822. You can:
a) read the 47 page standard and implement every single requirement, one if statement at a time, and end up writing 10s of thousands of lines of code that are nearly unverifiable.
b) copy that one regex from the standard, put it in a nice function with a comment on where to find the standard, and move on with your life.
Option b will not only save you a week, but also make it much less likely that you screwed up.
Actually, both A and B are incorrect choices. Option A is a complete waste of time. Option B uses a regex that is no longer standards compliant and will tell users that their entirely valid email address is invalid (this is usually a no-no...).
You're skipping entirely over option C:
c) Send the user an email asking them to validate their email.
You'll notice option C is actually the preferred method of almost every company that needs your email to be valid. Option C also does not use a regex at all.
What are you building? Almost every time you don't have to do a, you also do not need full email validation. Besides, the regex is wrong, like I said on my comment, so now what do you do?
But that's a diversion. You can import a library that will do the same just as well as you can import that regex. Both will have comparable levels of gains and problems, but of very different nature.
Of the two, the regexp from the spec should be preferable, because it is exactly what is in the spec. Some other library would be a once-removed version of it and could actually bring flaws of its own.
Yeah, I agree with you regarding email validators. My overarching point was that I don't think it is an all-or-nothing decision to include regexes. I shy away from them for most projects, but sometimes they're necessary.
Complex regexes should use the 'x' flag so they can be multiline with comments.
I think of regexes like I think of code modules that hide abstractions: they should be described via documentation, interfaces, and tests. You generally should not need to read/parse the regex to understand what it does.
I think regular expressions are very useful, but trying to push too much parsing work into them is how you end up with unreadable code.
Luckily, regexes are extremely easy to unit test.
I wish there was a way to send this to my programmer coworker without seeming like a jerk. He insists we "work" together, meaning, we're in the same physical space, working on two different projects, and he interrupts my train of thought every 2 minutes with an arbitrary question about the thing he's working on. I'm close to a snapping point and don't know how to address it tactfully. What I want to say: "us working together may enhance your productivity by 50% but slows mine down by 80%."
The next time it happens, have a conversation with him about time-boxing. That is, say something like this:
Him: <some question> You: "Hey, that's an interesting question. Time-box an investigation into that to 20 minutes, and then we'll dig in after that if you're still not sure about the solution."
If he asks what time-boxing is, then you can have the discussion about actively investigating a solution for a specific amount of time. If, at the end of that time, you don't have a solid lead on a solution, then you reach out to your peers. That means, you have input to answer questions if your peers have any, and you know some simple solutions that don't work, and why.
That's what I've done with jr engineers for a number of years now and it works great. You have to do it a couple of times before they turn it into a habit, and internalize it, but eventually they bother you less and become better engineers in the process.
Thanks. Admittedly, I've been giving positive re-enforcement to his questions by answering them immediately. Which I realize is doing us both a disservice. It's now easier for him to ask me than to spend time investigating, and it's easier for me to answer quickly to get back to my work. But it's just making the problem worse. It's become programming by proxy.
Just be honest with him. Tell him you want to help but at the same time, when you break off you are losing track and it's making it hard to do deep work. Tell him you want him to get the help he needs but you also need to complete your own work, and open up a dialog for ways you can both strike a balance.
If he's asking questions out of laziness, using you as a replacement for Google, next time he asks tell him to do some investigating and that you can't always give him the easy answer. It's a key programmer skill.
Seems like it's worth having that conversation before you hold more resentment.
Lots of options.
1. Talk to him about it.
2. Make it more difficult for him to talk to you (put headphones on)
3. Decline working in the same space as him regularly because "I really need to concentrate today".
Do you work with my coworker? Does he make little noises (sighs, hmmmms, Ohhhhs) when he wants to talk to you? Does he make the noises more frequently and louder when you ignore him? Does he stare at you, trying to catch your eye while you studiously stare at your monitor?
Ah yes, the staring. People being polite not to disturb you so they sit next to you staring at you.
You could try a bit of roleplay with your friends, script out some different ways to get the point across and so on. The most important thing is that you talk to this person sooner rather than later, when you’ll be unable to control your emotions because you’re ready to snap. Be polite, be direct, be firm, and then stick by it.
You seem to have internalized Geek Fallacy #1. https://pkeros.wordpress.com/2014/02/21/on-the-five-geek-soc...
Could you elaborate a little further in this context? I don't want to ostracize because, in the end, we're both working towards the same goals. It's just our methods are becoming unhealthy. I'm looking to make them more healthy.
What I mean is that it's perfectly fine to politely but firmly reject people from activities you take part in with no special reason given. It isn't "being a jerk" to just tell someone you don't want to pair program with them, and you don't need an excuse for it.
I think being able to be productive without having the whole thing in your head is one of the important skills of working at BigCo. It's definitely more awkward and I'm sure leads to an increased number of bugs, but it lets shifting teams and priorities move forward. Some people struggle a lot with this though, especially if they've only ever worked on their own programs or programs designed in a small and tight group (e.g school projects). It's definitely a skill.
After an intern experience I changed up one of my phone screen sections -- we have to do a phone screen (I modeled my favorite variation on Yegge's), I try to change it up from time to time though to make it somewhat interesting for both people, test different ideas of what questions produce what signals and what signals are valuable, and at least never make it a whiteboard or accurately-code-well-known-freshmen-problems-like-sorting-algos hazing. I used to reserve a section for discussing program design in a hypothetical board game program, I took that out to instead give the person a description of a simple web app with an accompanying ~100 LoC describing the architecture and component hooks (simplified and mocked a bit from our actual code, it has a startup that creates top level components like the PageBody component and the Header component, the Header component creates a Navigation component and whatever else, the Navigation component creates navigation elements...) and a problem statement that involves threading some new Component data from the root to a leaf using the existing framework in order to make a proposed code change work. So it's not that hard objectively, it's mainly adding a new function param in a couple component initializing functions and updating call sites, but it requires being able to read a chunk of code, ignoring most of it, but understanding just enough of it in context to make the change. The new intern who got through this (took ~20-25 mins) ended up being pretty solid, and I think the test would have negatively differentiated the prior intern with other candidates we had thought of as about equivalent...
I'm a talented programmer but wasn't always so. The most important thing I ever learned was how to plan a program into abstract components that are as isolated as possible BEFORE ever touching any code.
If you plan an abstract hierarchy correctly you don't have to try and keep the entire swarm in your head. You can focus on the individual components and when they are done, forget about the implementation and refocus on the next level in the hierarchy. If you run into issues step back out into the planning phase. Never write code without a plan.
When I was younger I could barely get 1k lines of code down without it being a mess. Now I can build complex applications with multiple libraries in the 20k-40k SLOC range without much mental effort. It is almost like entering and leaving functions in your brain.
I think that's what he was referring to by "bottom-up programming". Essentially, make your abstractions air-tight and grok-able and you never have to worry about what the abstraction contains again.
Do you write the plan down? Like an outline? I want to get better at what you are describing.
Better put you build the components with the least dependencies first after your software architecture is decided, up to and including pseudo code, to the point you are confident in the design and are not required to think about it while implementing it in real code. If a design issue comes up stop programming and fix the architecture. I'm a graph paper guy myself.
Say a racetrack library. We have a 'racetrack', it has a 'list' of 'cars', they have a 'list' of 'tires', and 'list' is a custom class and we know how it is all going to fit.
Now normally you might start with a 'racetrack' class, start a car class, start the list class so you can have the cars on the racetrack, start the tires class. Add functionality to the list class to get something to work. You leave a trail behind you that if you try to keep in your head can get pretty awful in big projects.
If you start with the objects with least dependencies you can forget about them once you are done. 'list' has no dependencies, you can make the class (and all test code if desired) and forget about it. Now you can focus entirely on a tire (and all test code if desired), and forget about it. Then you do the car class fully, then the racetrack. At every level you have everything below it done so you can fully complete the implementation in one shot instead of bouncing all over the place . It isn't always perfect but it really cuts down on the brainpower you have to expend.
This is best suited with unit testing. At first it can feel ungratifying to program this way, even scary. You could be 85% done with your planned architecture and have almost no visible behavior of your final product, with that last 15% being the actual high level abstractions that really tie everything together and the project just seems to pop into completion.
A couple months ago I quit my software job to incubate a startup idea. I’m working out of home now and this definitely rings true. I’ll add a couple additional thoughts. One is that “whole program” thinking is tiring, and I can only muster about 6 hours a day. After that, my brain resists thinking deeply until the next day. Naps help somewhat, but I’ve learned to just switch gears and either enjoy some entertainment, or work on less demanding tasks. I keep a queue of lite tasks now for this purpose. Another discovery is how helpful a warm shower is to creative thought (which includes refactoring IMO). It sounds rediculous, but when I’m working on a tough problem now, I may take 3 or more showers a day. And it works! Like magic, I don’t think I’ve ever not come out of a shower with the problem solved. There’s nobody to judge me :) But if I do create a company someday, I will have onsite personal showers and encourage their use.
I find nearly anything that gets me away from the screen can help with creativity. For me, gardening is one of the best.
An interesting property of the shower is that it’s one of the few places where hardly anyone takes electronics. Even if you’re someone who reaches for a smartphone without a second thought while walking, the chances are you put it down before showering (the garden has a similar property —- I don’t want to handle electronics with muddy hands or gloves...)
> An interesting property of the shower is that it’s one of the few places where hardly anyone takes electronics.
I've been finding plane flights are good for that kind of unstructured free association problem solving too, for precisely that reason.
What I'd like for regular thinking is a spa. Whenever I'm fortunate enough to actually own a house, I'm going to put in a spa.
LOL my best ideas come in the shower too, but it's never occurred to me to take _extra_ showers in order to solve more problems. :-D
> Oddly enough, scheduled distractions may be worse than unscheduled ones. If you know you have a meeting in an hour, you don't even start working on something hard.
This hit a little too close to home, since I'm procrastinating working on something right now because lunch is scheduled to be in ten minutes and won't be able to start anything meaningful…
See also: http://paulgraham.com/makersschedule.html
One thing that this list of prescriptions would probably contain if this were written today is: write tests. This lets you make a lot of classes of change without actually having to hold the whole thing in your head. You can make an educated guess as to what needs to change, write a test for it, and confirm that you made the right change. More importantly, if the tests you wrote months ago are complete enough, you can be confident that your recent changes didn't fuck up anything up that you didn't intend.
Depends on how you write the tests. If your tests are very low-level, making cross-cutting changes can be extremely painful, to the point that you'll avoid doing them b/c of the cost of fixing the tests.
Unfortunately, such low-level tests are becoming increasingly common as managers and engineers focus on optimizing metrics like code coverage, and those kinds of tests are very easy to write to improve code coverage.
As a manager, I haven't bothered to look at code coverage for years.
What I care about is functional coverage for the boundaries and exception cases of our use cases or APIs. Run them CD in your pipeline.
Cover customer expectations at the highest level of reasonable abstraction (I also tend to avoid ui testing, and frontend JavaScript too).
Low level tests or unit tests only matter insofar as they make it easier to trust your dependecies. Breaking abstraction or encapsulation to test is a hard stop NO.
That is also how I work when I am at my most effective. As a Lisp programmer, I especially like: “You can magnify the effect of a powerful language by using a style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above. If you do this right, you only have to keep the topmost layer in your head.”
I work for a company right now in an office and the only way I get uninterrupted time slots is to either start work very early in the morning or occasionally go home for two hours when I need quiet heads down work time. When I work at home as a remote consultant I usually do 4 to 5 hour sprints and my wife and my pet parrot usually don’t interrupt me. A five hour sprint starting early in the morning followed by four hours of exercise and relaxation, then another hour or two of work is just about perfect for me.
This is certainly true when solving hard problems.
A counterpoint would be that most software used to run businesses is complicated, but not hard (or the hard parts are very localized.) And for these problems it's best to factor your codebase such that it is not required to hold the whole thing in your head.
Because if you do, at some point in the project's history, someone is going to blunder in and make a change that fractures your crystal palace without you even realizing it.
I'd say the second part of #3 is addressing this.
> You can magnify the effect of a powerful language by using a style called bottom-up programming, where you write programs in multiple layers, the lower ones acting as programming languages for those above. If you do this right, you only have to keep the topmost layer in your head.
This is pretty much how you should write programs regardless of the programming language. Otherwise the program would become virtually impossible to understand and maintain as it grows.
I think the fundamental is the description of software as an idea.
It is very important to understand, because it helps explains that the essence/idea part of software writing will never be automated and that any repetitive part in software must be automated to make progress.
This is why the concept of software sweat shops/and some bad flavors of pair coding/agile development are stupid, it's all built around systematizing what should have been expressed in higher-order software anyways. They're only effective at still making money for the shop while the market hasn't caught up.
Amazing how modern corporate "catch the cheaters and slackers" management structures are designed exactly the opposite of this.
#2 Work in long stretches.
This is wrong or at least worded incorrectly IMHO. I believe the author knew it wasn't quite right when they wrote it, since the second paragraph is an attempt to qualify it away.
The rule would perhaps be better stated as "Think in a distraction free environment".
Plenty of productive programming is done at rest, and long stretches of "work" produce subpar code.
I do think working in long stretches is effective, but I'm not talking about a single "butt-in-seat" marathon coding session. I think of it more as a focus on solely one project for a number of days, without multi-tasking on 3+ projects at once giving each one only a couple hours per day.
The processing that you do when not sitting at a desk, and those eureka moments that come along with it happen more often when you are working one thing at a time. Or, at least they do for me.
Yep. The tricky thing is that "help a colleague on some little problem" can end up being an expensive context switch, but can also be pretty valuable. A degree of asynchrony, and perhaps some kind of "office hours" type mechanism to handle questions in batch mode seem like good solutions. But that goes against processes which treat "resolving blockers" as top priority.
Thanks!
That seems worthy of its own "Work on one thing at a time" rule to me. Task switching is the bane of agile.
Thinking is work. "Work in long stretches" includes "think in long stretches."
As someone who does this with a lot of programs I've written, I'll say that avoiding abstractions is just as important as distractions --- if doing the simplest of tasks takes half a dozen layers of abstraction and indirection, it makes it all the more difficult to see the whole program at once. "Too many moving pieces makes it hard to see what's going on", as the saying goes.
The amount of "fluff" introduced by "modern" languages and their culture, like Java and C#, is certainly an obstacle to this; it seems like they encourage overcomplexity and the creation of huge complex software, so that all programs written in them naturally take on that style, no matter how trivial they actually should/need to be.
The languages themselves sort of forces people's hands depending on how rigid they are and how easy it is to change your mind afterwards.
Take Python for example. Firstly they have a culture of "we're all adults here" and don't really bother with public/private distinctions in classes, and secondly the language makes it easy to turn a variable into a property with the @property decorator.
I'm not that familiar with Java or C#, but I guess turning a variable into a property after the fact is non-trivial. In Python all the calling code remains unchanged (`val = object.field`)
I manage to keep my main side project (~20,000 lines of code) in my head despite not meeting conditions #1 (I'm always distracted), #2 (I work in my free time at home), and #3 (C++, woo) because I have extensive documentation about the why of code, and this helps me put myself in the same state of mind I was in when I wrote it.
It does help that all the others apply to me, though. I've rewritten it kinda twice, I already mentioned my comments, I work basically alone on it, so nobody else edits anything, and it started out very small.
"Sometimes when you return to a problem after a rest, you find your unconscious mind has left an answer waiting for you."
I love when this happens. For me it's usually after a good sleep.
That's the best programming article I've read in a long time.
I can't say all that is true for me, but some of it is.
It's truly interesting to see how others write and understand programs. It does seem a lot like art (and less like engineering) sometimes.
>Maybe we could define a new kind of organization that combined the efforts of individuals without requiring them to be interchangeable.
Epihany: Tolerance for uniqueness is one way the open source trumps closed source.
That's a wonderful article describing the work of a programmer, why bigger companies cause them trouble and how good programmers reason about code which is not even yet written. Paul Graham, which is a notable figure in the software engineering world if you didn't hear his name yet, names 8 steps that make you able to load a whole program into your head. I'm proud to say that I have mastered all 8 steps multiple times, though from my experience in my current engineering team I struggle with step 7. I do actually believe that what he describes shouldn't be done, is actually feasible - through pair programming.
> 7. Don't have multiple people editing the same piece of code.
Do others find this pretty crucial? We do this at my org and I’m interested in knowing if it’s detrimental.
I think it's such a basic issue that one shouldn't even need to point. It's like remembering people to breath.
And now you just said you don't breath, and I am really questioning my understanding of the world. Do people at your org get anything done?
Paul Graham's essays are a consistent joy to read.
I totally agree with PG here. This is also how Steve Wozniak designed the first Apple hardware