Chain of Thought for Humans

Posted by ibobev 2 days ago

Comments

Comment by mceachen 8 hours ago

In almost every interview I gave at the last several companies I worked for, I'd start with "This may feel a bit odd, but with this coding problem, I'd appreciate if you spoke through your problem solving. If it helps, pretend like I am your intern. Good ideas, bad ideas, pros and cons of whatever you're considering, it's all valuable information for me to learn a bit about how you approach problems."

I feel like being explicit like this put candidates at a bit more ease than having them try to suss out what the interviewer is expecting from them implicitly (and I know there are many interviewers that will auto-veto any candidate with the audacity to suggest a possibly imperfect approach).

Comment by wahern 5 hours ago

I always hated when interviewers did this. My chain of thought is often backwards, very abstract, or otherwise wildly different from how it seems most others think, and it's very difficult if not impossible for me to articulate my thinking on the spot. In school for writing assignments I would have to spend alot of time reversing or reordering how I arrived at a conclusion before even writing my first sentence. And even just arriving at a conclusion involves taking many dead-ends; the process can be seemingly instantaneous in places, or require laboriously following rabbit holes (often with throwaway code) that I know are dead ends but nonetheless help me to explore context. I need to have the entire problem in my head and able to explore it to at least some extent before even starting to formulate a linear chain of thought, let alone commit something comprehensible to code.

IOW, I can't explain in real-time how I'm working things out. I have to work them out to completion first, quietly, or at least have a good idea about what the full path looks, end-to-end, in detail and only after the fact could I ever hope to explain my thinking. Asking me to explain my thinking while I'm thinking is easily the most stressful and impossible task anyone could ask of me. I'd rather you break my fingers as part of the interview process.

I understand as an interviewer that makes for quite a dilemma. And I suppose if you find this approach reasonable then you may not like working closely with someone like me (though if you're just a manager, then it's a bias to be aware of).

Similarly, practices like pair coding just don't work for me. I'm not not a team player. I like and benefit from design and ad hoc technical discussions, just don't ask me to sit down and write code with someone else, or even with someone looking over my shoulder. All parties involved, including myself, will just end up pulling their hair out. Though I do work best with people who think similar to me. We instinctively know when and how to set our boundaries and expectations.

To interview someone like me it's probably best to focus on having the interviewee identify waypoints (both abstract and concrete) to confirm they have the experience and understanding to navigate difficult problems. Just don't ask them to actually navigate on the spot. If you want to see how they ultimately connect the dots in practice, look at code they've already written. Unlike in most other fields, software developers often have a corpus of work that is easy to share and often already published.

Comment by skohan 1 hour ago

I have interviewed dozens of engineering candidates. A big part of what I am looking for in an interview is how the candidate thinks, and what it's like to work with them in an interview.

If someone can't communicate well on technical matters, that's generally a negative.

That said, if a candidate was frank with me that they don't do well being put on the spot with a question like that, and can articulate how they do actually solve problems, and give examples of how it's worked for them in professional settings, I would consider them for the role if their answer was convincing.

I.e. if I asked you to explain how you would approach a problem, and you said you might need to go off and write some throwaway code to explore the edges of the problem that's a good answer.

Comment by rc1 4 hours ago

I don’t think I think so differently from you and managed through a few interviews with success. I almost go further than the OP by suggesting to interviewees that I have a blank page of notes I need to fill in with evidence to support hiring them, for someone else to read. The coding question is to kick start that process.

The thinking gives more than just the answer and makes it more of a conversation. As a candidate I will mention projects I had similar when think about anything. As the interviewer it is hard to gather experience: so many people stay abstract and it’s infuriating (let me help you get this job! Yes you can do something means nothing; when you did do something means everything.)

I am sure that many interviewers don’t see it like this however and treat it like a quiz sadly. Best advice for interviews I was given as a candidate: the interviewer will decide in the first five minutes if they want to hire you. The rest of interview is them trying to convince themselves and others of that decision.

Comment by noduerme 6 hours ago

I really wanna ask, because... I'm 46. Been writing code for a living about 30 years, never on a salary, always contractor. I wrote and maintain enough code for non-tech companies that aren't going to take a risk on AI rewrites anytime soon that I can probably coast until retirement. But I haven't interviewed or even needed to sell myself for a long time.

I always thought I would easily pass an interview, but I may have been wrong before all this on the basis that I don't work that well on a team.

OTOH ...is the question now really valuable whether a candidate can explain their thought process rather than deliver? Why wasn't that a question 20 years ago when it actually mattered? Or does it matter more now because the bar needs to be higher, and there's no cache of more experienced devs to train them up?

Comment by ghoul2 1 hour ago

I do something exactly like what the parent said, and my own view.

The question isn't about figuring out junior or senior. Its about the candidate being able to think thru a problem. They don't even necessarily have to be good at articulating their thought, I just want to see how (and if at all) they are splitting open the problem, walking thru it, figuring it out and then converging on something that (at least to them) feels like a reasonable solution.

If they get stuck, I am happy to help them out, without disrupting them or deviating them. If they go down a wrong rabbit hole, I may ask a leading question that helps them realize the issue and gets them back on track.

The bar isn't being set higher. Its just that there are a LOT of people interviewing (in my experience) unfortunately, who seem to be unable to model problems in their head, model how their code would work (mentally model the execution of the code), and reason things out. They have frequently coasted on blind copy paste from stackoverflow/help/tutorials. A huge fraction.

I also have been coding for a LONG time (older than you), and in the early days, this sort of conversation didn't have to be explicitly asked for, it happened naturally, like in every interview (where I was the one being interviewed) - cause you present me with a problem, and I can't help but be nerd-sniped by it. People who were candidates for tech jobs were there only cause they actually went into it with a bit of love/passion/skill.

But then it became a large industry, and most people who went into it were not 'hackers (problem solvers)' at all. They just liked the great pay. They may even be intelligent, but a lot were not capable as independent code owners. Still useful for basic crud with the right frameworks/etc. So thriving in enterprise applications, java/c# mega-platform/mega-framework domains.

Comment by vbezhenar 4 hours ago

There are too many candidates for too few job openings. So people come up with all matters of ridiculous filters because they feel they have to test everyone. In reality they can take almost anyone with relevant experience and they'll get the job done, but that wouldn't look like good hiring process.

Comment by skohan 1 hour ago

I have ~15 years experience with hands on engineering. I also served as a hiring manager in a couple of roles, and interviewed dozens of candidates.

I would love to live in a world where every candidate represented their experience accurately, and every applicant was capable of fulfilling the role, but sadly it's not the case.

I agree there's some filters which are mostly performative (like leetcode), but being able to talk through the kind of work you do is table stakes. And not just for engineers, for any professional role.

Comment by swiftcoder 2 hours ago

> on the basis that I don't work that well on a team

I think you are likely to find that "working well on a team" is the number #1 thing tech firms are screening for. Technical skills are all well and good, but communicating/collaborating are half the job in a big company.

Comment by adrian_b 1 hour ago

"Working well on a team" is something completely uncorrelated with the ability of explaining your thinking during the time when you are solving a problem.

All the teams in which I have ever worked have greatly appreciated my contribution precisely because I was always willing to spend a lot of my time explaining to other colleagues, in a differentiated way appropriate for the understanding abilities of each of them, how I had solved my problems or how they should solve their problems.

On the other hand, I am also completely unable to explain my thinking while it is in progress, because reformulating it to become intelligible to the interlocutor would slow me down far too much.

Explaining on-line my thinking would require even a kind of language translation, because when I am speaking to myself I use a special language with precise non-ambiguous words, whose meanings correspond in many cases with old meanings of those words, because in the modern language a very large number of words have vague meanings, which have become very diluted due to the frequent use of those words in inappropriate contexts.

So when I describe some technical solution to others, I typically do not use the words with which I think about it, but whatever words are more fashionable today, so I expect familiarity with them, even when I think that those are not the right words.

Doing such a translation while I think is too difficult, and I do not attempt to think with whatever words become fashionable from time to time, without any good reasons, because in my opinion using the right names for concepts is very important for understanding and using them properly.

Comment by noduerme 2 hours ago

I mean, communicating/collaborating with upper management and turning their requests into solid code? That's what I get paid to do. Communicating/collaborating with other coders? Not so much. Note that I specifically said I don't work for tech firms.

Comment by 5 hours ago

Comment by grigri907 5 hours ago

I think delivering is still a requirement either way, but the value is in hearing a candidate has the ability to problem solve in an unknown problem space vs regurgitate a memorized solution. Even better is witnessing some has the ability to try, fail, then self-correct, and rebound.

Comment by noduerme 4 hours ago

So, my policy has always been that if I sit through a meeting where a client asks for new feature "X" and I figure out a way to avoid building the new feature, and either use existing business logic / day-to-day processes, or else find some new way that doesn't involve code, I don't bill for that time. I'd say that I probably give up about 35-40% of potential coding work just by helping them work through that. Which is fine, because it leaves me free time to do other things.

On the other hand, I don't have the option to fail even slightly. I deployed a patch yesterday for an old, rare, logic bug in one reservation system, and within an hour I was being bombarded with edge cases of people seeing their reservations as cancelled during the 15 minute switch-over period.

Sometimes I feel like the job of a programmer for a company is just very different from what I do. If anything goes wrong with anything I deploy, I'm awake for however long it takes to make sure it's right. On the other hand: No PRs, no teamwork. Just a click to deploy once I'm relatively sure. And a lot of responsibility.

Comment by SwtCyber 4 hours ago

[dead]

Comment by vjerancrnjak 4 hours ago

Articulating thoughts out loud is a horrible requirement. Conscious verbal thinking is a detriment to thinking, thinking unconsciously is much more efficient, only requirement is involvement with the problem that does not need to be verbal at all.

Problem is solved before the solution is consciously perceived. The reverberations after perception can be of many different styles.

There is little value in chain of thought and I think humans are implicitly or explicitly instructed to verbally think and hinder their problem solving facilities. There are many examples of individuals that went much further with their own internal but externally indecipherable thinking models because they avoided language as a thinking faculty.

On top of that to ask for clear communication and decoding of the verbal thinking mush into sounds is just an extra hinderance. Id prefer to ask questions to evaluate understanding of the problem solved than force someone to dump their mush into a clear spoken summary.

Interviewing process is just a mush of nonscientific nonsense , a theatre for all involved, a sad affair.

Comment by maest 26 minutes ago

> Conscious verbal thinking is a detriment to thinking

TFA literally links to a study contradicting this core claim.

Comment by sinuhe69 6 hours ago

In some cases, it can certainly help. However, it isn't helpful for deep thoughts or when innovative, creative ideas are needed.

I have found that speaking the steps out loud helps, especially with new, confusing working steps. For example, it helps when tying a complex knot or following the steps/work of others without explicit instructions. In such situations, speaking out loud helps me to remember the steps much faster (simple repetition through multiple pathways) and also helps me to detect and correct spatial and sequential mistakes. It also helps enormously when working with or teaching someone else.

I have a funny memory from my youth: one time, I had to solve a particularly difficult maths problem on the blackboard. I don't remember exactly why, but while I talked out loud about solution A, I wrote a slightly different step (B) on the board. Looking back, I realised I had made mechanical mistakes, so I had to correct them to get the final valid solution. My teacher laughed and joked, 'A sign of genius is to write A, say B, but only C is the true solution.' :) (no, he was just too kind).

I have often experienced my handwriting being much slower than my speaking, so sometimes they don't sync and funny mistakes are made (one line contains the words of the next line). But I fondly remember that particular moment.

Comment by xorvoid 7 hours ago

I basically "chain of thought" to myself in my own notes when working through a hard task. Mayne we can call it "live tweeting in private".

Aometimes it feels like a super power because it takes away the "starting anxiety" and makes everything a trivial sequence of "following simple steps"

Comment by human_llm 6 hours ago

I do the same. My impression is that very few people do this, or take proper notes at all. This is based on my observation that most people I work with seem to have zero recollection about things that happened more than a few months ago.

Comment by Waterluvian 10 hours ago

Witnessing a candidate’s thought process is probably the most valuable part of the interview process for me. Nothing else comes close. Just let me listen/watch you solve problems.

Comment by purplepatrick 9 hours ago

As someone with zero exposure to my own thought processes, I find this always so discriminatory! ;)

Comment by lioeters 7 hours ago

I wonder if this may be related to anendophasia, the absence of an inner voice. Apparently it has an influence on self-reflection and verbal working memory.

Not Everybody Has an Inner Voice: Behavioral Consequences of Anendophasia - https://doi.org/10.1177/09567976241243004

Comment by SoMomentary 6 hours ago

I have aphantasia so anendophasia sounds like such a nightmare to me, though I imagine those so have it would likely feel the opposite of me. I've met some folks in the past who I'd suspect have anendophasia, but I'm not sure how I'd ever bring that up without coming across as uncouth.

Anyone with anendophasia care to opine on their experience?

Comment by drivebyhooting 9 hours ago

Verbalizing my thoughts as I try to solve a problem drops my IQ down by like 20 points at least.

Comment by neronuser 3 hours ago

good thing interviews don't assess IQ :)

Comment by 4m1rk 9 hours ago

Exactly, specially not in your native language

Comment by SwtCyber 4 hours ago

[flagged]

Comment by florkbork 8 hours ago

This is basically https://www.christopherroosen.com/blog/2020/4/20/how-the-rit...

Comment by totetsu 8 hours ago

After one to many miss-rides, I now do this often even when just riding the train.

Comment by armcat 4 hours ago

I’ve been doing this with voice mode in agent harnesses. In Codex for example I often speak orally and I find that (1) I discover something new myself, (2) agents seem to understand much better than pure text prompts. The only downside is it costs lot of tokens.

Comment by aswegs8 2 hours ago

Do it all the time. If I want to understand something, just copy over all the notes, start writing down the important parts. Write down the key questions, answers, my thought process. I think of it as a crutch to focus and concentrate and wind up the wetware.

Comment by algoth1 10 hours ago

I wonder if the author is one of those people that doesn't have an audible inner monologue?

Comment by thanhhaimai 9 hours ago

I'm not the OP, but I think everyone might have a different definition of what "audible inner monologue" mean.

For example, I don't talk to myself in "monologue". But I can close my eyes or stare at the screen and visualize the solutions. The words that I usually tell myself during that are "what?", "why?", "strange!", "really",... not a monologue or sentences.

Comment by avallach 3 hours ago

Like the response from the UltraSane: my thinking is and always was 100% made of internal language-based chain of thought ("imaginary talking to myself"), with some amount of introspection / meta reasoning when problem is particularly hard. This article indeed reads to me a bit like someone proposing a physical storage medium inspired by those pictures of little squares traditionally used as "save" icon, without realizing that this digital representation was in the first place based on an existing real world thing.

Anyway, a surprising related observation is that rubber ducking by describing the problem to quiet listener still often helps me, which suggests that internal chain of thought makes intuitive shortcuts that sometimes are counter productive, while talking to another person creates subconscious pressure to make the thoughts fully complete.

Comment by pulvinar 9 hours ago

That's my impression too, that my thoughts are a hybrid, where I'll think in a rapid sequence of whole concepts and visual flashes, and then summarize that in my inner voice "for the record", as a jumping-off point for the next batch of thoughts. But I don't have much control over this, as my inner voice is hard to shut up for any length of time, always giving a running narrative.

Comment by FuckButtons 8 hours ago

Yeah I’m the exact opposite, thought is almost entirely auditory.

Comment by UltraSane 8 hours ago

I'm essentially thinking out loud in my head constantly.

Comment by purplepatrick 9 hours ago

The author says they could narrate their own thought process out loud.

I couldn’t do that. I have no insight into my thought processes whatsoever. There are no semantic (or any other) processes to witness. It’s complete crickets at all times…

My wife is the opposite - constant chatter in her head. And she can trace every thought exactly. (No idea how people like her can focus or even function!)

Comment by cnees 9 hours ago

We focus/function via the inner monologue. It doesn't just do its own thing while we're trying to focus or get something done, it walks through the issue and comes up with solutions and then criticizes them and iterates, and as far as we're concerned, that's us solving the problem.

Comment by zer00eyz 7 hours ago

Anendophasia? Fair number of people have this.

Comment by idiotsecant 10 hours ago

I am such a person and I've always done 'chain of thought' in a notepad in order to think through complex things, well before LLMs were a thing. There are certainly tasks I can 'one-shot' but I've learned from experience that for some problems that just results in me focusing on irrelevant things.

Comment by apt-apt-apt-apt 9 hours ago

Curious whether the lack of audible inner monologue is a benefit for you?

I feel like I'm the opposite, where most of my thinking is done in inner English, which feels slow. I.e. as opposed to instantly realizing the result of deep subconscious processing or something, I usually go the slow language way, if you get what I mean.

Comment by avallach 3 hours ago

My experience: intuitive thinking it's a shortcut, but it's less reliable when it comes to complex problems, especially involving multiple concepts that have to be reasoned about with non trivial relationships or rules. In a way I see a spectrum of approaches that range from fast to reliable:

1. Intuitive thinking

2. Language based internal monologue

3. Written scratchpad

4. Explaining the problem to another person until they understand it (rubber ducking)

Comment by slopinthebag 6 hours ago

i think everyone has the same inner monologue and it's entirely a matter of how we define "monologue"

Comment by jamienk 8 hours ago

"Free association" in psychoanalysis just asks people to let themselves speak, to fall into their speech, let it unfold - but it is very, very hard to do, because what you say tends to not be what you want to say. You want to avoid certain things, you can see your chain of speech leading up ahead to somewhere you don't want to go. So you stop yourself, or resist, and then what you are saying is not at all free but strained, hedging, tense and anxious.

Comment by clivedup 8 hours ago

> eh, "Chain of Thought for Humans" sounds stupid

> but maybe it has some nuance?

> eh, I think I still don't care enough to click

> but maybe I could post a funny comment

> it doesn't sound that funny, and besides, HN doesn't usually like these sorts of comments

> okay, let's just read the next story

> unless

Comment by avallach 3 hours ago

My favorite comment of the year because of how relevant it's here :D Good old https://tvtropes.org/pmwiki/pmwiki.php/Main/SelfDemonstratin...

Comment by 1auralynn 10 hours ago

We use this in my science education platform: we make students talk out loud about what they're thinking. Not a chatbot, but guided inquiry that's designed to make them actually mentally engage with what they're seeing

Comment by flir 2 hours ago

Various advanced driving tests in the UK do something similar - you narrate as you drive. It's called commentary driving.

Comment by tarkin2 7 hours ago

A Mathematicians Lament mentions students explaining rather than reciting the equations, too, to correctly learn them

Comment by whattheheckheck 8 hours ago

Just make a theoretical minimum type test like the landau physicist

Comment by akoboldfrying 4 hours ago

I often find it helpful to type comments (starting with a # character) directly in a shell session, to remind myself where I'm up to, what I need to do next, or what I've just done. Especially handy at the end of the day or when switching context to something else.

Comment by senectus1 9 hours ago

holy shit this would break my ability to think completely.

I think and chase alternate chains of thought way faster than i can speak.