GDB can almost be a dynamic computing environment:

  $ gdb ./txr
  GNU gdb [ ... ]
  [ ... ]
  Reading symbols from /home/kaz/txr/txr...done.
  (gdb) b eval
  Breakpoint 1 at 0x808f6b0: file eval.c, line 969.
  (gdb) r -p '(+ 2 2)'
  Starting program: /home/kaz/txr/txr -p '(+ 2 2)'

  Breakpoint 1, eval (form=0xb7fb919c, env=0xb7fb92ac, ctx_form=0xb7fb919c)
      at eval.c:969
  969     {
  (gdb) p d(form)
  (+ 2 2)
  $1 = void
  (gdb) p d(cons(form, form))
  ((+ 2 2) + 2 2)
  $2 = void
  (gdb) p d(cdr(form))
  (2 2)
  $3 = void
  (gdb) p d(cons(0x1, 0x1))
  (0 . 0)
  $4 = void
  (gdb) p cons(0x1, 0x0)
  $5 = (obj_t *) 0xb7fb921c
  (gdb) p cons(0x9, $5)
  $6 = (obj_t *) 0xb7fb920c
  (gdb) p cons(0x19, $6)
  $7 = (obj_t *) 0xb7fb91fc
  (gdb) p d($7)
  (6 2 0)

You can test functions that are not even called anywhere from the C code. The d function is such a function; it's only there for use out of GDB.

  (gdb) p (0)
  $10 = 0
  (gdb) p d(0)
  nil
  $11 = void
  (gdb) p t
  $12 = (val) 0xb7fe5eac
  (gdb) p d(t)
  t
  $13 = void
  (gdb) p typeof(t)
  $14 = (obj_t *) 0xb7fe5d3c
  (gdb) p d(typeof(t))
  sym
  $15 = void
  (gdb) p d(symbol_name(t))
  "t"
  $16 = void
  (gdb) p d(symbol_name(0))
  "nil"
  $17 = void

Take out the trash, and show it:

  (gdb) p gc()
  $18 = void
  (gdb) p d($7)
  #<garbage: 0xb7fb91fc>

And in combination with conditional breakpoints executing commands, you can have GDB "dance" any application to whatever tune you wish:

commands ... call d(cons(form)) call symbol_name(t) cont end

It really is a marvelous tool - unfortunately, people don't know anything except the basics (if that)

Is there anywhere I can learn specifically about the commands you mentioned?

I've used them too - just "info gdb" then search (/) for "Breakpoint Command".

Awesome, indeed.

Other classics: dumping the memory of an active process (check my GitHub utils repo, I have it there) and redirecting the standard output AFTER you've started a process.

Ah, dumping memory. I once took the dump-and-restart code from GNU Emacs, and planted it into GNU Make. I was then able to read a large and complex Makefile (consisting of included sub-Makefiles throughout a large tree) which took 30 seconds to process, dump the image to disk (with make --dump), and then use the dump to run incremental builds which were able to kick off build recipes almost instantly!

Amazing - you must blog about it!

I only wish that in a site called 'Hacker News', it would be technical comments like yours at the top of this discussion. Oh well...

Talk about some dark magic (to me at least). Well, not that dark, turns out people wrote about it here http://www.emacswiki.org/emacs/DumpingEmacs

Just curious about one of the last steps:

  cat /proc/53165/exe > /tmp/oldBash

I would have reached for 'cp /proc/53165/exe /tmp/oldBash', instead of using cat. Is there an advantage to using cat here, or is it just the same?

Indeed! No idea what went through my brain when I wrote that :-)

Could you just use `gdb --pid 53165 /proc/53165/exe`?

If that works, is there any reason why gdb shouldn't automatically use the /proc/PID/exe?

Probably because /proc is very Linux-specific, and gdb is not.

I wonder, too... It could pick it up automatically.

Probably - will try it out next week and adjust my GitHub script if it works.

I recently did exactly the same, it's those times when everything with similar functionality depending on what context you are in blurs together because you should probably be outside, taking a walk, or staring at a tree.

However, reaching for one way of doing things when the justification for the other way of doing things is because you've seen them done that way more, is not really good justification. So maybe that blurry brain is a good brain sometimes, in retrospect.

:-)

Hey, working code is the only justification you need :)

And cp would have also saved permissions (the executable bit). Not that GDB would have an issue without it, as the OP showed it can look for symbols inside an ELF regardless of the inode bits.

I would have tried running 'strings'on the process's memory instead of gdb. Ugly, but likely would have worked with some manual intervention.

As to 'cp' vs 'cat': neither is a builtin, so they both are risky. What if you also accidentally did a rm -rf / on that system?

Mac OS X doesn't have a /proc filesystem, so I cannot test it, but from reading the man page

   read -N 1000000000 /proc/53165/exe $bash

should read all of bash into $bash, if it isn't ridiculously large, and

   echo -nE $bash > myBash

should then complete the copy.

Why are cp and cat "risky"?

It's just that in some emergency situations, where for example someone deleted a lot of files on the disk, you might not have /bin/cat or /bin/cp anymore.

I don't see how that ("rm -rf /") is relevant... The OP feared for a reboot that would make him recode his command line.

Ah, fair enough. Good point :)

    read -N 1000000000 /proc/53165/exe $bash

That shouldn't work, due to NUL bytes.

Why not ctrl+c, then arrow up and enter? If the command can't be stopped even for a few milliseconds then there is something wrong with it.

I didn't want to send any signal - because I could not remember what I had done, and was afraid for sideeffects (I did remember that the logic had to do with database actions).

I wanted to see the command without "messing" with the execution in any way - after all, it was running fine for 6 months, why interrupt it (and face potential effort in fixing messed up state) if I could see the command "from the outside"?

And that I did :-)

C-z is actually your friend here. I've never ever seen SIGSTOP affect anything negatively. I even SIGSTOP Mac OS X apps sometimes. They just beachball until you SIGCONT them, then they happily continue, oblivious.

If you're worried about timing out of the process being unresponsive, just C-z and then "bg" real quick. Then you can up arrow at your leisure and "fg" when you've copied the command away...

Try doing the Ctrl-z/fg sequence in this: "while true ; do sleep 1 ; done". Under Linux at least, you'll see that after 'fg', the loop ends :-(

Apparently C-z followed by fg is not bulletproof...

Thats an interesting example.

zsh does the right thing, when you bg it back it recovers the loop as it was. Bash seems to forget about the loop and only recovers the "sleep 1", when it starts executing again it sleeps 1 and exits.

If you want to make sure it will work you should wrap it around another bash:

    bash -c 'while true ; do sleep 1 ; done'

should work fine.

Parens work too:

   (while true ; do sleep 1 ; done)

I tried it, and you're right - but why does this happen?

while, true, do, done are all [bash] shell builtins. They are not running processes that can be suspended.

sleep(1) is the only process that can be suspended in this command. When you "fg", the suspended sleep(1) is signaled, and resumes. And then it exits.

When you do this in a subshell (in parens), you suspend/resume the subshell in its entirety, which capture the loop construct.

Why does it work in zsh?!

Warning: Ctrl-Z sends SIGTSTP, not SIGSTOP. SIGTSTP can be caught by programs, while SIGSTOP cannot.

There was a realllly old OS X program that would automate this so the front-most app was the only one running, to speed up your machine.

Don’t know which old OS X program you’re talking about, but there are actually modern OS X programs which also do this. App Tamer¹, for example.

――――――

¹ — http://www.stclairsoft.com/AppTamer/index.html

I know a lot of programs that would have issues with Ctrl-C - I understand why the OP didn't want to do this.

The tip about getting the binary from /proc is also helpful if you upgrade screen/tmux and the newer version doesn't allow you to reattach to it. With that you can use the old one to reattach, work your magic, and restart the session.

You just explained why I could not attach to an already existing screen last month - it all fits now...

When I read the problem description, I decided to try and solve it myself.

I used gcore to dump the running bash process, then ran strings on the core dump, then grepped for likely command line stuff. 'while' is reasonable choice for a loop. There aren't that many strings in the dump file in any case, most of the sizeable ones are function definitions and environment variables.

Relying on gdb makes getting to the right data much more precise, but my approach took about 20 seconds.

I saw things being saved to /tmp, and kept expecting a turn for the worse from a sudden reboot...

Fantastic article though - I've learned a couple of things from it, thank you.

My pleasure :-)

Great story... but I read through it hoping to see the fabled command in its entirety at the end! Any chance..? :)

God, no :-)

Some people here are already rather mean in their comments - imagine what would happen if I revealed The Magic CmdLine (TM)...

No, that privilege is reserved for the Inner Circle :-)

But I'll give you a teaser... It involved a lot of ImageMagick and "pdfimages" and "zxing" and "tesseract" and "pdfjoin" invocations... and the uploaded files were PDFs with scanned barcoded pages.

Aw, there's nothing worse than a mystery! ;)

(I can imagine the gory details tbh - it sounds like there is every chance I have had a very similar command for a pretty similar purpose running for a good long while now... except mine's in a VBScript..!)

:-)

Across the road (on Reddit/programming), a commenter already described me pretty well:

"Everything about your (rather interesting) story... reaffirms everything I know about Linux sysadmins - if you encounter a problem, your go-to solution is to write a script ... Honestly, you guys would script a sandwich if you could :P"

The only thing wrong in that sentence, is that I am not a sysadmin (hence the knowledge of the Dark Arts - of GDB :-)

Now this is both interesting and well written post indeed.

This was a fun read and great hack. Thanks for sharing.

My pleasure! It was a lot of fun hunting all this down, I just had to share it :-)

It is indeed cool, but the single best lesson to come of it might be to start the work in a script in the first place. Still, cool beans. :)

You, sir, are hardcore. A steely eyed missile man, as they say. Awesome bit of forensic debuggery.

the best I've read since a long time!!

I do, indeed - and in Git repos, usually :-)

How that one got by me is a mystery - let's chalk it up to stress and pretend it never happened :-)

Hehe. And to contradict myself, if you HAD written it all down you wouldn't have the cool story :)

Since the shell where I wrote the command is still executing, the .bash_history has no mention of it (bash saves the commands in the history file when it exits). As for the up arrow, it would mean interrupting the commands - and I didn't want to :-)

Good question. Well, when you have forgotten everything about a 10 lines long command line you wrote 6 months ago, you don't know what side effects a signal may cause... You don't have the heart to signal anything, except surrender. Luckily, I found a workaround :-)

While I'm unable to come with this on my own, the article was pretty easy to digest and understand.

It seems that the take-away lesson from the article is that you should save it in a bash script.

screen doesn't have history?

It has a finite buffer, so I imagine 6 months worth of logs would have easily blown away the history of you running the command.

You can configure it though, to be much larger than its default settings. Still, 6 months... he'd probably have to hack it as he did, no matter what :-)

Exactly.

he didnt invoke it from within screen it looks like - it was invoked with the invocation of screen

Cool tricks, but reads like a how-to of things not to do when writing any type of code and deploying it.

A Jedi? Maybe at getting himself out of the quicksand he lays himself into ...

Oh come on :-)

You're telling me that you've never - under tremendous pressure - hacked things that quickly "handle" something - and then forgotten about them till much later?

Jeez, tough crowd :-)

Those who do, blog. Those who don't, snark at those blog posts on HN.

You're right, quicksand all around.

However, the post illustrates an important feature I often see overlooked: the systems we build on top of are not just black boxes. We should not be afraid to really understand how they work, opening up source code, asking questions like, "what the heck is this /proc/NUMBER section of the filesystem?" And I think this post illuminates that in a pretty fun way.

wouldn't a simple Perl script and cron do the job I recall back I 2000 doing 20 line script in an afternoon that tied together one of the major exhibits at the millennium dome that my employer was sponsoring - the 3d scanner booth thing from memory.

I appreciate the spirit and the knowledge that you wrote this blog post with BUT -

The extra few seconds to put this in a .sh file in the first place ... priceless. Even for the basic case of debugging it and getting it working.

I don't think in the 20-25 years I've been "doing" unix that I've ever done this (I doubt anyone that has worked with me considers me ocd fwiw but this is beyond the pale).

Behaviors like this lead to "magic" work environments and are just pants on head places to work at/with.

I would be really mad the first time I found you doing this and I'd look for a different work situation for one of us the second time :)

Thanks for the cool post.

On the other hand, the Internet is a multicultural and multinational community. It does make some sense to shed the typically American “right to not be offended” when you browse the Web.

Just as much as you ask for your culture to be respected, respect other cultures by acknowledging their more offensive and black humour and not taking everything personally.

It's worth noting that the author appears to be in Greece. Assuming he's a non-native speaker, it doesn't really make sense to be "offended" by his misuse of the term.

Maybe a better way to put it would be "We don't typically use 'Alzheimer's' to refer to forgetfulness because it brings up negative emotions for some people."

Greek language is more about emotion than content. Hyperbole is used to emphasize meaning. I cannot forget he first time I went to Greece and my aunt kissed me, bit me, and said she was going to eat me. I was 6. I was petrified. It is common in Greece to express disappointment affectionately by saying ' I am going to kill you'. It is our cultural ignorance in the US that makes Americans easy targets for ridicule. If we project ourselves as a superpower to the world, then where indeed are our superpowers of understanding?

It's considered 100% normal in everyday speech here in Greece, to use the name of the disease as an excuse when you forget something. I had absolutely no intent to offend anyone, and my heart goes out to the people fighting this disease - and the ones taking care of them.

I guess it's yet another cultural difference I was not aware of...