Path autocompletion in the PyCharm IPython console: a story of suspense

Disclaimer I: I love PyCharm; bellow just happens to be the description of one of its darker corners.

Disclaimer II: The whole post might be rendered obsolete any time soon (and I hope it will). It is relevant, though, at the time of writing, with PyCharm Community Edition 2017.2.3 and with Anaconda 5.0 / Python 2.7.13 and Anaconda 5.0.1 / Python 3.6 and below, on several different versions of Windows 7 (Home, Professional etc.).

How to fix path autocompletion in the PyCharm IPython console

There's a couple of questions scattered around the net related to path autocompletion in the PyCharm IPython console... but the useful answers are rare, almost non-existent, and scattered around as well.

So, what's there to fix after all?

Well for example, when I type:

c:/U<TAB>

I'd expect autocompletion to offer a sensible list of directories and files in folder c:/, starting with U.

On my computer this happens to be just c:/Users.

Yet what autocompletion proposes looks like a random list of keywords and variables from some Python namespace:

/images/nonsense.png

WTF?!

To make this even more weird, "my" IPython (which is part of an Anaconda installation), used in the Anaconda prompt, merrily autocompletes path items "all tha way down":

/images/better.png

... and c:/U<TAB> directly resolves to c:/Users.

WTF square. Isn't this the same software here, and if so, how comes it behaves differently?

I found a way to tame autocompletion in a manner good enough for my situation, and I'm just about to describe it. I should probably just mention I'm neither a specialist on the IPython API nor a champion of PyCharm configuration, so there are certainly better, more thorough ways to do this.

For the impatient: A quick fix in code

If you add this code:

# Completer, available to be imported in starting script
def create_my_file_list(self, event):
    my_files_list = ["c:/Path_A", "c:/Path_B", "c:/File_C"]
    return my_files_list


ip = get_ipython()  # Here we get access to IPython
ip.set_hook('complete_command', create_my_file_list, re_key='.*mymagic')

from IPython.core.magic import register_line_magic
@register_line_magic
def mymagic_some_functionallity(file_name):
    print "they asked me to print", file_name

to the Starting script (File->Settings->Build, Execution, Deployment->Console->Python Console) for the PyCharm IPython console, and get a new console window - or restart the console, writing:

%mymagic_some_functionallity c<TAB>

will offer:

/images/a_fix.png

Now, what do we have here?

The funtion/method create_my_file_list is a 'completer' in the vocabulary of the IPython API. It returns a list of (in this case, fictional) paths.

When added to the IPython-machinery with the call:

ip.set_hook('complete_command', create_my_file_list, re_key='.*mymagic')

the completer will provide autocompletion proposals if autocompletion is initiated in a line matching the regular expression '.*mymagic'.

The name of the self-written magic command mymagic_some_functionallity matches the regular expression above.

So when %mymagic_some_functionallity is at the beginning of the line, the list of autocompletion-proposals provided by the completer create_my_file_list is right at the top of the proposal list.

For the why-s that lead to the code above and the caveats, keep reading.

What is the reasoning that lead to the code above?

My premise to start with: I needed a way to feed my new magic commands with some paths as entry parameters, and it had to be more elegant than copy-and-paste, or typing the whole path.

I soon realised drag&drop will not work with PyCharm, so I went for the second best: autocompletion. Which also presented itself dysfunctional, in the way described above.

Knowing nothing about IPython configuration, I started googling and leaving questions at Stack Overflow, Reddit, the IPython-dev-list, PyCharm support site, all without much reaction.

In those places asked where I can find out about how to influence PyCharm IPython console autocompletion behaviour, any starting points?

I got no answers, but, turned out, a good starting point is the IPython documentation , including the IPython API documentation. No, I didn't try to learn all of the IPython API, but reading the documentation I learned there is something called completers, and that you can write custom ones.

A search for completers lead to a blog post describing how to write custom completers, which contains the essence of the solution above.

Some things to notice:
  • The form of the call to a completer function is def name_of_the_completer(self, event).
  • The parameter self is obviously in order for the completer to be added to some class (by calling ip.set_hook).
  • The parameter event defines when the code of the completer will take effect, e.g. be called. I our case this is when the beginning of the IPython command line corresponds to a certain regular expression.
  • In order to call ip.set_hook we need a code handle to the current IPython instance, ip. with the line ip = get_ipython()

Last thing to mention: obviously, knowing what you know now, the list of paths you might want to offer in the completer is up to you. I had a very specific problem to solve, so my choice is irrelevant here. But the principles presented apply nevertheless.

Why is the completer matched just by a specific line pattern?

Now, theoretically we could have written a completer matching any form of line beginning, instead of just matching our custom magic commands.

It just happens to be that adding our completer the way above will overwrite the addition of magic commands to the IPython autocompletion machinery, so this was not an option to me.

And the reason that adding our completer will overwrite magic commands autocompletion? I this document, a change proposal stating that the IPython tab completion machinery is messy, complicated, and difficult to extend, there's a section Limitations of the Current System stating "the fact remains that the custom completers are always exclusive-- once one returns sucessfully, its results are immediately sent to the frontend and no merging of completion lists is performed". Please consult it, if you want to dig deeper; it's probably a good starting point,.

How comes IPython behaves differently in Anaconda prompt and in PyCharm?

Last but not least: IPython behaves differently in Anaconda prompt and PyCharm - how comes?

After some banging-had-to-wall: while it's the same IPython, it does not use the same configuration / setup.

Ah, the pleasures of introspection!

In PyCharm:

In[5]: get_ipython
Out[5]: <bound method PyDevTerminalInteractiveShell.get_ipython of <_pydev_bundle.pydev_ipython_console_011.PyDevTerminalInteractiveShell object at 0x0000000003D73940>>

On my computer, the _pydev_bundle resides in C:\Program Files\JetBrains\PyCharm Community Edition 2017.2.3\helpers\pydev\_pydev_bundle.

In Anaconda prompt:

In [3]: get_ipython
Out[3]: <bound method TerminalInteractiveShell.get_ipython of <IPython.terminal.interactiveshell.TerminalInteractiveShell object at 0x0000000003B80198>>

Obviously, PyCharm forked its IPython configuration from PyDev, and it's not using the configuration machinery in Anaconda. That's the reason they don't behave the same.