Running shell commands in Jupyter with the ! prefix: capture output, avoid injection, and know the limits

Quick answer

\n

Prefix any shell command with ! in a Jupyter cell, and the IPython kernel will pass it to the operating system. You can capture the result into a Python variable—out = !ls—and then inspect out.stdout, out.stderr, and out.returncode. This lets you inspect files, run pipelines, or invoke tools without leaving the notebook, but the command runs with the kernel’s user privileges and inherits its environment, not your activated virtualenv.

\n

How the ! prefix works

\n

When IPython sees ! at the start of a cell input, it strips the mark and sends the remaining string to the system shell via subprocess. The kernel captures stdout as a string and routes stderr to the cell output area. If you assign the command to a variable, IPython returns a special object with .stdout, .stderr, and .returncode attributes, so you can inspect the result programmatically rather than just printing it.

\n

Worked example: listing recent files

\n

out = !ls -t | head -n 5

\n

After running the cell, out.stdout will contain the first five lines sorted by modification time, and out.returncode will be 0 if the pipeline succeeded. If you only want the text, print out.stdout or assign it to a regular Python string with files = out.stdout.splitlines().

\n

Checking the return code

\n

\n
1. Assign a command and inspect the attribute: out = !false; then assert out.returncode == 1 to confirm a non-zero exit status.
\n
2. Without assignment, the cell prints the return code at the bottom of the output area.
\n

\n

Limits and common mistakes

\n

\n
• Shell injection: building ! commands from unsanitized user input can execute arbitrary code. Always validate inputs or prefer subprocess.run with a list when working outside notebooks.
\n
• Environment isolation: the shell inherits the kernel’s environment variables but not necessarily your shell’s PATH or active conda/venv. If a command isn’t found, add its full path or adjust the kernel’s environment.
\n
• Cross‑platform differences: Windows notebooks default to cmd.exe, which handles pipes and redirection differently. Use %%bash or %%cmd magic commands to force a specific shell, or stick to ! with syntax compatible both places.
\n
• Output buffering: large outputs may truncate in the notebook UI. Redirect to a file (!cmd > /tmp/out.txt) for later inspection.
\n

\n

Practical verification

\n

\n
1. Open a Jupyter Notebook and create a new cell.
\n
2. Enter out = !echo hello and execute; verify that out.stdout contains hello.
\n
3. Test a pipeline: out = !ls -t | head -n 2; confirm out.stdout has two lines.
\n
4. Check return code: out = !false; assert out.returncode == 1.
\n
5. On Windows, prefix the cell with %%bash and run out = !ls -t | head -n 1 to use bash syntax.
\n