Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

aws emr --steps fails: Error parsing parameter, Expected '=', received 'EOF'

ErrorError parsing parameter '--steps': Expected: '=', received: 'EOF' for input:

Short answer

The AWS CLI tried to read your option as shorthand key=value pairs and ran out of text where it expected an equals sign. Usual reasons: a space between pairs, a comma inside an Args value, or JSON whose quotes the shell changed. Remove the spaces, or put the steps in a JSON file and pass file://step.json.

  • Applies to AWS CLI v2, Amazon EMR, bash, zsh, PowerShell
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Move the steps into a JSON file and validate it. If the file is valid and the command works with file://, the problem was how the value was typed on the command line.

    python3 -m json.tool step.json
    aws emr add-steps --cluster-id j-XXXXXXXX --steps file://./step.json
  2. 2Match what you see, then jump to its fix

    1. 1The step was typed with spaces after commas or inside unquoted valuesWhite space inside the shorthand valueFix
    2. 2The step is a Streaming or Spark step with -files a,b or a comma-separated list inside one argumentAn Args value contains a comma, so shorthand cannot hold itFix
    3. 3Inline JSON that works in one terminal fails in another, especially on WindowsThe shell changed the quotes around inline JSONFix

What does the error look like?

  • Error parsing parameter '--steps': Expected: '=', received: 'EOF' for input:

Cause 1 of 3

White space inside the shorthand value

1Check

Shorthand is a comma-separated list of key=value pairs with no white space between pairs. A space after a comma, or an unquoted value that contains a space (such as a step name), splits the argument and leaves a key with no equals sign.

echo 'Type=CUSTOM_JAR, Name=My step,Jar=s3://amzn-s3-demo-bucket/mytest.jar'

2Fix

Remove the spaces between pairs and wrap any value that contains a space in quotes, as the AWS CLI reference does for Name='Streaming Program'.

aws emr add-steps \
  --cluster-id j-XXXXXXXX \
  --steps Type=CUSTOM_JAR,Name='My step',ActionOnFailure=CONTINUE,Jar=s3://amzn-s3-demo-bucket/mytest.jar,Args=arg1,arg2

3Verify

The command prints a StepIds list instead of a parse error.

aws emr list-steps --cluster-id j-XXXXXXXX

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 3

An Args value contains a comma, so shorthand cannot hold it

1Check

In shorthand, commas separate list items, so a comma inside a single argument is read as a break. The CLI reference shows a Streaming step with several files as a JSON-only example.

grep -n 'Args' step.json

2Fix

Write that step as JSON, with each option and each value as its own list item, and pass the file.

[
  {
    "Name": "JSON Streaming Step",
    "Type": "STREAMING",
    "ActionOnFailure": "CONTINUE",
    "Args": [
      "-files",
      "s3://amzn-s3-demo-bucket/mapper.py,s3://amzn-s3-demo-bucket/reducer.py",
      "-mapper", "mapper.py",
      "-reducer", "reducer.py",
      "-input", "s3://amzn-s3-demo-bucket/input",
      "-output", "s3://amzn-s3-demo-bucket/output"
    ]
  }
]

3Verify

Run add-steps with the file.

aws emr add-steps --cluster-id j-XXXXXXXX --steps file://./multiplefiles.json

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 3 of 3

The shell changed the quotes around inline JSON

1Check

Linux and macOS want single quotes around JSON. The Windows command prompt wants double quotes with each inner double quote escaped by a backslash. PowerShell has its own rules. If the quotes get stripped, the value no longer looks like JSON and the shorthand parser fails.

aws emr add-steps --cluster-id j-XXXXXXXX --steps '[{"Type":"CUSTOM_JAR","Jar":"s3://amzn-s3-demo-bucket/mytest.jar"}]'

2Fix

Skip the quoting problem: save the JSON to a file and load it with file://. A relative path such as file://./step.json works, and a path with spaces needs quoting for your terminal.

aws emr add-steps --cluster-id j-XXXXXXXX --steps file://./step.json

3Verify

StepIds is printed.

aws emr list-steps --cluster-id j-XXXXXXXX

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Keep EMR steps in version-controlled JSON files and always pass them with file://.
  • Validate the JSON file in CI before it is used.
  • Use shorthand only for short steps with plain values.
  • Run the same file from every terminal your team uses, instead of retyping quotes.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Apache has a certificate, but https:// does not open

No line with :443 in the output of ss -ltnp

HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.

4 causes, each with a fix

Servers

AWS CLI says you must specify a region

You must specify a region. You can also configure your region by running "aws configure".

The AWS CLI sends every request to one region and has none to use.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.