[SOLVED] Google Cloud Ops Agent Mongo Integration Error - AuthenticationFailed: SCRAM authentication failed, storedKey mismatch

Issue

I'm getting this error when setting up the 3rd party integration for Mongo with Google Ops Agent:

AuthenticationFailed: SCRAM authentication failed, storedKey mismatch

Google Cloud Ops Agent Config

Mongo Integration

    mongodb:
      type: mongodb
      insecure: false
      cert_file: /etc/ssl/mongodb.pem
      key_file: /etc/ssl/mongodb.pem
      ca_file: /etc/ssl/mongodb.pem
      endpoint: 127.0.0.1:443
      username: opsplease
      password: Williwork$
      collection_interval: 60s
      insecure_skip_verify: true

The Full error:

"jsonPayload": {
    "context": "conn4277",
    "severity": "I",
    "attributes": {
      "authenticationDatabase": "admin",
      "principalName": "user",
      "mechanism": "SCRAM-SHA-256",
      "client": "1.1.1.1:34542",
      "result": "AuthenticationFailed: SCRAM authentication failed, storedKey mismatch"
    },
    "id": 20249,
    "message": "Authentication failed",
    "component": "ACCESS"
 },

I'm using a config that works on other instances, and validated the user and password are working and also have permissions needed to the database.

I can't find any useful answers relating to this error, but it does seem to be fairly common outside of the Google Ops Agent. Any insight or suggestions would be appreciated.

Solution

The problem was actually the password itself consisting of characters that the Ops Agent was not able to handle without escaping.

No other system had an issue with this password including Robo3T or the mongo shell itself connecting to the replication set. I had tried putting the password string in quotes but that did not fix it either.

The character that caused the issue so far is $. It needs to be escaped:

password: Williwork\$

How the string will look from mongoshell.

mongodb://opsplease:Williwork%24@server1:443,server2:443,server3:443/?authSource=admin&replicaSet=rs0&readPreference=primary&ssl=true&tlsAllowInvalidCertificates=true&tlsAllowInvalidHostnames=true

After more testing, I found that you cannot use percent encoding in the config.

password: Williwork%24

If you try this, it won't show the same error in any logs but the metrics data will also not show up which is arguably worse than having it throwing errors.

I looked all over for solutions to this, and nothing was relevant to the relatively new Ops Agent , so marking my own question as answered for anyone who may find this in the future. I wish there was a log from the ops agent that would show it choking on the value parsing, but the /var/log/messages and /var/log/google-cloud-ops-agent/subagents/logging-module.log do not have anything helpful for this issue.

Answered By - Edwrd_T_Justice

Answer Checked By - Terry (WPSolving Volunteer)