[SOLVED] POST for data from RESTful API returns valid string when using CURL, but "decoding failed" when using fetch

Issue

I have a simple expressjs proxy server relaying requests to a RESTful API and relaying the responses from that API back. If I use CURL to access it, like so:

curl -s "http://localhost:3000/api/findOne" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{ /* request body */ }'

then I get valid JSON back to STDOUT.

But if I try to fetch it from the JavaScript front end like this:

fetch("http://localhost:3000/api/findOne", {
  method: "POST",
  headers: { "Content-Type": "application/json", Accept: "application/json" },
  body: JSON.stringify(/* request body */),
})
  .then((res) => res.json()) // .text() doesn't work either!
  .then((res) => console.log(res))
  .catch((e) => e);

then I get this error returned from the catch block:

The worst part is, I have the proxy server printing out the requests it's receiving and the requests it produces to pull from the actual RESTful API, and I did a diff between the output it produces when the curl call is made, and the output it produces when the fetch call is made, and they're identical, character for character. Which means that the proxy server is doing the exact same thing each time, there's just something wrong with fetch that makes it unable to parse the result, even as plain text, which is bizarre.

I haven't been able to find anything on the internet about this at all, can someone clue me in? Thanks. I'm a bit rusty with my web development skills (been out of the loop for a couple years).

Edit: Okay, so I've gotten a little further along: it seems that what's happening is that the output I'm getting from the proxy server is truncated, like, it just stops in the middle of the output stream, compared to the raw output from the API endpoint. Maybe that's making it impossible to decode? Even then, it should be decodable as text though, since curl does it, yet it seems JavaScript can't do that. Also, the decoding error happens even on outputs from the proxy server that don't at least visibly truncate. Here's the (slightly stripped down) server code:

import express, { Express, Request, Response } from 'express';
import dotenv from 'dotenv';
import bodyParser from 'body-parser';
import cors from 'cors';

dotenv.config();

const app: Express = express();
const port = process.env.PORT || 3000;
const apikey = ...;
const pathsToProxy = '/api/';
const MONGODB_URL = ...;

const corsOption = {
    credentials: true,
    origin: ['http://localhost:8080', 'http://localhost:3000'],
};

app.use(cors());

app.use(bodyParser.json());

app.all('/api/*', async (req: Request, res: Response) => {
    const url = req.url.replace(pathsToProxy, MONGODB_URL);

    let headers = new Headers();
    headers.set('Content-Type', req.get('content-type') || 'application/json');
    headers.set('Accept', req.get('accept') || 'application/json');
    headers.set('apiKey', apikey!);

    const mongoReq = {
        method: req.method,
        headers: headers,
        body: ['PUT', 'POST'].includes(req.method)
            ? JSON.stringify(req.body)
            : undefined,
    };
    try {
        const mongoRes = await fetch(url, mongoReq);
        mongoRes.body?.pipeTo(
            new WritableStream({
                start() {
                    res.statusCode = mongoRes.status;
                    mongoRes.headers.forEach((v, n) => res.setHeader(n, v));
                },
                write(chunk) {
                    res.write(chunk);
                },
                close() {
                    res.end();
                },
            })
        );
    } catch (e) {
        console.log(e);
        res.status(500).send(e);
    }
});

app.listen(port, () => {
    console.log(`[server]: Server is running at http://localhost:${port}`);
});

I don't see anything obvious here that would lead to truncated output.

Solution

Simply call .json() on response object to get proper body.

I don't see useful reason to chunk data which is returned as whole json from mongodb proxy api.

Solution is following:

     try {
        const mongoRes = await fetch(url, mongoReq);

        const status = mongoRes.status;
        const body = await mongoRes.json();
        
        // I don't recommend to resend headers
        // since it may return different 
        // content-length, content-encoding and etc. fields
        // which may be incorrect behavior
        // I'm commenting it in my answer,
        // but if it's necessary, 
        // then better extract only needed headers
        //
        // res.set({
        //   ...mongoRes.headers,
        // });

        res
          .status(mongoRes.status)
          .json(body);
     }
     catch (error) {
         console.error(error);
         res
           .status(500)
           .json({
             message: error.message,
           });
     }
          

P.S. "valid string" returned from CURL does not mean that it's valid json.

Answered By - num8er

Answer Checked By - Mary Flores (WPSolving Volunteer)