Getting hands dirty with SOAP

So yeah, I happen to come across an interesting problem of getting my hands dirty with SOAP. SOAP was the king of APIs, well that was before REST API came along. If you found this while trying to connect to a SOAP Web Service, let this be a encouragement for you, you will get over it (eventually).

Simple Object Access Protocol (although not very simple when you start on it) is how the APIs were defined back in the day. SOAP Services and Clients were written in Java and C# and as these languages were popular back then they provide very good support in creating clients, letting the developer to traverse the service and ultimately do build what you want. But with present day languages like NodeJS and Python, this proved a little bit difficult when starting out.

The problem I faced was, when trying to connect to a SOAP Web Service through NodeJS. The library of choice was soap and the next in line was strong-soap.

Prior to actually connecting to the “desired” web service it was time to get familiarity with some sample web services that were publicly available.

Basic Understanding of a SOAP Web Service

If you are familiar with REST APIs, the most popular way of communicating the schema is through OpenAPI specification (Swagger). SOAP Web Services use a WSDL to communicate the specification for the services to integrate.

WSDL

WSDL (Web Service Definition Language) is a XML document that describes the services and operations (you can think of them as resources and methods in REST). WSDL can have references to the data types/schemas for these operations in the same file or different files called XSD (XML Schema Definition).

Following image relates SOAP terminology to REST.

Some publicly available SOAP API: https://www.postman.com/cs-demo/workspace/public-soap-apis/request/8854915-c9b8614d-2f25-4eb5-9b39-3715b0d04992

SOAP Message

SOAP Web Services communicate using what is called a SOAP Message which is a XML document. See the example below.

<?xml version=”1.0"?>

<soap:Envelope
 xmlns:soap=”http://www.w3.org/2003/05/soap-envelope"
 soap:encodingStyle=”http://www.w3.org/2003/05/soap-encoding">

 <soap:Header>
 …
 </soap:Header>

 <soap:Body>
 …
 <soap:Fault>
 …
 </soap:Fault>
 </soap:Body>

 </soap:Envelope>

SOAP Envelope is the outer most node and it contains the over all object within it. Main sub nodes of a SOAP message are Header and Body and optionally in case of error you would get Fault. Any SOAP message that you will be sending to the service should be of this format.

Namespaces

You got to have the namespaces correct for everything to work.

In the above example SOAP message “soap:Envelope” which is the outermost element or node, separated by a colon, contains two parts.

“soap”: This is the namespace. Namespaces play a significant role. The namespaces should be referenced in the XML element attributes. In the above SOAP message xmlns:soap=“http://www.w3.org/2003/05/soap-envelope”, “soap” namespace is referenced to a URL. This helps the Web Service locate/verify the element: “Envelope”.

A rule of thumb would be if there is any namespace added to element it should be properly referred to in XML attributes either in the same node or a parent node.

Header and Body

I am not going to talk in depth about the Header and the Body.

Header (Similar to headers in HTTP? Yes and No, will come to that later) But briefly, Header is where you would have Credentials, TimeStamp, Signatures and other mandatory headers. Header contains the information for the web service to do initial validations of your SOAP message.

Body is where your payload would be. Payload will be encoded in a XML format in a way that the web service understands it.

Authentication

Authentication is important in any service let it be REST or SOAP. But SOAP is known for its security. So it is a must that you have the authentication right for it to work. Similar to REST APIs, SOAP works with following Basic Authentication, Bearer Authentication, SSL Certificates.

However, SOAP has a defined Security specification WS-Security which imposes stricter rules on how to authenticate. One main way is to use X.509 Certificates for Signatures. This is the method that was used in our use case. Don’t consider this article as a guide on how the signing happen, just a brief guide. Spec here: https://groups.oasis-open.org/higherlogic/ws/public/download/16790/wss-v1.1-spec-os-SOAPMessageSecurity.pdf/latest

Here a digest is generated using a digest algorithm which would be defined in the digest itself, for SOAP Body element and other items such as timestamps separately. Each of them will be referred to the digest using a Id.

Once all the digests are done, the collection of the digests are hashed again using the private key. On the server side this would be decrypted using a public key and compared against the payload.

Nothing is working

Just glancing through the documents on the internet, I started coding the client to connect to the SOAP service with NodeJS NPM package. Provided the WSDL files, created a client, called the operation … “No Service Configuration” and it says a client error. I never expected the code to work on the first instance, but the assumption was that it would work with few tweaks to the library configurations. Fast forward 4 days we were stuck in the same place.

Alternatives

There was a second NodeJS package https://www.npmjs.com/package/strong-soap that looks very similar to the previous, so I tried with this library.. still nothing.

SOAP UI to the Rescue

SOAP UI proved to be of immensely helpful to get started. The client had provided a SOAP UI Project file that we could easily load to SOAP UI and voila … it works like magic. We were able to connect to the service and receiving the response. I was able to look at the XML SOAP message that was working. This was our ticket to the party.

So we had working SOAP message for comparison and the one that was generated by the package. They were different on multiple levels.

By this time I had looked at the soap and saw that it was using axios to send the SOAP message. I used the working SOAP message from SOAP UI and sent the request using axios directly. It worked. So our problem was now with the soap package.

Debuggers Assemble!

Initial thought, compare the SOAP messages and figure out the differences. So I did just that. Turns out the soap package was adding extra namespaces, missing some namespaces, missing soap headers. The response from the service was not helpful as it was pointing to a Service element missing issue even when we had that in place.

So we started doing all sorts of SOAP Client configuration changes, WSDL configuration changes. List of things we tried to make our SOAP message closer to the working example.

1. Used overrideRootElement to make root element the same.
2. Used ignoredNamespaces to get rid of unwanted namespaces. (Problem was it was getting rid of a required namespace entirely in the SOAP body, see below)
3. Override namespaces in the body using {':name': 'value'} instead of just {name: 'value'} But still we could not get one of the namespaces to show in the XML attributes. so… we hard coded it in the JSON object {':name xmlns=http://namespaces.com/somenamespace': 'value'}
4. Used signerOptionsto setup the SOAP header in WS-Security.
5. Added missing headers with addSoapHeader (SOAP headers had a namespace “a” (“wsa”) which refers to WS-Addressing which was not present in the SOAP message, we added the header fully from the working SOAP message)

Still it was not connecting.

The Culprit

With all the above changes we were still missing one namespace which is http://schemas.xmlsoap.org/ws/2004/08/addressing. Stackoverflow suggested few ways to add, but this (not recommended as we call a private method of the npm package) is the one that worked for us.

client['wsdl']['xmlnsInEnvelope'] += 'xmlns:wsa="http://...."'
client['wsdl']._xmlnsMap();

and.. Voila
It worked.
Final Words

If there is one thought that went through my head over and over again is, “How priviledged are we to use REST APIs now a days”. While REST APIs are the most popular, there are bigger organizations that still use SOAP APIs as their offering. SOAP is known for its security. So it still is a valid option for organizations who prioritize on security.

Cheers, and that’s how I got my hands dirty with SOAP.