Generic automated test that verifies your service follows a RAML document


#1

I would like to know if there are any plans or at least interest to have a generic automated test that verifies your service follows a RAML document that does the following given a base url for my deployed service implementation:

  1. Is language agnostic (my service implementation can be node, java, scala, ruby, anything)
  2. Verify that each resource defined in RAML exists in my service
  3. Verify that each endpoint (by http method) defined in RAML exists in my service
  4. Verify that each endpoint url params defined in RAML are supported in my service
  5. Verify that each endpoint request HTTP headers defined in RAML are supported in my service
  6. Verify that each endpoint request body defined in RAML is supported in my service - verify by validating the JSON schema
  7. Verify that each endpoint response HTTP headers defined in RAML are supported in my service
  8. Verify that each endpoint response body defined in RAML is supported in my service - verify by validating the JSON schema
  9. Be able to identify the RAML API contract version and verify my service not just against the current contract/RAML version, but also all previous minor versions to ensure backwards compatibility (only new endpoints or new and optional params, headers or json) - I know this is a bit of a stretch, but if the first 8 points are done this last one is based on them

#2

Definite interest, but I’m not yet sure exactly when it’s coming. That said, It’s definitely important to get one or more testing frameworks out there in the open-source RAML community, and I think we’ll figure out a way to get one released in the not-too-distant future. As to the specific features, AFAIK no-one in the community has specced something out yet - this is a great start.

What else do you look for in testing frameworks; should it integrate with CI? BDD? I’d love to get people’s thoughts!


#3

Yes, this is something I’m very interested in. Dredd makes API Blueprint very appealing, though I much prefer RAML’s robustness and flexibility.


#4

I see this more like a test tool that you can run standalone and be able to integrate with CI.
While I like BDD I don’t know how an automated test tool will integrate with BDD. It can use it under the hood, but the goal was to not have to write any tests (BDD or not) to verify that written contracts are followed - only the contracts are loaded and then verified against a URL.


#5

I know about Dredd, that’s why for now even though my team likes RAML more we are using Blueprint. However, once RAML has contract test support then we would like to switch to RAML since as a contract/model seems better and it has a few JVM based tools and we are a JVM based team.


#6

I have found that API console is quite good starting point to test if the API is working. After updating raml/api implementation you just press “paly” and review the results. Really great job, guys!

However, I was thinking is there any way to validate responses according to JSON schema? It would help to automate some manual tasks (like visual response inspection:) )


#7

Are you familiar with SoapUI?
It definitively could help in testing and automation tasks.

There is an existing RAML plugin for it http://olensmar.blogspot.se/2013/12/a-raml-apihub-plugin-for-soapui.html

Let me know if this helps.
Cheers!


#8

Unfortunately in the shortcomings section they are stating that no schemas from RAML file can be used.

What I am looking is only way to assert response with the schema and output the result. This would be much more convenient than desktop application.

Is there any way to include libraries from the web into the console?


#9

Hey Dagaz,

Including libraries directly isn’t supported, but you are certainly welcome to fork the project (it’s open source), make changes, and contribute back to the console.

The API Notebook allows you to include libraries directly (as it’s got full JS support), and might be able to fill the testing gap for you for now.

As an FYI - testing tools/validation frameworks for RAML are top of mind, and I hope to have a better sense in a few weeks about when we might be able to release a solution closer to the Dredd use-cases.


#10

Another interesting validation requirement would be to ensure that, for response bodies with both ‘schema’ and ‘example’ attributes, that the examples pass the validation schemas.

I suppose this is sort of a different category of validation, in that it’s validating the RAML document itself against itself, for internal consistency.


#11

+1.

Dredd like auto testing tool will strengthen RAML


#12

You can have a try of ramlev, it will validate all examples defined in RAML against jsonschema


#13

You can try Abao, automated API testing tool for RAML


#14

Thanks again @quanlong! :slight_smile:


#15

You can try raml-tester.
It hooks into integration tests and verifies that all test requests/responses are compliant to a RAML.
It reports if the tests cover all resources/headers/params etc.
It’s written in Java, but it provides a proxy so it can be used from other platforms.