RAML 1.0 fragments not useful ? Can't share code


#1

After some experience attempting to refactor larger RAML files into re-usable bits I am stuck. Only Library seems usable in non-trivial examples. How does one solve the apparently simple problems below?

An API has the following:

resourceTypes:
  readOnlyCollection:
    get:
      200:
        body:
          application/json:
            type:
              properties:
                totalCount?: integer
                items: [<<itemType>>]
  collection:
    type: { readOnlyCollection: { itemType : <<itemType>> } }
    post:
      201:
        body:
          application/json:

etc. Note before we try and refactor this into ResourceType files, we already have an awkward parameter redefinition situation. But at least we are DRY.

If we try and create a resourceType/readOnlyCollection.raml file as #%RAML 1.0 ResourceType we have no issue.
But how would one create a resourceType/collection.raml file as #%RAML 1.0 ResourceType ? It can not reference the type it is trying to extend in the other file. The type readOnlyCollection can not be imported into collection.raml.

My conclusion so far is that #%RAML 1.0 ResourceType is only useful for ‘leaf’ resource types that do not reference anything externally – but also only for ‘top’ nodes that are not expected to be referenced by others since they can’t be used from most other file types.

The same problem happens with #%RAML 1.0 DataType – even worse as composing or extending types is more common than extending resourceTypes.

types/role.raml:

#%RAML 1.0 DataType
# types/role.raml
displayName: Role
properties:
  id: integer
  description: String

types/user.raml:

#%RAML 1.0 DataType

# no way to define / import the Role type!
displayName: User
properties:
  id: integer
  roles: [Role]

But wait, what about Libary types and includes? Well you can use these to access external definitions. But it seems nonsensical to even have the ResourceType and DataType file types if this is the answer. So I have to move Role into a Library to use it from User? That implies that i absolutely should also move User into one, because others may need to use it. Library is viral. As far as I can tell, everything should be a Library because nothing else supports modularity. So why do the others exist?

Or else I’m missing some way to include / reference DataType definitions from other DataType files, or ResourceType files in other ResourceType files.

If I’m not missing something, why not remove the DataType and ResourceType (and similar) flavors from the spec? They just take up space and aren’t useful in the real world unless they can be consumed by each other. In other words, I have a hunch that in short time, the best practice will be "never use DataType or ResourceType, always use Library".


#2

Some examples of things I wish worked:

#%RAML 1.0 NamedExample

This doesn’t work in API workbench, and there are no examples of this in the raml-examples git repo. At least I generally don’t have to share examples outside of the type definition itself so this probably won’t impact DRY-ness.

The spec has fragment includes, but the API workbench does not work with them. I’m not sure what parsers do:

#%RAML 1.0
title: Example API
version: v1
resourceTypes: !include patterns/resourceTypes.raml
#%RAML 1.0
# This file is located at patterns/resourceTypes.raml

collection:
  get:

( https://github.com/raml-org/raml-spec/blob/master/versions/raml-10/raml-10.md/#resolving-includes )

You can have custom parameters in a resourceType in a root RAML document, but not in any Library or other include, as far as I can tell. This is an impediment to DRY.

this works:

#%RAML 1.0

title: Simple API
version: v1
baseUri: http://foobar.com/simple/{version}
resourceTypes:
  collection:
    get:
      responses:
        200:
          body:
            application/json:
              type: <<myType>>[]
types:
  Foo:
    properties:
      id: integer
      name: string
  Bar:
    properties:
      id: string

/foos:
  type:  { collection: { myType : Foo } }

/bars:
  type:  { collection: { myType : Bar } }

but it does not if you try and extract the collection into a #%RAML 1.0 ResourceType to share with other apis. The parser complains that external types are not allowed.

UPDATE: The above “external resource types are not allowed” is hit or miss in API workbench. I have it working in one example and failing in another nearly identical one.

UPDATE2: The above does not work consistently, although the API workbench will parse the extracted ResourceType with a parameter, it won’t work as a Library. The inconsistency is killing me. I can use parameters from the outside with ResourceType but can not with Library. Unfortunately I can’t use the former because I can’t inherit the read-only-collection from the read-write-collection if it is not in a Library.

UPDATE3: Inconsistent results. I think most of this is bugs in API Workbench. One of my previously failing examples now works, with no change.


#3

I spent so much time struggling with api-workbench and libraries that I almost gave up on RAML. I agree that it is terribly frustrating.


#4

Keep in mind RAML 1.0 is still new, and work is ongoing to implement the tooling for API workbench, as well as parsers in some languages. It takes time after a spec is introduced for tooling to appear. Case in point, Java EE (pick your version) comes out a long time after the JDK version it is built on, and vendors take several months after the Java EE spec is released to update their products to support it.

So dont give up on RAML because the tooling is not mature yet. It is a work in progress and posts like this help those working on the tools to identify issues.


#5

I’ve been struggling with RAML 1.0 support since November. This excuse is getting old very quickly.


#6

To be fair, RAML 1.0 was finalized only a few weeks ago. Prior to that it was in Beta, and in my experience pretty much everything in beta comes with one of any number of clauses about support, production use, etc.

What are you in particular struggling with at this time? There have been almost weekly updates to API Workbench… have you been installing the updates and seeing if that resolve an issue you may have come across? If not, did you look for or file an issue concerning it on the API Workbench Github project?


#7

Resolving whether issues are spec or workbench can be tricky. I’m not sure if any tools are done and can be trusted. RAML IMO is NOT 1.0, as no spec can be truly done without an implementation or two to find the rough edges. The examples in the examples repo are nice, but it took me just a few hours to run into use cases not covered by them.

I’m trying to build tools on top of RAML 1.0 I’m using the workbench to validate that I’m doing things right, as the spec has gaps and ambiguities. I’ll have to wait.

Filing a bug like “Workbench sometimes produces errors intermittently in areas not covered by the spec” — in my experience developers will just ignore that and ask for a reproducible case which I don’t have.

Yes, I’ve been applying the updates.


#8

Hi @scottc, thanks for your feedback. I think that might be very helpful to improve the specification.

Some things that I have noticed going through the different examples, you described.

  1. Reusing data types

The way how you can achieve that is by including the actual fragment. Please remember that fragments are snippets that you can easily include as you did in 0.8 before. This time, you just give a name to that fragment to get some design-time validation. So for example:

 #%RAML 1.0 DataType

# no way to define / import the Role type!
displayName: User
properties:
  id: integer
  roles: 
    type: array
    items: !include role.raml
  1. The problem with ResourceTypes and other fragments

I do agree that it seems these are not useful in any sense, but I’d say you are right that they can be still very helpful for any leaf nodes or even in libraries.

#%RAML 1.0 Library

resourceTypes:
  readOnlyCollection: !include resourceTypes/collection.raml
  collection: 
    type: { readOnlyCollection: { itemType : <<itemType>> } }
      post:
        201:
          body:
            application/json:

Using fragments in parallel with libraries might still not be perfect, but the problem here is that any fragment is nameless, and you need to name them before using. If you have any idea on how you would like to have that, let’s discuss the possibilities. RAML is an open specification and very much relies on the community right? So let’s use the opportunity to make necessary and important suggestions.

  1. Example including a collection

Fragments are meant to be a single node such as a single example or single resource type, not multiple. So in your example, you should remove #%RAML 1.0 from the patterns/resourceTypes.raml as the parser would validate that as a root RAML file which requires the title: node. That file will be just a normal YAML file not being parsed until you bring it into its context (here its resourceTypes: !include patterns/resourceTypes.raml)

  1. Last example

That example

#%RAML 1.0

title: Simple API
version: v1
baseUri: http://foobar.com/simple/{version}
resourceTypes:
  collection:
    get:
      responses:
        200:
          body:
            application/json:
              type: <<myType>>[]
types:
  Foo:
    properties:
      id: integer
      name: string
  Bar:
    properties:
      id: string

/foos:
  type:  { collection: { myType : Foo } }

/bars:
  type:  { collection: { myType : Bar } }  

can easily be

#%RAML 1.0 ResourceType
# collection.raml

get:
  responses:
    200:
      body:
        application/json:
          type: <<myType>>[]

#%RAML 1.0 DataType
# foo.raml

properties:
  id: integer
  name: string  

#%RAML 1.0 DataType
# bar.raml

properties:
  id: integer

#%RAML 1.0

title: Simple API
version: v1
baseUri: http://foobar.com/simple/{version}
resourceTypes:
  collection: !include collection.raml
types:
  Foo: !include foo.raml
  Bar: !include bar.raml

/foos:
  type:  { collection: { myType : Foo } }

/bars:
  type:  { collection: { myType : Bar } }

Not saying that it is THE solution, and there is room for improvement, and that’s where the community can help. Same for the tools. RAML 1.0 is supported by Mulesoft already in their various tools and also the API Workbench. There are a lot of people that are already planning their update as well since the spec is now in a very stable state. To @rojocapo point, RAML 1.0 was a release candidate in November, and there was only one tool; the API Workbench. That was on purpose since we had a single platform to show our community what we are working on and give them a chance to very early influence the direction. And that was a full success since we did another release candidate in April, and the GA in May. Until GA, I would not have expected to see any other support for a release candidate version that supposed to change. That is different now.

Hope that clarified some points. Maybe not all though :stuck_out_tongue:


#9

Hi Kevin,

I have been using api-workbench since the first beta came out in November. I have used every single api-workbench update and sometimes even manually updated parsers from Github when the parser included in api-workbench was borked. As you can imagine this has been very improductive and frustrating, full of regressions. To be honest if a customer hadn’t been paying for my time, I would have given up and switched to something else.

Currently my biggest problem is that one of the consumers of my APIs (one of the top 5 fuel retailers in the world) says that they want API documentation and JSON schemas, and are not willing to work with RAML format directly. They say “write the APIs using whatever tools you want, just give us API documentation and JSON schemas” which sounds pretty reasonable if you ask me.

RAML2HTML (raml1.0 branch) can’t even compile a 10 line single file example, much less create documentation for my 50 file projects with includes, libraries, etc. Regarding JSON schemas, I don’t think there is a way to generate JSON schemas automatically. Both these features were supposed to be available by now according to answers in theses forums.

I really like the language now that it’s a bit more stable and I’ve gone through the learning curve, but without documentation and JSON schema generation we’ll need to look for other alternatives.


#10

Thanks for clarifying. I cant fault your need to find something sooner if the timeline for some of the projects that will eventually meet your needs is too far out (or unknown as the case may be). RAML2HTML I believe is being worked on as you noticed, but as it is a community project, I suspect the process is slow going and will take some time to fully utilize all of RAML 1.0. Likewise, a project I am potentially considering doing for myself is a RAML2JAXRS for RAML 1.0. The current flavor hasnt started towards a 1.0 implementation yet, and I like the ability to generate both client side and server side components from my RAML, allows me to mock/test APIs very rapidly. There are other ways possibly, but I currently use RAML 0.8 with the RAML2JAXRS and it has worked very well for my needs. I am guessing you may not have the time to jump in and help with the effort, but there is always that option as most of these projects are community driven and are always looking for others to fork/pull to help if possible.

Given the other options, I personally wouldnt give up on RAML to find other options. I dont think anything else compares in terms of designing the APIs, as well as the tooling to work with another format to generate what I would want from it. What might be good is a RAML 1.0 to converter, so you could benefit from better tooling while waiting for the RAML tooling to catch up.

I still liken what is going on with other products in the industry. JEE spec comes out many months before the typical first vendor releases their implementation of the spec. Usually the work is going on while the spec is being defined as well, as has been the case for some RAML 1.0 tooling since the first beta spec was announced. I suspect there are many individuals and companies working on RAML 1.0 tooling that havent announced anything as well. At some point we will see a good set of tools that support RAML 1.0 and everyone will benefit.


#11

@rojocapo I understand your frustration as you have been one of the very early adopters around the RAML 1.0 concept and been through the same stages we have been through which were often painful since there has been a lot movement, stabilizing exercises, and more. I agree that there is a gap right now that people in the community try to close, and I am not sure what you mean that raml2html cannot convert a 10 line RAML 1.0 file since I have many working examples already. Anyways, there are people that drive all that and do their best to provide people with the necessary tool set. One being Mulesoft, which updated their designer and console already to RAML 1.0, published the workbench to play around with the spec, and there is more in their pipeline. Others I can see as well that update their parser, documentation generators, building new ones as well, and all that started a couple of weeks ago when the RAML Workgroup finally released RAML 1.0 GA.

I agree that there had been some posts and promises (even from me) around some of the tools, but every single person was very much focused on providing the foundation for so many other tools in the last couple of months. The foundation being first the spec and then the parsers. It was a crazy ride, and no one would have expected that we get so much feedback that we had to spend more time improve the specification and the parser before anyone is able to build tools. Every single person in the community helped us here and we will always be grateful for that.

Now it is about them again right? Most of the projects had been created by the community and we need to give them time to adapt to the changes. To make that even easier the JS parser team went back to the JSON object representation that was widely adopted to this point and we still improving that to help people that want to have RAML 1.0 support. Obviously, and to Kevin’s point, all the projects are driven by a community and there is only limited time for one person. These people always appreciate help with their projects in any cases. There is a lot open and if you have the skills, help us :wink: You need documentation and you know about javascript and HTML, support us on the raml2html project. Maybe there is more you can help the community grow. Be part of such an amazing bunch of people that already give everything. :wink:

I love you all and continue with the hard work! :heart_eyes: :smiley:


#12

Christian,

I will give raml2html another try. Could you update raml2obj branch ram1.0 to use the latest parser? I thinks it’s currently set to use 0.2.14 which is obsolete.

If you know of another documentation generator that will accept RAML 1.0 files, please let me know. I don’t have a problem paying a subscription.

Thanks


#13

Sure! I will update that next week since I am on the road. Sorry mate


#14

@scottc well said. I’m struggling with this right now.

But wait, what about Libary types and includes? Well you can use these to access external definitions

But even then, I still can’t compose a type from external types, right? E.g. type: (ExternalType | << typePassedIn >>)[]. I’ve tried multiple ways of including external types in my library but none seem to work. It seems my only option left is to defining all of my types in one library.

Resolving whether issues are spec or workbench can be tricky. I’m not sure if any tools are done and can be trusted.

I agree. I’ve given up on other tools and use RAML2HTML to check what I’m doing, but I don’t trust it 100%. There really needs to be a solid validation tool at least, given the learning curve.


#15

Hi @adam,

Sorry to hear that you are struggling with the existing tools. Let’s see if we find a way to improve that.

I’ve given up on other tools and use RAML2HTML to check what I’m doing

What tools did you use before? When you say “to check” you mean validate, correct? So you are using raml2html to validate if a RAML file is correct?

But even then, I still can’t compose a type from external types, right?

Have a look at this example and let me know if that would work for you!


#16

Hey @christian_vogel,

I used API workbook, API designer, and more. It seemed like these tools were complaining about stuff the spec said was valid, didn’t resolve relative paths, etc. I wrote a good bit of it at first without any tool, just the spec, so once I tried these tools, I saw a lot of cascading errors (i.e. if it didn’t resolve a relative path there would errors in multiple files). Of course, some of what I had wrote was incorrect as well.

Yes I’m using RAML2HTML to validate as well as see the output HTML. I’m finding it better but of course there are times I wonder if I’ve written something incorrectly or if RAML2HTML / the template doesn’t support it. For example, I was using a type inside body.application/json to define common response keys but this seems to break things (the HTML output at least). With it, none of the properties were being shown in the HTML, but without it, they are (although I need to spell out the same common response keys in each response so it’s not as DRY but it’ll do).

No, that doesn’t cover what I need. The example I gave is more complicated; type: (ExternalType | << typePassedIn >>)[].

P.S. I’ve just spent the last day refactoring everything into a single library and it seems to be working a lot better.


#17

We are actually already thinking about better ways to reference without using libraries, but that might need some substantial changes how things work in RAML in general. Nothing concrete yet, but just that we know about that and try to find a good alternative.

Problems with the design tools are strange, to be honest. Do you have any more concrete example that I can evaluate?


#18

I don’t have time right now to set stuff up in API designer again but here’s an example you gave me on the API Designer Gitter:

rt.raml:

#%RAML 1.0 ResourceType

get:
  is: [ paging ]

api.raml:

#%RAML 1.0 
title: Test

resourceTypes: 
  collection: !include rt.raml

traits:
  paging: !include paging.raml

/resource:
  type: collection

And you said:

in my above example, you might get an error in the designer since its parsing the fragment and does not know what “paging” is, but in the context of your root RAML, it is valid


As an aside, maybe there should be a clear onboarding flow for RAML 1.0 in which devs have to look over some non-trivial examples before diving in. It would’ve helped me. I’ve been looking through raml-examples but it’s not perfect and aslo found the world music API example here useful: https://github.com/raml2html/default-theme/tree/master/examples.


#19

Good feedback. You are always invited to add more example that you think might be useful of course. The world music example in raml2html is actually the one from the raml-examples repo :slight_smile:

The thing with fragments is, if you use identifiers that are not known to the parser; it will show you error messages of course. Nothing really we can do. The intention around libraries is also to introduce identifiers for fragments so that you can reuse them in other fragments as well. It’s the nature of using typed fragments. What you can do is to remove #%RAML 1.0 ResourceType and transform it to a non-typed fragment that we don’t validate. Hence, you will not see any errors.