CouchBase: how can you make a filterable list in SpringData?

CouchBase filters

Sure it’s not Instagram filters; Couchbase filters are to be created here:

What is Couchbase?

Couchbase Server is a NoSQL document database with a distributed architecture for performance, scalability, and availability. It enables developers to build applications easier and faster by leveraging the power of SQL with the flexibility of JSON.

http://www.couchbase.com/nosql-databases/couchbase-server

Why do you need a generic way to make queries?

It is easy. Every CMS system needs filterable lists so the users can look at the entities they want. On the other hand, as a developer you don’t want to copy-paste a lot of boilerplate code. We are developing an application in Spring Data (Java8), and according to this stackoverflow question (thanks to Simon Baslé) we know that there is no Couchbase module yet. So basically Ill show you a way to build N1QL queries easily from dynamic parameters.

The CouchbaseQueryExecutor

Basically you want a controller function. Something like this:

userTaskService.getUserTasks(…) will give you the entities matching the params in the request’s query string, and additionally you can provide an optional pageable param to the service if you want to make a page with Infinite-scroll, Pagination, or something like that on the UI. So yeah, this is what we want, but should we build the query matching the requirements explained earlier? Let’s take a look at the layers.

Controller

The controller should transform the request’s query string into some generic object, so all of the services’ find function parameter lists can be the same. You should drop all the unnecessary query string params, and transform them into a map. This oneliner will do it for you:

This will ignore multiple query string params like ?param=asd&param=basd. You will only get the ‘asd’ with the key ‘param’ in the map. If you want to use multiple params with the same name (for example arrays) in your query string, remove this from the code. So the controller function now looks like this:

Your controller is done now. Let’s talk a bit about the other lines in this function.

userTaskMapper.userTasksToUserTaskDTOs(…) converts the entities to DTOs (because you don’t really want to send sensitive data to the client; like password, or the privilege to detonate printers… (smile) ).

PaginationUtil.generatePaginationHttpHeaders(cbpage) is responsible for generating the headers for our Infinite-scroll.

CouchbasePage contains information about the page such as totalElements, pageNumber, totalPages, size, and the data itself in a generic list.

Service

Let’s take a look at the UserTaskService. The service’s responsibility is to make filters from the request params (which is a Map<String, String> now). It can ignore fields if necessary, e.g. if there is a parameter named userId and the value is -1. You don’t want to filter negative IDs, because that would mean you want to give the currently logged user’s tasks back. In Java8 you can make each param transformation a oneliner with the following util class:

There are some useful functions. If you need any other generic transformation, you can implement your own. The functions do the following:

toJsonObject is the function that converts the map to JsonObject. The N1QL query function expects a JsonObject.

putIfNotEmpty – As simple as it seems. If not empty, puts the filter as a String. The type of the value is important, because N1QL is type-sensitive. If your filtered field’s type is not a string, then you should use

putIfNotEmptyAndApply – Same as putIfNotEmpty, but there is an additional method reference parameter. Before putting the filter into the map, this function applies the parser function.

putIfConditionAndApply – Of course ifNotEmpty is not the only condition that you want to implement. You can implement your own conditions in the CoucbaseFilterConditions class.

putCustom – Sometimes it’s not trivial whether or not you should put the filter. For complicated situations, there is the putCustom function. You can implement it with more parameters if you want. The last parameter is the most important, that is a method’s reference which contains the custom logic that you want to apply. Java8 has only Function and BiFunction functional interfaces defined, but if you want a TriFunction or so on, then you can easily define it by looking at Function and BiFunction’s source code.

As you see there is a CouchbaseFilterEntry class that helps the putCustom function. It returns the key and the value of a filter, and contains information about whether the filter is empty. Only not empty filter entries will be put in the filter.

Okay so now we have a ton of util classes implemented, but why is it good? Because you don’t have to copy-paste a lot of code. Let’s compare them a bit!

The defaultFilters function returns a Map of the filters that every query must contain, for example if you have organizations in your app, and you don’t want them to see each other’s data, then a default filter can be the currently logged in user’s organizationId.

The two code snippets above are doing exactly the same. The string parameters were extracted to constant variables. Ignore that a little bit and take a look at the number of lines. 51 vs 34. That means 33% of code has disappeared. And what if you want to add a new parameter? You don’t have to copy-paste the line getting the value from the parameter, only that one line of code. The if condition checks whether you should put the param as a filter. I hope you like it too (smile) Using this is pretty simple.

I’ll talk later about CouchbaseQueryExecutor.CONTAINS_FILTER, FROM_FILTER, TO_FILTER and others

Repository

Couchbase stores documents. To filter only UserTasks in a query, I recommend you to add a type variable to every Couchbase document, for example in a common superclass. After you have that field, the Repository shall add that filter to the filters, so the query will only return the specified documents.

The CouchbaseQueryExecutor

Now we have all our filters set up. We have params transformed from the query string to filters, we have default filters (if needed), we have type filter. All we need to do is compose the query, execute it, and we got our list of entities.

Earlier you saw the CouchbaseQueryExecutor.CONTAINS_FILTER for example. This is a postfix to the field name. The query executor will decide the kind of matching  you want to run by looking at the postfixes. We use the following postfixes in our app, but if you want, you can define anything you want.

As you see the CouchbaseQueryExecutor’s createExpression checks if the key has any known postfix, and creates the expression accordingly. If none of the postfixes match, then it will be a simple EqualsExpression. After this we can build expressions from filters. Now we need to concat the expressions.

The “$” + key is the parameter placeholder. It is the key in the JsonObject so it contains the postfixes. Property key is the key without a postfix.

This function will concat the expressions and add a special expression that’s needed to query the Couchbase server through N1QL queries according to this stackoverflow question. Also according to this you CANNOT update/delete/insert with N1QL. It is only for querying the data. If you need to do any of those operations, use the SyncGateway API.

Okay so now we have the ‘where’ section of the query. The following code will compose the remaining part of the query for you.

fromPageable is a function that will compose the order by section from the Spring Data Pageable object. Ordering has a postfix, too if you want to sort ignoring the letter case. If the client sends property_ignorecase sorting, then the CouchbaseQueryExecutor will order ignoring case… really… (big grin)

You need the meta.id to filter out ids starting with ‘_sync:’

Limit and offset are trivial. if you have entities 1, 2, 3, 4, 5, offset is 1, limit is 2, then you’ll get 2, 3 only.

If you don’t want to make the response pageable the following createQueryStatement function will help:

After all of this we have the queryStatement ready with the parameter placeholders, where the placeholder’s keys are the keys in the JsonObject. We have only 2 things left. The first is parametrizing the query, and the second is transforming the response to POJO:

You can see the Paged and non-Paged type of the find query.

Why do we need the convertToDataList function? After you execute the query, you get a LinkedHashMap like this:

That function will take the id and put it into the data. After that you can convert it to POJO.

If you want to see the full code, or use CouchbaseQueryExecutor as a lib checkout TeamWanari’s github repo https://github.com/TeamWanari/couchbase-query-executor

Last thoughts

I hope my post was useful for you. If you want to be notified about any changes in the lib, follow the GitHub repo, or if you have any suggestions, please make an issue (smile)

Alex Sükein

Alex Sükein

- Süxy, can you tell me about robust software development Nasa uses?
- Yes. If we have a final exam tomorrow.