In the previous part of my Spring Data Solr tutorial, we learned how we can configure Spring Data Solr. Now it is time to take a step forward and learn how we can manage the information stored in our Solr instance. This blog entry describes how we add new documents to the Solr index, update the information of existing documents and delete documents from the index.
We can make the necessary modifications to our example application by following these steps:
- Create a document class which contains the information stored in the Solr index.
- Create a repository interface for our Spring Data Solr repository.
- Create a service which uses the created repository.
- Use the created service.
These steps are described with more details in the following sections.
Creating the Document Class
The first step is to create a document class which contains the information added to the Solr index. A document class is basically just a POJO which is implemented by following these rules:
- The @Field annotation is used to create a link between the fields of the POJO and the fields of the Solr document.
- If the name of the bean’s field is not equal to the name of the document’s field, the name of the document’s field must be given as a value of the @Field annotation.
- The @Field annotation can be applied either to a field or setter method.
- Spring Data Solr assumes by default that the name of the document’s id field is 'id'. We can override this setting by annotating the id field with the @Id annotation.
- Spring Data Solr (version 1.0.0.RC1) requires that the type of the document's id is String.
More information:
Let’s move on and create our document class.
In the first part of my Spring Data Solr tutorial, we learned that we have to store the id, description and title of each todo entry to the Solr index.
Thus, we can create a document class for todo entries by following these steps:
- Create a class called TodoDocument.
- Add the id field to the TodoDocument class and annotate the field with the @Field annotation. Annotate the field with the @Id annotation (This is not required since the name of the id field is 'id' but I wanted to demonstrate its usage here).
- Add the description field to the TodoDocument class and annotate this field with the @Field annotation.
- Add the title field to the TodoDocument and annotate this field with the @Field annotation.
- Create getter methods to the fields of the TodoDocument class.
- Create a static inner class which is used to build new TodoDocument objects.
- Add a static getBuilder() method to the TodoDocument class. The implementation of this method returns a new TodoDocument.Builder object.
The source code of the TodoDocument class looks as follows:
import org.apache.solr.client.solrj.beans.Field; import org.springframework.data.annotation.Id; public class TodoDocument { @Id @Field private String id; @Field private String description; @Field private String title; public TodoDocument() { } public static Builder getBuilder(Long id, String title) { return new Builder(id, title); } //Getters are omitted public static class Builder { private TodoDocument build; public Builder(Long id, String title) { build = new TodoDocument(); build.id = id.toString(); build.title = title; } public Builder description(String description) { build.description = description; return this; } public TodoDocument build() { return build; } } }
Creating the Repository Interface
The base interface of Spring Data Solr repositories is the SolrCrudRepository<T, ID> interface and each repository interface must extend this interface.
When we extend the SolrCrudRepository<T, ID> interface, we must give two type parameters which are described in the following:
- The T type parameter means the type of our document class.
- The ID type parameter means the type of the document’s id. Spring Data Solr (version 1.0.0.RC1) requires that the id of a document is String.
We can create the repository interface by following these steps:
- Create an interface called TodoDocumentRepository.
- Extend the SolrCrudRepository interface and give the type of our document class and its id as type parameters.
The source code of the TodoDocumentRepository interface looks as follows:
import org.springframework.data.solr.repository.SolrCrudRepository; public interface TodoDocumentRepository extends SolrCrudRepository<TodoDocument, String> { }
Creating the Service
Our next step is to create the service which uses the created Solr repository. We can create this service by following these steps:
- Create a service interface.
- Implement the created interface.
These steps are described with more details in the following.
Creating the Service Interface
Our service interface declares two methods which are described in the following:
- The void addToIndex(Todo todoEntry) method adds a todo entry to the index.
- The void deleteFromIndex(Long id) method deletes a todo entry from the index.
Note: We can use the addToIndex() method for adding new todo entries to the Solr index and updating the information of existing todo entries. If an existing document has the same id than the new one, the old document is deleted and the information of the new document is saved to the Solr index (See SchemaXML @ Solr Wiki for more details).
The source code of the TodoIndexService interface looks as follows:
public interface TodoIndexService { public void addToIndex(Todo todoEntry); public void deleteFromIndex(Long id); }
Implementing the Created Interface
We cam implement the service interface by following these steps:
- Create a skeleton implementation of our service class.
- Implement the method used to add documents to the Solr index.
- Implement the method used to delete documents from the Solr index.
These steps are described with more details in the following.
Creating a Skeleton Implementation of the Service Class
We can create a skeleton implementation of our service interface by following these steps:
- Create a class called RepositoryTodoIndexService and annotate this class with the @Service annotation. This annotation marks this class as a service and ensures that the class will be detected during the classpath scanning.
- Add a TodoDocumentRepository field to the RepositoryTodoIndexService class and annotate that field with the @Resource annotation. This annotation instructs the Spring IoC container to inject the actual repository implementation the service’s repository field.
The source code of our dummy service implementation looks as follows:
import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import javax.annotation.Resource; @Service public class RepositoryTodoIndexService implements TodoIndexService { @Resource private TodoDocumentRepository repository; //Add methods here }
Adding Documents to the Solr Index
We can create the method which adds new documents to the Solr index by following these steps:
- Add the addToIndex() method to the RepositoryTodoIndexService class and annotate this method with the @Transactional annotation. This ensures that our Spring Data Solr repository will participate in Spring managed transactions.
- Create a new TodoDocument object by using the builder pattern. Set the id, title and description of the created document.
- Add the document to the Solr index by calling the save() method of the TodoDocumentRepository interface.
The source code of the created method looks as follows:
import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import javax.annotation.Resource; @Service public class RepositoryTodoIndexService implements TodoIndexService { @Resource private TodoDocumentRepository repository; @Transactional @Override public void addToIndex(Todo todoEntry) { TodoDocument document = TodoDocument.getBuilder(todoEntry.getId(), todoEntry.getTitle()) .description(todoEntry.getDescription()) .build(); repository.save(document); } //Add deleteFromIndex() method here }
Deleting Documents from the Solr Index
We can create a method which deletes documents from the Solr index by following these steps:
- Add the deleteFromIndex() method to the RepositoryTodoDocumentService class and annotate this method with the @Transactional annotation. This ensures that our Spring Data Solr repository will participate in Spring managed transactions.
- Delete document from the Solr index by calling the delete() method of the TodoDocumentRepository interface.
The source code of the created method looks as follows:
import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import javax.annotation.Resource; @Service public class RepositoryTodoIndexService implements TodoIndexService { @Resource private TodoDocumentRepository repository; //Add addToIndex() method here @Transactional @Override public void deleteFromIndex(Long id) { repository.delete(id.toString()); } }
Using the Created Service
Our last step is to use the service which we created earlier. We can do this by making the following modifications to the RepositoryTodoService class:
- Add the TodoIndexService field to the the RepositoryTodoService class and annotate this field with the @Resource annotation. This annotation instructs the Spring IoC container to inject the created RepositoryTodoIndexService object to the service’s indexService field.
- Call the addToIndex() method of the TodoIndexService interface in the add() method of the RepositoryTodoService class.
- Call the deleteFromIndex() method of the TodoIndexService interface in the deleteById() method of the RepositoryTodoService class.
- Call the addToIndex() method of the TodoIndexService interface in the update() method of the RepositoryTodoService class.
The source code of the RepositoryTodoService looks as follows:
import org.springframework.security.access.prepost.PreAuthorize; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import javax.annotation.Resource; import java.util.List; @Service public class RepositoryTodoService implements TodoService { @Resource private TodoIndexService indexService; @Resource private TodoRepository repository; @PreAuthorize("hasPermission('Todo', 'add')") @Transactional @Override public Todo add(TodoDTO added) { Todo model = Todo.getBuilder(added.getTitle()) .description(added.getDescription()) .build(); Todo persisted = repository.save(model); indexService.addToIndex(persisted); return persisted; } @PreAuthorize("hasPermission('Todo', 'delete')") @Transactional(rollbackFor = {TodoNotFoundException.class}) @Override public Todo deleteById(Long id) throws TodoNotFoundException { Todo deleted = findById(id); repository.delete(deleted); indexService.deleteFromIndex(id); return deleted; } @PreAuthorize("hasPermission('Todo', 'update')") @Transactional(rollbackFor = {TodoNotFoundException.class}) @Override public Todo update(TodoDTO updated) throws TodoNotFoundException { Todo model = findById(updated.getId()); model.update(updated.getDescription(), updated.getTitle()); indexService.addToIndex(model); return model; } }
Summary
We have successfully created an application which adds documents to the Solr index and deletes documents from it. This blog entry has taught us the following things:
- We learned how we can create document classes.
- We learned that we can create Spring Data Solr repositories by extending the SolrCrudRepository interface.
- We learned that Spring Data Solr assumes by default that the name of the document’s id field is 'id'. However, we can override this setting by annotating the id field with the @Id annotation.
- We learned that at the moment Spring Data Solr (version 1.0.0.RC1) expects that the id of a document is String.
- We learned how we can add documents to the Solr index and delete documents from it.
- We learned that Spring Data Solr repositories can participate in Spring managed transactions.
The next part of my Spring Data Solr tutorial describes how we can search information from the Solr index by using query methods.
P.S. The example application of this blog entry is available at Github.
Hi, Petri
When will next part of Spring Data Solr tutorial?
I will start writing it tomorrow and publish it as soon as possible.
Thanks, this is very useful tutorial.
By the way, one of the last commits in Sprig Data Solr, removes limitation (only String) on type id field.
"DATASOLR-48 - Add support for numeric document id " -
https://github.com/SpringSource/spring-data-solr/commit/b9afaa58627390b69f95b6ef715f2f1f008cab1b
I noticed that commit a bit earlier and it is indeed a useful fix that helps us to clean up the code a bit. I think that I will wait for the next release before updating this tutorial.
Anyway, thanks for pointing this out here. It might be valuable to someone who wants to use the build snapshot in his project.
when i am running your project i am getting illegalstateexception failed to load applicationcontext and java.lang.IllegalStateException: Failed to load ApplicationContext
at org.springframework.test.context.TestContext.getApplicationContext(TestContext.java:157)
at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:103)
Update: I removed most of the stack trace - Petri
Hi,
Are you by any change using Java 8? The reason why I ask this is that the example application doesn't support Java 8 (its dependencies are too old). If you want to run it, you have to either update the dependencies manually or use Java 7.
This documentation helped me lot.. Thank you..
Is there anything for upload xml data files with different schema and tags into solr(i.e. indexing)?
You are welcome!
You can index XML documents by using the DataImportHandler or implementing a custom XML reader which adds new documents to the Solr index by using Spring Data Solr.
If you have to index only a few different XML documents, you should probably implement your own indexer component. On the other hand, if you have to support multiple different XML documents, I would use the DataImportHandler.
Check out this StackOverflow question for more details about this.
I hope that this answered to your question.
As you mentioned, using DataImportHandler(DIH) I created solr cores for import xml/csv files and now I am getting query results as I expect, thanks for your guidance.
My application is combination of Spring 3.0 and Sencha Touch (ohh! mobile app), where the next step is to integrate it with solr so that we can provide search functionality on the documents which we uploaded through DIH.
Can you please suggest the approach to integrate solr? Performance is one more my worry part in integration due to it is mobile app.
I would probably create a REST API which provides the required search function to the client. I would also save the used search terms so that I know how users are using the API. This is helpful if the you encounter performance problems.
Here are a few links which explains how you can tune the performance of Solr:
Also, if the performance of a single Solr instance is not enough, you might have to use SolrCloud.
I hope that this was helpful to you.
Hi Petri,
I was trying out your examples (solr data series); it was a great help getting me going with spring-data-solr. I have a question related to solr date fields. How do i index the date fields? I tried modifying the solr schema.xml to add solr.TrieDateField as fieldType and added a field that used this. Then i tried passing a utc date string to this field. I am not able to get it to index though.
Any ideas?
Hi,
First, you have declare the date field in the schema.xml file:
Second, you have add the
Date
field to your document class:This should do the trick!
That worked like a charm!
Thanks
You are welcome!
Is there any problem with tdate field?
I get this error with latest snapshot.
Failed to convert from type java.lang.String to type java.util.Date for value '1365517622000'
Thanks.
Are you talking about the 1.1.0.BUILD-SNAPSHOT? I haven't used it personally (I use 1.0.0.RELEASE in my projects) so I am not sure if the latest snapshot has problems with
tdate
fields.You might want to create a new issue to the Spring Data Solr issue tracker.
Hi Petri,
I have one more complication in my spring data solr project. With your help I've been able to get everything indexed and working properly but i haven't been able to deal with a complex object scenario. I am not being able to index a list or a set inside my main class.In your todo example, I added a set of todopoints. Then i added a field called point of type text_general:
I was expecting the RepositoryTodoIndexService.addToIndex to add the todopoints to my solr index but apparently that's not happening.
Is there a different way of doing this?
My todo document has a set :
private Set todoPoints;
And my TodoPoint is where I added the solr annotation:
@Field("point")
private String point;
Well, I figured it out!
As solr does not seem to handle composite classes, I had to flatten the todoPoints Set to List todoPoints. And then had to annotate this List for solr indexing.
@Field(“point”)
List todoPoints;
I had to then change my schema.xml to add the point field:
Thanks
Great! It is good to hear that you solved your problem.
Also, I haven't found a way to use composite classes with Spring Data Solr. As far as I know, your solution (flattening the class hierarchy) is the only one.
Hi, Thank you for the tutorial.. it was really helpfull.
I do however have a question about the use of a generic document object.. in Solj one can just use the SolrInputDocument object to load data into Solr. This does not require us to create a document object with present field values.. My use-case is to allow users to specify an arbitrary xml doc or database table (I have no idea what field names would be used) and have my application load this into solr. Is there a way in Spring Data to make use of the SolrInputDocument where I can specify the fields as key-value pairs?
Thanks you.
Actually this is possible. Depending from your use case, you might want to
These tutorials explain how you can extend your Spring Data Solr repositories with custom methods. You can create the indexed
SolrInputDocument
object in your custom method and add it to Solr index by calling thesaveDocument(SolrInputDocument doc)
method of theSolrOperations
interface.I hope that this answered to your question.
I followed your tutorial. It is very nice and helpful tutorial, But I am facing small problem in "interface TodoIndexService" here you mentioned a
"void addToIndex(Todo todoEntry)" method adds a todo entry to the index. But Todo.java is missing
I have left a lot of things out from this tutorial because I felt that were not related to Spring Data Solr. You can get the full source code of the example application from Github. Here is a direct link to the source code of the Todo class.
I hope that this solves your problem.
Thank your for your tutorial very well explained :)
You are welcome!
hi thank u for your tutorial very well explained :)
i have a doubt.. if my document class having a field - which is list of another document object
eg.
class University{
@Field
List colleges;
@Fileld
List departments;
}
here College and Department are other document classes or simply other POJO classes.. how can i define type for these fields in schema.xml file
Thanks in advance.
Hi,
You should check out these StackOverflow questions:
Unfortunately I cannot give you a better answer since I have never done this myself.
Hi sir,
Thanks for your post about Apache Solr.
I'm based to your docs to create Solr Spring data project with Spring annotations, Apache Solr 4.9.1, JDK8, Cassandra 2.x(config is OK), multiple Documents: User, Group, Directory.
I get following error :
org.springframework.data.solr.UncategorizedSolrException: Document is missing mandatory uniqueKey field: id; nested exception is org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Document is missing mandatory uniqueKey field: id
at org.springframework.data.solr.core.SolrTemplate.execute(SolrTemplate.java:145)
at org.springframework.data.solr.core.SolrTemplate.saveBean(SolrTemplate.java:184)
at org.springframework.data.solr.core.SolrTemplate.saveBean(SolrTemplate.java:178)
at org.springframework.data.solr.repository.support.SimpleSolrRepository.save(SimpleSolrRepository.java:149)
Repository:
Entity:
Config:
Update: I removed irrelevant code since the comment was unreadable - Petri
Hi,
Check out this StackOverflow question. It describes a few reasons that can cause this error.
Hy Petri,
I tried to implment your above tutorial, but wont be able to run succesfully. Actually i want to create core at run time, so whenever i use any repository method, then it gives an exception=======
Expected mime type application/xml but got text/html=====
=> I am using solr template method to create solrTemplate with specific coreName
SolrTemplate solrTemplate = new SolrTemplate(solrClient,coreName);
new SolrRepositoryFactory(solrTemplate).getRepository(SolrRepository.class).save (toDoDocument);
I have a bean (ToDoDocument) which i wanted to get indexed
class ToDoDocument{
@Field
private String id;
@Field
private String name;}
so whwenever I use save() then it gives an exception======= Expected mime type application/xml but got text/html=====
=> but whe i use single core and mention @solrDocument annotation over the ToDoDocument, then it works fine
@SolrDocument(solrCoreName = "solrcore")
public class ToDoDocument
BUT, i dont want to hadcoded core name, so can you please give me any solution for my scenerio.
Hi Petri,
I am using Spring Data Solr 2.1.7 in one of my application and I am getting the following exception while saving the document in solr.
org.springframework.data.solr.UncategorizedSolrException: Error from server at http://localhost:8983/solr: Expected mime type application/octet-stream but got text/html.
Error 404 Not Found
HTTP ERROR 404
Problem accessing /solr/productitem/update. Reason:
Not FoundPowered by Jetty://
I spoild my two days in resolving the issue but I failed and finally decided to email you regarding the same . Plz help me out.
Thanks / Mreetunjay Kumar
Hi,
Is productItem the name of your collection? The reason why I ask this is that one possible reason for the 404 error is that the specified collection is not found. If this is the case, you are using the wrong URL.