techbits.de » java

Recursive file iteration in Java

Florian — Thu, 11 Feb 2010 21:15:48 +0000

In a java frameworks talk I gave recently I showed the following example for finding files recursively…

public void listFilesInDirectory(File dir) {
  File[] files = dir.listFiles();
    if (files != null) {
      for (File f : files) {
         if (f.isDirectory()) {
	    listFilesInDirectory(f);
	 }
	 else {
	    System.out.println(f.getName());
	}
     }
  }
}

… which in real projects often grows to a larger block of code. The web is full of code blocks like this for walking a directory tree. The best version I came across is this object oriented one by Torsten Curdt. Since you usually don’t want to write this yourself, I suggested in my talk to use FileUtils which makes recursive iteration much easier:

Collection jspFiles = FileUtils.listFiles(rootDirName,
                        new String[] { "jsp" }, true);

This looks concise and useful but as I tried to use it, I wasn’t too pleased with the FileUtils’ solution. Here is why:

The recursion is processed in one go, i.e. all results are written to a List even when using the iterateFiles method. The recursion is not processed iteratively.
You can not influence the directories that are searched.
Only files are returned, you can not search for directories.
The API is not very expressive (e.g. what does the “true” mean).
No generics (raw collection types are returned).

A Better API

Not being satisfied with the solutions I found, I “dreamed up” my own API for listing and finding files. I don’t consider it complete but for the most part I am pleased with the ease of use that the builder pattern provides. The code for this can currently be found in an unrelated goole code project. The rest of this article shows the functions that are currently supported.

Find files two ways

There are generally two ways to use the result – as interator or as list:

1. Iterate over all files in the windows directory:

for (File f : Files.find("c:\\windows")) {

}

2. Get all the files in a directory as a list of files:

List allFiles = Files.find(somedir).list();

Except from the return type the second version does the same as the JDK command listFiles:

File[]  allFiles = (new File(somedir)).listFiles()

Easy recursive listing

To iterate all the files in the C:\Windows directory, you would use:

for (File f : Files.find("c:\\windows").recursive()) {

}

Note: This actually works iteratively, i.e. the recursion happens as you fetch files from the iterator. The result is not fetched into a huge list.

With a Predicate you can limit the recursion to specific directories. In this example all .svn directories within a source tree are skipped:

Predicate noSvnDirs = new Predicate() {
 boolean apply(File file) {
 return !file.getName().equals(".svn");
 }
}

for (File f : Files.find("src/java/").recursive(noSvnDir)) {

}

Want Files, Directories or both?

Define if you want only files, only directories or both in your result with yield*()-Methods.

Files.find(someBaseDir).recursive().yieldFiles()  // this is the default
Files.find(someBaseDir).recursive().yieldDirectories()
Files.find(someBaseDir).recursive().yieldFilesAndDirectories()

Filtering the results

To get all textfiles within a dir use:

Files.find(dir).withExtension("txt").list();
Files.find(dir).ignoreCase().withExtension("txt").list();

You can also filter by Name, e.g. to find README files:

Files.find(dir).withName("README").list();
Files.find(dir).ignoreCase().withName("readme").list();

Note that the default matching is case sensitive. The commands caseSensitive() and ignoreCase() can be used to toggle the matching behaviour.

For special needs you can also specify a Predicate to filter the resulting files.

Files.find(dir).recursive().withFilter(somePredicate).list();

Finding Directories

When looking for directories there are some special usecases that are supported, e.g. looking for directories that contain a specific file:

Files.find(dir).recursive().yieldDirectories()
               .containingFile("Thumbs.db");

Android exploration continued

Florian — Mon, 04 Jan 2010 12:01:13 +0000

I extended my android application with a preferences screen now, which is quite easy to do.

A tutorial for creating a Preferences Activity got me started – unfortunaltely the xml preferences definition it uses is incorrect. The tags are names of Classes which have to be capitalized.
The open the preferences activity I added a Option Menu.
I wanted to add some kind of progress indicator. I ended up using the ProgressDialog and the AsyncTask to run the downloading and xml parsing in the background. To fix issues with device rotation I might have a look at the BetterAsyncTask in the Droid-FU library later.
I also ran into DateFormat and Date issues with the UTC formated Date in the XML file. I Thought about using joda-time at least twice but then stuck to the JDK implementation for smaller app size. The fact that android brings it’s own class named DateFormat which just provides a localized JDK-DateFormat object doesn’t help either.

First Steps with Android

Florian — Wed, 30 Dec 2009 10:02:54 +0000

In the last few days I started out with some android development. Here are some things learned so far developing my first app:

The Android Tutorials are a great starting point, though i only followed through with the HelloWorld Tutorial. In retrospect I should have looked at the Notepad Tutorial a little closer because it explains important concepts (namely activities/intents).
http://www.anddev.org/ is a useful source for tutorials and code snippets
It’s still java but a completely different API, so you often have to look for classes and methods via code completion oder in examples to get things done.
Downloading: Can be done with the included HTTP Client library. Unfortuantely Android still uses an old version of the HTTP Client though, which made it hard to find documentation (e.g. how to set authentication credentials). Additionally you shoudn’t forget to declare the INTERNET-permissions in your application manifest.
Storing and retrieving Files looks fairly easy (getDir(), getCacheDir()-Methods are there) at first sight but you have to unerstand the Android filesystem security model if you don’t want to spend hours with debugging. The before mentoined methods use internal storage where each application stores it’s data independently. Public read/write (e.g. file exchange with other applications) is only possible when you store your content with the specific method openFileOutput(). The external SD card on the other hand can be openly accessed with the regular Java File API.
XML Parsing: I started out with the sax parser but since my XML file was pretty complex I ditched it and downloaded dom4j which has a really easy to use API. Unfortunately it adds at least 200KB of final app size. I now realized I could have gone with the regular DOM parser which has a decent API. I’ll have to reevaluate this later – maybe the end user responsiveness does require the faster streaming parser approach (sax).
UI design: Declarative XML based looks powerful and well thought out but I mostly stuck to tutorial layout for now. This is an area I still have to get into.

Ok, that’s it for now. Android is turning out to be a great plattform – exciting times.

Adding Googles GData Java API to your maven repository

Florian — Thu, 06 Aug 2009 17:09:34 +0000

The google gdata apis do not come with maven POM-files. Someone went through the trouble to “mavenize” the source but it is limited to linux as build plattform and currently out of date (compile errors). So I installed the JARs from the binary distribution of the APIs into my local repository – which are of course missing the dependencies between the individual JAR files. Here are two batch files which I used to install the JARs quite painlessly:

install.bat:

@SET mvn=d:\java\maven\bin\mvn
@%mvn% install:install-file -DgroupId=com.google.gdata
       -DartifactId=%1 -Dversion=%2 -Dfile=%3 -Dpackaging=jar
       -DgeneratePom=true

installall.bat:

call install.bat gdata-analytics 2.0 gdata-analytics-2.0.jar
call install.bat gdata-appsforyourdomain 1.0 gdata-appsforyourdomain-1.0.jar
call install.bat gdata-base 1.0 gdata-base-1.0.jar
call install.bat gdata-blogger 2.0 gdata-blogger-2.0.jar
call install.bat gdata-books 1.0 gdata-books-1.0.jar
call install.bat gdata-calendar 1.0 gdata-calendar-2.0.jar
call install.bat gdata-client 1.0 gdata-client-1.0.jar
call install.bat gdata-codesearch 2.0 gdata-codesearch-2.0.jar
call install.bat gdata-contacts 3.0 gdata-contacts-3.0.jar
call install.bat gdata-core 1.0 gdata-core-1.0.jar
call install.bat gdata-docs 2.0 gdata-docs-2.0.jar
call install.bat gdata-finance 2.0 gdata-finance-2.0.jar
call install.bat gdata-health 2.0 gdata-health-2.0.jar
call install.bat gdata-maps 2.0 gdata-maps-2.0.jar
call install.bat gdata-media 1.0 gdata-media-1.0.jar
call install.bat gdata-photos 2.0 gdata-photos-2.0.jar
call install.bat gdata-spreadsheet 3.0 gdata-spreadsheet-3.0.jar
call install.bat gdata-webmastertools 2.0 gdata-webmastertools-2.0.jar
call install.bat gdata-youtube 2.0 gdata-youtube-2.0.jar

You surely could get fancy and automate the splitting between artifact-name and version number, but hey, I needed those JARs installed quickly and that’s what it does.

IndexedList: A hybrid of a Java List and a Map

Florian — Sat, 18 Jul 2009 11:00:09 +0000

If been working on legacy data import code in the last months with a lot of code searching for exisiting objects in lists. I realized I needed a different collection to speed up searching for identifiers and couldn’t find any standard collection that matches my needs. I basically need an index like Maps.uniqueIndex() from the Google Collections Library provides but it should be updating dynamically and not only be generated once.

To summarize my requirements:

A List (ArrayList) which can be modified as usual
A way to find objects (contains and get) in constant time by a custom attribute (like a HashMap)
No reliance on equals() and hashcode() because they cover more attributes (an can not be changed)

The LinkedHashMap seems to almost fit the bill but it is not a true List and therefore not a great drop in replacement.

What I’m using right now is a wrapper around a List which maintains an additional HashMap to find objects by the index criterium. The index is automatically updated whenever the list is modified. This is an example on how I use it:

// function to index Persons by their social security number
Function getSsnrForPerson =
  new Function() {
    String apply(Person person) {
      return person.getSocialSecurityNumber();
    }
  }

To create an indexed list for an empty ArrayList with an index on a person’s social security number you can then easily use the Class UniqueIndexedList:

UnqiueIndexedList myList =
  UnqiueIndexedList.create(getSsnrForPerson);

Of course all the usual methods of the List interface can be used as usual:

myList.add(somePerson);
myList.add(anotherPerson);
myList.get(i);
myList.iterator();

Additionaly there are methods to access objects by index:

boolean personIsInList = myList.containsByIndex("34897634853");

Person p = myList.getByIndex("34897634853");

The code for the list implementation currently resides here:

It is basically a hybrid of a List and a special index-Map and works well so far. I am sure that finding objects in list for a given attribute is a very common usecase. Why hasn’t anyone solved this yet? Are there (better) alternatives that I’ve missed?

EJB3 persistence with hibernate

Florian — Mon, 17 Apr 2006 12:30:07 +0000

Since my trip to java posting I looked into EJB3 persistence (JSR-220) with hibernate as promised. It’s shocking how much the EJB3 specification resembles hibernate – Gavin King apparently got most if his hibernate semantics in there. I believe it’s a good thing though. Former hibernate developers can adapt the new sepecification with little effort and on the other hand the definition of persitience classes has been cleaned up nicely with properly defined annotations.
So far I’ve been able to create a simple example application with the hibernate entitymanager/annotations engine. It simply creates two objects, stores them in the database an reads them again with a query. The rest of this post shows how this example app is set up.

Configuration

First of all a configuration file for the database and the persistent classes is needed. The xml file META-INF/persitence.xml is used for that purpose. In my example application I used a local hsqldb database. In addition to the database each persistent class has to be specified in a class-element, in this case example.pojo.Cat.




      transaction-type="RESOURCE_LOCAL">

    example.pojo.Cat


        value="org.hibernate.dialect.HSQLDialect"/>

        value="org.hsqldb.jdbcDriver"/>

        value="jdbc:hsqldb:file:database/ejb3test"/>




true

Data classes with annotations

As seen above only one persistent class is used in this application. It defines a cat, which has an id, a name and a birth date. In addition to that it also has a parent child relationship as a self reference (attributes mother and kittens). You can also see the annotations which define how all these attributes are persisted. This is the bare minimum of annotations that is required for EJB3 persistence to work. In a more complex application where you want a detailed control over relations, database tables, datatypes and query characteristics the annotations will also look a lot more complex.
package example.pojo;


import java.util.ArrayList;

import java.util.Date;

import java.util.List;

import javax.persistence.CascadeType;

import javax.persistence.Entity;

import javax.persistence.GeneratedValue;

import javax.persistence.GenerationType;

import javax.persistence.Id;

import javax.persistence.ManyToOne;

import javax.persistence.OneToMany;
@Entity

public class Cat {
	private String name;

	private Date birthDate;

	private int id;

	private Cat mother;

	private List kittens;
	public Cat() {

		super();

		setKittens(new ArrayList());

	}
	public void addKitten(Cat kitten) {

		this.getKittens().add(kitten);

		kitten.setMother(this);

	}
	// -- GETTERs and SETTERs --
	@OneToMany(cascade = CascadeType.ALL, mappedBy="mother")

	public List getKittens() {

		return kittens;

	}

	public void setKittens(List kittens) {

		this.kittens = kittens;

	}
	@ManyToOne(optional=true)

	public Cat getMother() {

		return mother;

	}

	public void setMother(Cat parent) {

		this.mother = parent;

	}
	@Id	@GeneratedValue(strategy=GenerationType.AUTO)

	public int getId() {

		return id;

	}

	public void setId(int id) {

		this.id = id;

	}
	public String getName() {

		return name;

	}

	public void setName(String name) {

		this.name = name;

	}

public Date getBirthDate() { return birthDate; } public void setBirthDate(Date birthDate) { this.birthDate = birthDate; } }

The entity manager

The EJB3 entity manager is what used to the session in hibernate. It provides all functionality to store, update and query objects from the database. In the example code below it is created from a EntityManagerFactory which makes use of the persistence.xml configuration file – you can see the reference to the persistence-unit manager1 here.
Using the entity manager a transaction is started, 2 cats are created, linked as mother and kitten and finally persited with a call to persist() of the entity manager. This step is as simple as with a good old hibernate. Same is true for the query part which follows after that – a parameterized query returns all cats that where born before 1.1.2003.
package example;


import java.text.ParseException;

import java.text.SimpleDateFormat;

import java.util.Iterator;

import java.util.List;
import javax.persistence.EntityManager;

import javax.persistence.EntityManagerFactory;

import javax.persistence.EntityTransaction;

import javax.persistence.Persistence;
import example.pojo.Cat;
public class Test {
	public static void main(String[] args) throws ParseException {
		// Read configuration from META-INF/persistence.xml

		EntityManagerFactory emf = Persistence.createEntityManagerFactory("manager1");
		EntityManager em = emf.createEntityManager();
		// --------- CREATE CATS -----------
		EntityTransaction t = em.getTransaction();

		t.begin();
		// Create two cats

		SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd");

		Cat mimi = new Cat();

		mimi.setName("Mimi");

		mimi.setBirthDate(df.parse("2001-03-01"));

		em.persist(mimi);
		Cat max = new Cat();

		max.setName("Max");

		max.setBirthDate(df.parse("2003-05-18"));

		em.persist(max);
		// Mimi is the mother of max

		mimi.addKitten(max);
		// Commit transaction to store the cats in the database

		t.commit();
		// --------- QUERY CATS -----------
		// Get all cats that where born before 1.1.2003

		List catList = em.createQuery(

	    	"select cat from Cat as cat where cat.birthDate < ?1")

	    	.setParameter(1, df.parse("2003-01-01"))

	    	.getResultList();
		// Print the list of cats with the number of their children

		Iterator catIterator = catList.iterator();

		while (catIterator.hasNext()) {

			Cat mycat = (Cat) catIterator.next();
			System.out.println(mycat.getName() + " children:"+

					mycat.getKittens().size());

		}
		// Close the Entity Manager

		em.close();

		emf.close();

	}

}

Preliminary conclusion

I realize, this simple example doesn't say much about the feature set of EJB3 compared to the hibernate features (caches, optimized queries, relations, etc..). It is a good starting point for further experiments though which is why I provide the example eclipse project as a download including all required libraries. This means this should run in any 3.1 eclipse out of the box.
Did you notice that none of the import statements reference a hibernate specific package? The application only uses the javax.persistence interface so theoretically you should be able to switch from hibernate to any other ejb3 persistence solution without changing a single line of code.
Getting rid of the hibernate mapping files which had to be edited separately from the code or be generated with xdoclet is a big plus in my opinion. I would move my last project from xdoclet tags to annotations in an instant if there wasn't the requirement to use java 1.4.

Download the example

If you like to experiment yourself, download the Ejb3PersistenceTest.zip (5MB) eclipse project. It contains all hibernate libraries (core, entity manager, annotations) to get you started.

Diving into java scripting frameworks

Florian — Tue, 21 Mar 2006 20:10:01 +0000

As I wrote sunday, I experimented with beanshell and it worked pretty well und faster than I anticipated. Nevertheless I heard about some other scripting frameworks (and their integration in java 1.6) which motivated me to investigate a bit more in this direction. Apparently Mozilla’s Rhino is a very powerful yet fast framework that provides javascript aka ECMAscript. Performance benchmarks by Pankaj Kumar in his article BeanShell, Rhino and Java — Performance Comparison show that my choice of beanshell as a scripting framework might not have been that best – performance wise. I’ll definately check out Rhino for use in my application.

Another interesting framework I came across when looking into scripting with java is the Apache Jakarta Bean Scripting Framework (BSF) which provides a framework to create JSP pages in different scripting languages. To achieve this it offers a uniform interface which wraps several scripting languages like javascript, python, tcl and many more.

A trip to java

Florian — Sun, 19 Mar 2006 22:09:39 +0000

Once again a long time has passed where I couldn’t find the time or motivation to write a post. I was mainly occupied with the specification of a java application which gave me the opportunity to get to know Sparx System’s Enterprise Architect. I only used the common uml diagram types and the code generation features for this project and they work pretty well. I’d love to make use of the more advanced features though and look into the whole MDA approach a bit more. There is a fundamental difference in just drawing some class diagrams and really developing model driven from the ground up. Projects like AndroMDA to generate complete applications look promising and certainly could reduce the expenses for software development but I doubt I’ll find time to check it out properly any time soon. As for the modelling language I don’t think everything should be done with UML though – the approach to model every aspect in UML and using stereotypes to define “what it means” makes the modeling too confusing and complex imho. Some interesting views on MDA and Software Factories can be found in the podcast MDA vs. Software Factories and some older episodes.

At the moment I’m working on a prototype for another java application and got used to the new java 1.5 features like generics and the new for each loops. The generics are definately the way to go for typesafe collections and I’ll try to use them from here on out. I found the wildcard and some other constructions confusing at first but I do understand that there are several restrictions (german link) when using generics. Someone should publish some design patterns for typical cases that require generics.

For that project I also experimented with BeanShell which is basically a java scripting interpreter. I’ll probably use it for custom formulas which are evaluated dynamically within the java application. Easy to use and powerful.

The next component I’ll look into is EJB3 persistence using Hibernate. I have worked with the hibernate tools and reverse enginered some database tables but since I already have the POJO classes and don’t want to create all the old hibernate xml mapping files I might as well check out how to create EJB3 persistence by using annotations. More on that soon.