Groovy Course Correction - Chapter 21 (Files and Templates)

I've been made aware today (thanks to a customer asking us to help solve a problem with his code relating to chapter 21 of the Groovy course) of a mistake on the video and with the file supplied for this chapter.

In the exercise I set you to practice with templates, I show on screen the file called DailyCheckInTemplate.txt from the practicals and code folder. This is at approximately 15:22 on the video.

The video tells you to copy the file from the templates folder in chapter 17, and shows you the file on screen. Unfortunately the file provided and shown is not right - it includes fields like $it.date - these should be $date.

The problem with using $it.date is that Groovy will be looking for a key of it.date in the map of properties we supply to the template engine, but the keys won't be preceeded with the "it" - that is we'll be creating a map with a key of "date" and not "it.date".

If you use the version of the template supplied in the starting workspace, you'll get an error message similar to this:

Caught: groovy.lang.MissingPropertyException: No such property: it for class: SimpleTemplateScript1
groovy.lang.MissingPropertyException: No such property: it for class: SimpleTemplateScript1
at SimpleTemplateScript1.run(SimpleTemplateScript1.groovy:1)

The version of this file in the final workspace for chapter 21 is correct, so please pick up the DailyCheckinTemplate.txt file from the following location instead, and you'll not have any problems with your code:

PracticalsAndCode / End of Chapter Workspaces / Chapter 21 - Files and Templates / Hotel Manager / templates / DailyCheckInTemplate.txt

Hadoop - new course coming soon!

I'm pleased to announce that recording of my next Virtual Pair Programmers course is now almost complete. The course is covering Hadoop for Java Developers. If you haven't heard of Hadoop (is there anyone who hasn't?) this is a framework for distributing the processing of large amounts of data accross a a network of computers.

The course assume some basic Java knowlege, but no prior knowledge of Hadoop or its Map Reduce programming model.

Once recording is complete, there will be an edit phase and of course some post-production work to complete, but the likely running order is as follows:

• An overview of what Hadoop is, and introducing the concept of the map-reduce programming model
• Getting to grips with map-reduce, including creating some map-reduce code in standard Java
• Hadoop operating modes, and how to set up and install Hadoop
• Creating our first Hadoop Map-Reduce job
• The Hadoop Distributed File System (HDFS)
• Understanding the map-reduce process flow, including combine and shuffle
• Looking at map reduce job configuration options, such as file formats, runtime options
• Creating custom data types 
• Chaining multiple jobs, and adding extra Map steps to jobs 
• Optimising jobs
• Working with JDBC databases
• Unit testing (with MRUnit)
• Secondary Sorting (sorting the values as well as the keys)
• Joining Data from multiple files
• Using the Amazon EMR service

The course has a number of real world examples throughout and two large case studies to work through too so there's lots of practical exercises. As well as model answers and sample code throughout, I'm also including some templates that I use for my own map-reduce jobs which you'll be able to re-use in your own projects.

If you are a Microsoft Windows user, then you need to know that installing Hadoop on Windows is hard, so in the course, I ask you to use a virtual machine running Linux... and I'll talk you through how to install and configure that... no prior knowledge of Linux is required.  Mac and Linux users can either install Hadoop directly, or also use a virtual machine - all the options are covered.

The course should be going live some time in September so keep an eye out on this blog or the Virtual Pair Programmers' facebook page for more information.

Why you can't use Derby with Hadoop

I'm currently in the middle of writing my next course for Virtual Pair Programmers, which will be on using Hadoop. Typically in Virtual Pair Programmers courses, we use Apache's Derby database. We choose this because  that's because it's light-weight, and so easy to distribute. It needs pretty much no installation / configuration etc. We can provide students with a full copy of the application and sample databases, and they can avoid having to spend time setting up and configuring a database server, such as MySQL, and having to import a database.

One of the topics we'll be covering on the Hadoop course is the use of the DBInputFormat and DBOutputFormat to read from and write to a relational database (we'll be learning about Sqoop too but the same issue will affect Sqoop... it's just that I've not got to that part of the script just yet!).

In preparing some data and test code to use on the course, I've today discovered that Hadoop just won't work with Derby. I find this somewhat surprising, given that both projects come from the Apache camp, but having spent several hours digging to find out why this might not work, I've finally found the issue. There's really not much available online about this point so I thought I'd write a blog post about it in the hope that it helps someone in the future avoid the pain I've been through today!

On trying to get database reads working, I've been coming up with a horrible looking error message. I won't bore you with the full stack trace; the important part of it is:

java.sql.SQLSyntaxErrorException: Syntax error: Encountered "LIMIT" at line 1, column 75.

The issue here is that Hadoop generates SQL statements in the background to read from the database. Rather than reading the whole table in one go, each map method call will read the next record. The SQL that Hadoop generates (that we can't see) includes the LIMIT keyword... and as per the derby FAQ this keyword is not supported.

So it seems that there's just no easy way to read in or write out to a Derby database from Hadoop. So on the course we'll be using MySQL to learn how to work with relational databases directly from Hadoop, but for anyone using Derby and wanting to work with Hadoop, I think the only option is going to be to create a dump of the data in text format for Hadoop to import.

If you have found a way to get Derby working with Hadoop please do let me know!

Groovy Programming is now available!

I'm excited to announce that Groovy Programming, the latest training course from Virtual Pair Programmers (my second course for them) is now available to purchase!

As with all Virtual Pair Programmers' courses, Groovy Programming is written from scratch for delivery by video, but is based on many years of experience in working with and teaching the language. We believe you'll learn far more quickly than from reading books - in fact you'll cover everything you need to be a competent Groovy programmer but at a fraction of the cost of a face-to-face course, and naturally with the convenience that our unique training methods give - the ability to download and keep all the video files, so you can study at a time and place that suits you!

 Groovy Programming contains 12 hours of video, but with lots of practical exercises it will take most students around a week to complete. Also included with the download are complete code for all worked exercises, and guidance notes for the tasks, as well as all the software you need (except Groovy and Eclipse, but we cover on the videos how to install and configure these!)

There's a full breakdown of the content of Groovy Programming on our website, but if you have any queries about this course, you're welcome to contact me through https://www.virtualpairprogrammers.com/contact.html. In the meantime, thank you for your continued support, and we hope you continue to enjoy our courses.

Running db-derby with recent releases of Java

This blog post is an errata item for my Java Fundamentals course, and will also apply for the other Virtual Pair Programmers courses where we use the db-derby database.

I discovered while recording the upcoming Groovy course that there has been a security change in the most recent release of Java (1.7.0.51) that has meant that the default configuration for db-derby no longer works. Running db-derby with the startNetworkServer command will result in an error message which says somewhere early on in the error message:

access denied ("java.net.SocketPermission" "localhost:1527" "listen,resolve")

The easiest and quickest way to overcome this seems to be to run the database on a higher port number, such as 50000 - to do this, instead of running the startNetworkServer command, run the following instead to start the db-derby database:

NetworkServerControl -p 50000 start

In your code, you'll need to change the connection strings to incorporate the new port number, so that the line of code which creates the connection includes the port number - it should look like this:

conn = DriverManager.getConnection("jdbc:derby://localhost:50000/library");

This should overcome the error - if you find you have any further unexpected errors that you can't resolve however, do get in contact via the Virtual Pair Programmers contact us page!

Groovy Course coming soon

One of the topics that we've been asked to provide a course on at Virtual Pair Programmers a number of times over the last few years is Groovy... so I'm pleased to announce that I'm currently making good progress on creating the Groovy Course, which I hope will be released around the end of March 2014.

While I've been talking about the Groovy course to fellow Java programmers and trainers, in particular I've had a number of conversations about what it should cover , everyone seems to agree that once you start with Groovy you don't want to switch back to plain old Java, and I guess this is primarily because everything you can do in Java can be done in Groovy, often more quickly, and of course you can do more too. I've come to the conclusion that for many Java programmers it's really quite easy to get up and running in Groovy, but to really harness it's power, and get the most out of the language, it needs a really quite thorough course.

For anyone who doesn't know what Groovy is, I've pasted below a response to the question "what is groovy" taken from Stack Overflow... although this isn't the top answer on that site, it's the one that I like the most as I think it gives a real flavour of what Groovy is really about from a programmer's perspective:

Groovy is a dynamically typed language that runs on the Java platform and includes some features that a lot of people wish were in Java (ex: closures). One nice thing about Groovy is that it reduces the amount of code needed to do common tasks such as parsing XML files and accessing databases. While learning Groovy you can always mix in straight Java code. This is nice since it allows you to ease into Groovy at your own pace while delivering functional code. If you've been using Java for a while I think you'll appreciate the simplicity of using Groovy since you can program more functionality using less keystrokes. The inclusion of closures was a big selling point for me. One word of caution: if you use Groovy for production code you should make sure there is descent test coverage (since Groovy is a dynamic language). Even if you decide not to use Groovy, it's not a huge time investment to learn and experiment a bit.

The chapter list for our forthcoming course isn't quite complete, but the likely structure and outline content is as follows:

• Introduction: The difference between Groovy and Java, installing Groovy and setting up a development environment
• Part One: Getting started - looking at the power of key aspects of programming in Groovy, such as objects, methods, strings, operators, closures, ranges and looping and unit testing
• Part Two: Case Study - using everything we have learned to start to build a working application - our example is going to be a hotel room booking system
• Part Three: Going deeper - more on closures, working with files including XML, databases, Meta Object Protocol, using stubs and mocks in unit testing and finally packaging and deployment

Our case study will start after we have studied the basics, and we'll be developing a back-end system to help a small hotel manage its room bookings. As we learn more advanced topics, we will then enhance the back-end system to provide more functionality. 

We'll put an update on progress nearer the release date, but in the meantime, if you're welcome to contact me through www.virtualpairprogrammers.com

Java Fundamentals - follow up post on Automatic Resource Management

This is the second of two follow up videos for the Java Fundamentals course. In this video, I take the Java try with resources statement a little bit further to tidy up the code we created in chapter 26 of the course.

The video is hosted on youtube - for best viewing results, watch in full screen and choose the HD quality option if that doesn't come up by default when you click on play!

Click here for the video