Comments on: Supporting Multiple Versions of a File Format

By: Joel Sumner

Joel Sumner — Thu, 26 Apr 2007 18:08:44 +0000

Good advice Jim. You can see over the years, we’ve added some file formats to LabVIEW (http://ideasinwiring.blogspot.com/2005/11/storing-and-retrieving-data.html) that make it hard for customers to follow your rules and that can cause problems.

I’ve seen folks hang themselves with the Datalog file (http://www.kip.uni-heidelberg.de/fp-computer/LabVIEW/FP_LabView/html/Writing_Datalog_Files.html) by writing a file with one particular type definition (i.e. cluster) and then can’t read the file any more when they change clusters without understanding it will affect backward compatibility.

The newer file formats like .TDM (http://zone.ni.com/devzone/cda/tut/p/id/3542) are much more maintainable because they store the type definition (typically called a ’schema’) in the data file so you can’t lose it and so it can be browsed by an external application.

By: Jim Kring

Jim Kring — Wed, 18 Apr 2007 15:44:07 +0000

Aristos Queue,

Thanks for the tip. Yes, backwards compatibility of newer file versions with older readers is a very useful feature. I can think of a few ways to achieve this. For INI files, don’t remove or rename sections or keys. For CSV files, access columns by name (header), rather than by column index, and don’t remove or rename columns. For XML files, don’t remove or rename entities.

Thanks,

-Jim

By: Aristos Queue

Aristos Queue — Tue, 17 Apr 2007 23:31:23 +0000

I also recommend defining some syntax that means “comment” in your file which can be nested somehow (so some symbol that starts the comment and another that ends it such as /* and */ in C++ code). This allows for you to have different versions of your file reader/writer that can embed new features and still produce an older style document that older readers can load. For example, you might use /** **/ to encode some actual data that only a more recent reader can read, but an older reader will skip over it as a comment. Otherwise every time you want to add a feature or increase the data recorded in a file, you must bump your file definition version and then you’re stuck being unable to load the file in an older version.

By: Jim Kring

Jim Kring — Tue, 17 Apr 2007 17:48:54 +0000

Dave,

You’re welcome — I’m glad you liked the article. I agree with you that the INI files are extremely flexible, and are a perfect compliment to the technique described in the article. Quickly evolving projects don’t necessarily benefit from a rigorous file versioning scheme, but with the right tools and templates for implementing such a scheme, the small cost to implement it can be well worth the reward.

Thanks,

-Jim

By: Dave T

Dave T — Tue, 17 Apr 2007 16:12:52 +0000

Jim,
Thanks for the great article. There are some good ideas there for maintaining a well-documented, robust file-format chain.
I’ve used .ini files to create less-rigorously maintained, flexible file formats. New fields can be added without breaking the reader/writer, and default values, if properly managed, can make old files work fairly well with the new format. Fields can be removed from the format, with no harm done other than the unused fields cluttering up the .ini files. Like I said, it’s a less rigorous approach, but it makes it really easy to change file formats on quickly-evolving R&D projects.
Now I’ll start thinking about combining your suggestions with my current practices to get some of the best features of each.

Cheers,
Dave T.